Rate limiting

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Rate limiting

System Support

I send several weekly newsletters where a number of the e-mails are all relayed to a separate
host.  I would like to rate limit the traffic to this host.  I tried using:

smtp_destination_concurrency_limit = 5
smtp_destination_rate_delay = 1s
smtp_extra_recipient_limit = 5

these evidentially apply to the final destination, not the relay.  What are the correct options for
rate-limiting the traffic to the relay host?

...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Wietse Venema
System Support:
>
> I send several weekly newsletters where a number of the e-mails
> are all relayed to a separate host.  I would like to rate limit
> the traffic to this host.  I tried using:
>
> smtp_destination_concurrency_limit = 5

smtp_destination_concurrency_limit has no effect when you turn on
smtp_destination_rate_delay. It specifies a delay BETWEEN deliveries,
meaning the deliveries cannot be in parallel.

> smtp_destination_rate_delay = 1s
> smtp_extra_recipient_limit = 5

There is no smtp_extra_recipient_limit parameter.

> these evidentially apply to the final destination, not the relay.

The rate limits apply to the next-hop destination. In order
of decreasing precedence, that is:

- the destination in main.cf:transport_maps,
- the destination in main.cf:sender_dependent_relayhost_maps,
- the destination in main.cf:default_transport,
- the destination in main.cf:sender_dependent_default_transport_maps,
- the destination in main.cf:relay_transport,
- the destination in main.cf:relayhost,
- the recipient's domain.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support
Wietse,

Thanks.  But now I am confused.  From what I understood you to say, I should be sending 1msg/s
to the next hop, but according to my logs, I am sending ~50msg/s to the relay, and I would like to
get it to under 5msg/s.


On 24 Feb 2015 at 8:55, Wietse Venema wrote:

System Support:
>
> I send several weekly newsletters where a number of the e-mails
> are all relayed to a separate host.  I would like to rate limit
> the traffic to this host.  I tried using:
>
> smtp_destination_concurrency_limit = 5

smtp_destination_concurrency_limit has no effect when you turn on
smtp_destination_rate_delay. It specifies a delay BETWEEN deliveries,
meaning the deliveries cannot be in parallel.

> smtp_destination_rate_delay = 1s
> smtp_extra_recipient_limit = 5

There is no smtp_extra_recipient_limit parameter.

> these evidentially apply to the final destination, not the relay.

The rate limits apply to the next-hop destination. In order
of decreasing precedence, that is:

- the destination in main.cf:transport_maps,
- the destination in main.cf:sender_dependent_relayhost_maps,
- the destination in main.cf:default_transport,
- the destination in main.cf:sender_dependent_default_transport_maps,
- the destination in main.cf:relay_transport,
- the destination in main.cf:relayhost,
- the recipient's domain.

        Wietse
...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Wietse Venema
System Support:
> Wietse,
>
> Thanks.  But now I am confused.  From what I understood you to
> say, I should be sending 1msg/s
> to the next hop, but according to my logs, I am sending ~50msg/s
> to the relay, and I would like to
> get it to under 5msg/s.

Then you need to follow instructions in the mailing list
welcome message.

TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail

TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html

Thank you for using Postfix.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Viktor Dukhovni
In reply to this post by System Support
On Tue, Feb 24, 2015 at 08:42:08AM -0500, System Support wrote:

> I send several weekly newsletters where a number of the e-mails are all relayed to a separate
> host.  I would like to rate limit the traffic to this host.  I tried using:
>
> smtp_destination_rate_delay = 1s

What is the name of the transport used to reach the nexthop relay?
(from the transport table entry).

What version of Postfix are you using.

And of course:

    $ postconf -n
    $ pcregrep -v '^\s*#' /etc/postfix/master.cf

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support


On 24 Feb 2015 at 17:09, Viktor Dukhovni wrote:

> On Tue, Feb 24, 2015 at 08:42:08AM -0500, System Support wrote:
>
> > I send several weekly newsletters where a number of the e-mails are all relayed to a separate
> > host.  I would like to rate limit the traffic to this host.  I tried using:
> >
> > smtp_destination_rate_delay = 1s
>
> What is the name of the transport used to reach the nexthop relay?
> (from the transport table entry).

There is no transport table

>
> What version of Postfix are you using.

2.9.6-7.4.1 on an OpenSuSE system

>
> And of course:
>
>     $ postconf -n

A couple of items in <> were masked.

INTERNAL_USERS = check_sender_access hash:$config_directory/internal_users, reject
alias_database = hash:$config_directory/aliases
alias_maps = hash:$config_directory/aliases
alternate_config_directories = /etc/postfix-relay1, /etc/postfix-in
append_at_myorigin = yes
append_dot_mydomain = yes
biff = no
command_directory = /usr/sbin
config_directory = /etc/postfix-relay1/
daemon_directory = /usr/lib/postfix
data_directory = /var/spool/postfix-relay1/var/run
default_privs = nobody
empty_address_recipient = MAILER-DAEMON
html_directory = /usr/share/doc/packages/postfix/html
inet_interfaces = localhost, falcon
inet_protocols = ipv4
local_maps = hash:$config_directory/local
local_recipient_maps = $alias_maps $local_maps
mail_name = Falcon mail server
mail_owner = postfix
mail_spool_directory = /var/mail
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
message_size_limit = 20480000
mydestination = $config_directory/local_destinations.dat
mydomain = MicroTechniques.com
myhostname = <myname>
mynetworks = $config_directory/mynetworks.dat
myorigin = <myname>
newaliases_path = /usr/bin/newaliases
queue_directory = /var/spool/postfix-relay1
readme_directory = /usr/share/doc/packages/postfix/README_FILES
recipient_delimiter = -
relayhost = <relay1>
sample_directory = /usr/share/doc/packages/postfix/samples
sendmail_path = /usr/sbin/sendmail.posfix
setgid_group = maildrop
smtp_destination_rate_delay = 1s
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:<relay1>_password
smtp_sasl_security_options = noanonymous
smtp_tls_note_starttls_offer = yes
smtp_use_tls = yes
smtpd_banner = $myhostname ESMTP
smtpd_restriction_classes = INTERNAL_USERS
syslog_facility = local3


>     $ pcregrep -v '^\s*#' /etc/postfix/master.cf


anvil              unix  -       -       n       -       1       anvil
bounce             unix  -       -       n       -       0       bounce
cleanup            unix  n       -       n       -       0       cleanup
defer              unix  -       -       n       -       0       bounce
discard            unix  -       -       n       -       -       discard
error              unix  -       -       n       -       -       error
flush              unix  n       -       n       1000?   0       flush
lmtp               unix  -       -       n       -       -       lmtp
local              unix  -       n       n       -       -       local
pickup             fifo  n       -       n       60      1       pickup
proxymap           unix  -       -       n       -       -       proxymap
qmgr               fifo  n       -       n       300     1       qmgr
relay              unix  -       -       n       -       -       smtp
retry              unix  -       -       n       -       -       error
rewrite            unix  -       -       n       -       -       trivial-rewrite
scache             unix  -       -       n       -       1       scache
showq              unix  n       -       n       -       -       showq
<local_add>:25     inet  n       -       n       -       -       smtpd
smtp               unix  -       -       n       -       -       smtp
tlsmgr             unix  -       -       n       1000?   1       tlsmgr
trace              unix  -       -       n       -       0       bounce
verify             unix  -       -       n       -       1       verify
virtual            unix  -       n       n       -       -       virtual

slow               unix  -       -       n       -       -       smtp


proxywrite unix  -       -       n       -       1       proxymap


And here is part of the log:

Feb 23 21:59:50 <hostname> postfix/smtpd[3200]: connect from <myhost>
Feb 23 21:59:50 <hostname> postfix/smtpd[3200]: 77E62139155: client=<myhost>
Feb 23 21:59:50 <hostname> postfix/cleanup[3202]: 77E62139155: message-id=<54EBE925.7293.3D5A0@<me>
Feb 23 21:59:50 <hostname> postfix/qmgr[2465]: 77E62139155: from=<me>, size=260087, nrcpt=1 (queue active)
Feb 23 21:59:50 <hostname> postfix/smtpd[3200]: disconnect from <myhost>
Feb 23 21:59:50 <hostname> postfix/cleanup[3202]: 96CF9139167: message-id=<54EBE925.7293.3D5A0@<me>
Feb 23 21:59:51 <hostname> postfix/qmgr[2465]: 96CF9139167: from=<owner-me@myhost>, size=260232, nrcpt=848 (queue active)
Feb 23 21:59:51 <hostname> postfix/local[3203]: 77E62139155: to=<??>, relay=local, delay=0.61, delays=0.12/0.01/0/0.47, dsn=2.0.0, status=sent (forwarded as 96CF9139167)
Feb 23 21:59:51 <hostname> postfix/qmgr[2465]: 77E62139155: removed
</snip>
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
Feb 23 21:59:56 <hostname> postfix/smtp[3205]: 96CF9139167: to=
</snip>



 --
> Viktor.
>

...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Wietse Venema
System Support:
> alternate_config_directories = /etc/postfix-relay1, /etc/postfix-in

Settings in this main.cf file have no effect on the postfix-relay1
or postfix-in instances.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Gerard E. Seibert
In reply to this post by System Support
On Tue, 24 Feb 2015 16:17:10 -0500, System Support stated:

> >     $ pcregrep -v '^\s*#' /etc/postfix/master.cf  

I believe that postconf -Mf would display what postfix sees in your master.cf
file. It would supply a cleaner output too.

--
Jerry
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support
Here is the postfinger output:


Postfinger - Postfix Configuration on Tue Feb 24 19:13:55 EST 2015
$Revision: 1.25 $

Warning: Postfinger output may show private configuration information,
such as ip addresses and/or domain names which you do not want to show
to the public.  If this is the case it is your responsibility to modify
the output to hide this private information.  [Remove this warning with
the --nowarn option.]

--System Parameters--
mail_version = 2.9.6
hostname = <hostname>
uname = Linux <hostname> 3.11.10-25-default #1 SMP Wed Dec 17 17:57:03 UTC 2014 (8210f77) x86_64 x86_64 x86_64 GNU/Linux

--Packaging information--
looks like this postfix comes from a RPM package: postfix-2.9.6-7.4.1.x86_64

--main.cf non-default parameters--
alias_database = hash:$config_directory/aliases
alias_maps = hash:$config_directory/aliases
biff = no
config_directory = /etc/<relay1>
daemon_directory = /usr/lib/postfix
data_directory = /var/spool/<relay1>/var/run
html_directory = /usr/share/doc/packages/postfix/html
inet_interfaces = localhost, <hostname>
inet_protocols = ipv4
INTERNAL_USERS = check_sender_access hash:$config_directory/internal_users, reject
local_maps = hash:$config_directory/local
local_recipient_maps = $alias_maps $local_maps
mail_name = <hostname> mail server
manpage_directory = /usr/share/man
message_size_limit = 20480000
mydestination = $config_directory/local_destinations.dat
mydomain = <mydomain>
myhostname = maila.<mydomain>
mynetworks = $config_directory/mynetworks.dat
myorigin = maila.<mydomain>
queue_directory = /var/spool/<relay1>
readme_directory = /usr/share/doc/packages/postfix/README_FILES
recipient_delimiter = -
relayhost = <relay-host>
sample_directory = /usr/share/doc/packages/postfix/samples
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = maildrop
smtpd_banner = $myhostname ESMTP
smtp_destination_rate_delay = 1s
smtpd_restriction_classes = INTERNAL_USERS
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/<relay1>/sasl_password
smtp_sasl_security_options = noanonymous
smtp_tls_note_starttls_offer = yes
smtp_use_tls = yes
syslog_facility = local3

--main.cf parameters defined as per defaults--

--master.cf--
anvil              unix  -       -       n       -       1       anvil
bounce             unix  -       -       n       -       0       bounce
cleanup            unix  n       -       n       -       0       cleanup
defer              unix  -       -       n       -       0       bounce
discard            unix  -       -       n       -       -       discard
error              unix  -       -       n       -       -       error
flush              unix  n       -       n       1000?   0       flush
lmtp               unix  -       -       n       -       -       lmtp
local              unix  -       n       n       -       -       local
pickup             fifo  n       -       n       60      1       pickup
proxymap           unix  -       -       n       -       -       proxymap
qmgr               fifo  n       -       n       300     1       qmgr
relay              unix  -       -       n       -       -       smtp
retry              unix  -       -       n       -       -       error
rewrite            unix  -       -       n       -       -       trivial-rewrite
scache             unix  -       -       n       -       1       scache
showq              unix  n       -       n       -       -       showq
<local-address>    inet  n       -       n       -       -       smtpd
smtp               unix  -       -       n       -       -       smtp
tlsmgr             unix  -       -       n       1000?   1       tlsmgr
trace              unix  -       -       n       -       0       bounce
verify             unix  -       -       n       -       1       verify
virtual            unix  -       n       n       -       -       virtual
slow               unix  -       -       n       -       -       smtp
proxywrite unix  -       -       n       -       1       proxymap

-- end of Postfinger output --



On 24 Feb 2015 at 16:45, Postfix User wrote:

> On Tue, 24 Feb 2015 16:17:10 -0500, System Support stated:
>
> > >     $ pcregrep -v '^\s*#' /etc/postfix/master.cf  
>
> I believe that postconf -Mf would display what postfix sees in your master.cf
> file. It would supply a cleaner output too.
>
> --
> Jerry
>

...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Viktor Dukhovni
On Tue, Feb 24, 2015 at 07:44:53PM -0500, System Support wrote:

> --main.cf non-default parameters--
> mydestination = $config_directory/local_destinations.dat
> mydomain = <mydomain>
> myhostname = maila.<mydomain>
> relayhost = <relay-host>

Is the traffic in question using the "relay" transport (destination
domain matches relay_domains, which defaults to include all
sub-domains of the domains listed in $mydestination)?

> smtp_destination_rate_delay = 1s

This applies to the "smtp" transport, but not the "relay"
or "slow" transport.

> --master.cf--
> relay              unix  -       -       n       -       -       smtp
> smtp               unix  -       -       n       -       -       smtp
> slow               unix  -       -       n       -       -       smtp

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support


On 25 Feb 2015 at 0:52, Viktor Dukhovni wrote:

> On Tue, Feb 24, 2015 at 07:44:53PM -0500, System Support wrote:
>
> > --main.cf non-default parameters--
> > mydestination = $config_directory/local_destinations.dat
> > mydomain = <mydomain>
> > myhostname = maila.<mydomain>
> > relayhost = <relay-host>
>
> Is the traffic in question using the "relay" transport (destination
> domain matches relay_domains, which defaults to include all
> sub-domains of the domains listed in $mydestination)?

The origianl e-mail is to a destination that matches a sub-domain in $mydestination and which is
aliased to an address list.  Only one of the expanded address list address matches a sub-domain
in $mydestination.

> > smtp_destination_rate_delay = 1s
>
> This applies to the "smtp" transport, but not the "relay"
> or "slow" transport.

Looking at the log, I see one e-mail handled by postfix/local, and the rest by postfix/smtp

>
> > --master.cf--
> > relay              unix  -       -       n       -       -       smtp
> > smtp               unix  -       -       n       -       -       smtp
> > slow               unix  -       -       n       -       -       smtp

...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Wietse Venema
System Support:

>
>
> On 25 Feb 2015 at 0:52, Viktor Dukhovni wrote:
>
> > On Tue, Feb 24, 2015 at 07:44:53PM -0500, System Support wrote:
> >
> > > --main.cf non-default parameters--
> > > mydestination = $config_directory/local_destinations.dat
> > > mydomain = <mydomain>
> > > myhostname = maila.<mydomain>
> > > relayhost = <relay-host>
> >
> > Is the traffic in question using the "relay" transport (destination
> > domain matches relay_domains, which defaults to include all
> > sub-domains of the domains listed in $mydestination)?
>
> The origianl e-mail is to a destination that matches a sub-domain
> in $mydestination and which is aliased to an address list.  Only
> one of the expanded address list address matches a sub-domain
> in $mydestination.

Unless you have set relay_domains, such mail will be delivered with
the "relay" transport, meaning you would need to set
relay_destination_rate_delay.

Unfortunately, this is not obvious from logging. In hindsight, that
should have been recognized and fixed long ago.

At least the relay_domains default has been fixed as of Postfix 3.0.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support


On 25 Feb 2015 at 11:33, Wietse Venema wrote:
 
> Unless you have set relay_domains, such mail will be delivered with
> the "relay" transport, meaning you would need to set
> relay_destination_rate_delay.

I set relay_destination_rate_delay = 10s

and then ran a test with an address list that had 3 addresses 1 local and 2 at gmail.  I  believe that
there should be a 10 second delay between the 2 gmail deliveries, but they were delivered
together.  Here is the log for that session ( with addresses masked):

Feb 25 12:28:16 <myhostname> postfix/master[6030]: terminating on signal 15
Feb 25 12:28:23 <myhostname> postfix/postfix-script[6901]: starting the Postfix mail system
Feb 25 12:28:23 <myhostname> postfix/master[6902]: daemon started -- version 2.9.6, configuration /etc/postfix-amazon/
Feb 25 12:28:57 <myhostname> postfix/smtpd[7110]: connect from MailServer.<mydomain>[NN.NNN.N.NN]
Feb 25 12:28:57 <myhostname> postfix/smtpd[7110]: 9BB9713923C: client=MailServer.<mydomain>[NN.NNN.N.NN]
Feb 25 12:28:57 <myhostname> postfix/cleanup[7112]: 9BB9713923C: message-id=<54EE0659.2292.2CA513@Editor.<mydomain_2>
Feb 25 12:28:57 <myhostname> postfix/qmgr[6911]: 9BB9713923C: from=<Editor@<mydomain_2>, size=780, nrcpt=1 (queue active)
Feb 25 12:28:57 <myhostname> postfix/smtpd[7110]: disconnect from MailServer.<mydomain>[NN.NNN.N.NN]
Feb 25 12:28:57 <myhostname> postfix/cleanup[7112]: B8298139240: message-id=<54EE0659.2292.2CA513@Editor.<mydomain_2>
Feb 25 12:28:57 <myhostname> postfix/qmgr[6911]: B8298139240: from=<owner-TestList@maila.<mydomain>, size=925, nrcpt=3 (queue active)
Feb 25 12:28:57 <myhostname> postfix/local[7113]: 9BB9713923C: to=<TestList@<mydomain_2>, relay=local, delay=0.37, delays=0.11/0.01/0/0.25, dsn=2.0.0, status=sent (forwarded as B8298139240)
Feb 25 12:28:57 <myhostname> postfix/qmgr[6911]: 9BB9713923C: removed
Feb 25 12:28:58 <myhostname> postfix/local[7113]: B8298139240: to=<me@mydomain_2>, relay=local, delay=0.29, delays=0.24/0/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Feb 25 12:28:58 <myhostname> postfix/smtp[7114]: B8298139240: to=<[hidden email]>, relay=email-smtp.<relayhost>, delay=1, delays=0.24/0.04/0.46/0.25, dsn=2.0.0, status=sent (250 Ok 0000014bc1c8d1d1)
Feb 25 12:28:58 <myhostname> postfix/smtp[7114]: B8298139240: to=<[hidden email]>, relay=email-smtp.<relayhost>, delay=1, delays=0.24/0.04/0.46/0.25, dsn=2.0.0, status=sent (250 Ok 0000014bc1c8d1d1)
Feb 25 12:28:58 <myhostname> postfix/qmgr[6911]: B8298139240: removed



...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Viktor Dukhovni
On Wed, Feb 25, 2015 at 12:50:29PM -0500, System Support wrote:

> and then ran a test with an address list that had 3 addresses 1 local and 2 at gmail.  I  believe that
> there should be a 10 second delay between the 2 gmail deliveries, but they were delivered
> together.  Here is the log for that session ( with addresses masked):

No, there should be no such delay because this is a single delivery
of a single message with two recipients.

> Feb 25 12:28:57 <myhostname> postfix/qmgr[6911]: B8298139240:
>   from=<owner-TestList@maila.<mydomain>, size=925, nrcpt=3 (queue active)
> Feb 25 12:28:58 <myhostname> postfix/local[7113]: B8298139240:
>   to=<me@mydomain_2>, relay=local, delay=0.29, delays=0.24/0/0/0.04,
>   dsn=2.0.0, status=sent (delivered to mailbox)
> Feb 25 12:28:58 <myhostname> postfix/smtp[7114]: B8298139240:
>   to=<[hidden email]>, relay=email-smtp.<relayhost>,
>   delay=1, delays=0.24/0.04/0.46/0.25, dsn=2.0.0, status=sent
>   (250 Ok 0000014bc1c8d1d1)
> Feb 25 12:28:58 <myhostname> postfix/smtp[7114]: B8298139240:
>   to=<[hidden email]>, relay=email-smtp.<relayhost>,
>   delay=1, delays=0.24/0.04/0.46/0.25, dsn=2.0.0, status=sent
>   (250 Ok 0000014bc1c8d1d1)
> Feb 25 12:28:58 <myhostname> postfix/qmgr[6911]: B8298139240: removed

Note the identical smtp[7114] pids, with identical queue-id, delays
down to .01s and the same remote server queue id.

You're not reading your logs correctly.  Note also that your previous
50/s is likely due to the default recipient limit, and we've just
been chasing ghosts.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support


On 25 Feb 2015 at 18:22, Viktor Dukhovni wrote:

> On Wed, Feb 25, 2015 at 12:50:29PM -0500, System Support wrote:
>
> > and then ran a test with an address list that had 3 addresses 1 local and 2 at gmail.  I  believe that
> > there should be a 10 second delay between the 2 gmail deliveries, but they were delivered
> > together.  Here is the log for that session ( with addresses masked):
>
> No, there should be no such delay because this is a single delivery
> of a single message with two recipients.
>
> > Feb 25 12:28:57 <myhostname> postfix/qmgr[6911]: B8298139240:
> >   from=<owner-TestList@maila.<mydomain>, size=925, nrcpt=3 (queue active)
> > Feb 25 12:28:58 <myhostname> postfix/local[7113]: B8298139240:
> >   to=<me@mydomain_2>, relay=local, delay=0.29, delays=0.24/0/0/0.04,
> >   dsn=2.0.0, status=sent (delivered to mailbox)
> > Feb 25 12:28:58 <myhostname> postfix/smtp[7114]: B8298139240:
> >   to=<[hidden email]>, relay=email-smtp.<relayhost>,
> >   delay=1, delays=0.24/0.04/0.46/0.25, dsn=2.0.0, status=sent
> >   (250 Ok 0000014bc1c8d1d1)
> > Feb 25 12:28:58 <myhostname> postfix/smtp[7114]: B8298139240:
> >   to=<[hidden email]>, relay=email-smtp.<relayhost>,
> >   delay=1, delays=0.24/0.04/0.46/0.25, dsn=2.0.0, status=sent
> >   (250 Ok 0000014bc1c8d1d1)
> > Feb 25 12:28:58 <myhostname> postfix/qmgr[6911]: B8298139240: removed
>
> Note the identical smtp[7114] pids, with identical queue-id, delays
> down to .01s and the same remote server queue id.
>
> You're not reading your logs correctly.  Note also that your previous
> 50/s is likely due to the default recipient limit, and we've just
> been chasing ghosts.

OK. Is there any way to rate limit the traffic to the relay for this use case?
I tried reducing the default_destination_recipient_limit to 1 along with the relay and smtp versions.


...don

support (at) microtechniques.com
  
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Viktor Dukhovni
On Wed, Feb 25, 2015 at 02:46:29PM -0500, System Support wrote:

> > No, there should be no such delay because this is a single delivery
> > of a single message with two recipients.
> >
> > Note the identical smtp[7114] pids, with identical queue-id, delays
> > down to .01s and the same remote server queue id.
> >
> > You're not reading your logs correctly.  Note also that your previous
> > 50/s is likely due to the default recipient limit, and we've just
> > been chasing ghosts.
>
> OK. Is there any way to rate limit the traffic to the relay for this use case?
> I tried reducing the default_destination_recipient_limit to 1 along with the relay and smtp versions.

You seem to not be paying attention.  The deliveries *are* rate-
limited.  You're reporting a *single* delivery.  Naturally it is
not possible to rate limit just one delivery, rate limits apply
to multiple things that happen one after the other...

You need to send multiple messages to see evidence of rate limiting.

    Email transaction (with 50 recipients):

        EHLO <client>
        MAIL FROM:<sender>
        RCPT TO:<rcpt1>
        RCPT TO:<rcpt2>
        ...
        RCPT TO:<rcpt50>
        DATA
        From: <sender>
        To: <some-list>
        Subject: Announcing rate limits on things that happen once.

        This single message has been rate limited, it will never
        happen again
        .

The above outlines a single message forwarded atomically to an SMTP
server.  If the client were Postfix, it would log one log entry
for each of the 50 recipient addresses, recording that each was
delivered (with otherwise identical log information).

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Wietse Venema
In reply to this post by System Support
System Support:
Content-Description: Mail message body

> > Note the identical smtp[7114] pids, with identical queue-id, delays
> > down to .01s and the same remote server queue id.
> >
> > You're not reading your logs correctly.  Note also that your previous
> > 50/s is likely due to the default recipient limit, and we've just
> > been chasing ghosts.
>
> OK. Is there any way to rate limit the traffic to the relay for
> this use case?  I tried reducing the default_destination_recipient_limit
> to 1 along with the relay and smtp versions.

If you set this to 1, you rate-limit per-recipient instead of
per-domain (this is not a secret: anyone can RTFM).

By setting it to a small number you actually create MORE work
for the relay MTA than when you leave it at the default.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Viktor Dukhovni
On Wed, Feb 25, 2015 at 02:58:21PM -0500, Wietse Venema wrote:

> > OK. Is there any way to rate limit the traffic to the relay for
> > this use case?  I tried reducing the default_destination_recipient_limit
> > to 1 along with the relay and smtp versions.
>
> If you set this to 1, you rate-limit per-recipient instead of
> per-domain (this is not a secret: anyone can RTFM).
>
> By setting it to a small number you actually create MORE work
> for the relay MTA than when you leave it at the default.

In other words, leave the default recipient limit in place, it is
helping you to send fewer messages to the list by batching multiple
recipients in a single delivery.  The rate limits are working
exactly as documented.  You're just not understanding your logs.

If you think you need to rate limit recipients rather than messages,
please explain why that would be wise.  Typically, such a goal is
the result of some sort of misunderstanding or confusion.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

System Support
In reply to this post by Viktor Dukhovni


On 25 Feb 2015 at 19:56, Viktor Dukhovni wrote:

> On Wed, Feb 25, 2015 at 02:46:29PM -0500, System Support wrote:
>
> > > No, there should be no such delay because this is a single delivery
> > > of a single message with two recipients.
> > >
> > > Note the identical smtp[7114] pids, with identical queue-id, delays
> > > down to .01s and the same remote server queue id.
> > >
> > > You're not reading your logs correctly.  Note also that your previous
> > > 50/s is likely due to the default recipient limit, and we've just
> > > been chasing ghosts.
> >
> > OK. Is there any way to rate limit the traffic to the relay for this use case?
> > I tried reducing the default_destination_recipient_limit to 1 along with the relay and smtp versions.
>
> You seem to not be paying attention.  The deliveries *are* rate-
> limited.  You're reporting a *single* delivery.  Naturally it is
> not possible to rate limit just one delivery, rate limits apply
> to multiple things that happen one after the other...

Yes, I understand that this is not the correct way to limit the traffic to the relay host, but my
question is how can I do that - rate limit traffic to a relay host that is generated by a Postfix
address list.  Having to write an application/script to send the same message a 1,000 times, once
to each user, instead of using an address list, seems silly.

The reason for working on this is because the relay host is complaining about excessive traffic
(>5msg/s) and freezing the connection after the first 50 deliveries, so apparently they do not
consider it a single message:

<snip after 50 successful deliveries>

Feb 23 22:00:01 <myhost> postfix/smtp[3205]: 96CF9139167: host email-smtp.<relayhost> said: 454 Throttling failure: Maximum sending rate exceeded. (in reply to end of DATA command)
Feb 23 22:00:04 <myhost> postfix/smtp[3205]: 96CF9139167: to=<[hidden email]>, relay=email-smtp.<relayhost>, delay=12, delays=0.47/6/4.5/1.4, dsn=4.0.0, status=deferred (host email-smtp.<relayhost> said: 454 Throttling failure: Maximum sending rate exceeded. (in reply to end of DATA command))

<snip a 1,000 or so similar errors follow>

And, I have spent quite a bit of time with the FM and other forums with Postfix configuration
suggestions.  

Having the error occur after the first 50 e-mails,

and reading this:

> No, there should be no such delay because this is a single delivery of a single message with
two recipients.

and this:
>  Note also that your previous > 50/s is likely due to the default recipient limit, and we've just
been chasing ghosts.

and this in the manual with a default of 50:
default_destination_recipient_limit (default: 50)

    The default maximal number of recipients per message delivery.

I think it was reasonable to try reducing the default_destination_recipient_limit from 50.  I did try
several values between 50 and 1.


...don

support (at) microtechniques.com

Reply | Threaded
Open this post in threaded view
|

Re: Rate limiting

Wietse Venema
System Support:
> The reason for working on this is because the relay host is complaining about excessive traffic
> (>5msg/s) and freezing the connection after the first 50 deliveries, so apparently they do not

Use "whatever_rate_delay = 1" and do NOT set the
whatever_destination_recipient_limit to 1.

That will send one MESSAGE per second to the destination.  That is
one message per second, NOT one recipient per second, so if you
look at your logs, you need to take that into account.

If you must send one recipient per delivery to a rate-limited
server, see http://www.postfix.org/VERP_README.html 

I will surely regret this, because VERP_README assumes that the
reader understands the difference between messages and recipients.

        Wietse