Limiting global number of outgoing connections

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Limiting global number of outgoing connections

Lorenzo Milesi-2
Hi.

I need to limit the maximum number of outgoing SMTP connections done by Postfix for delivering messages.
Our VPS provider is limiting to 5 conns/s, so I need Postfix not to open more than 5 connections to remote SMTP servers.

I cannot use config param *destination_concurrency_limit because they’re related to a single recipient domain. So I was pointed to master.cf, where I could limit the maximum number of “smtp” processes.

So I changed the smtp (or what I think it is) line as follows:

smtp unix - - y - 5 smtp

Is this the correct approach? I’m asking because it didn’t fully work. I delivered a newsletter today and I still got blocked by ISP.

Thanks
maxxer
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

John Fawcett
On 03/01/2019 11:23, Lorenzo Milesi wrote:

> Hi.
>
> I need to limit the maximum number of outgoing SMTP connections done by Postfix for delivering messages.
> Our VPS provider is limiting to 5 conns/s, so I need Postfix not to open more than 5 connections to remote SMTP servers.
>
> I cannot use config param *destination_concurrency_limit because they’re related to a single recipient domain. So I was pointed to master.cf, where I could limit the maximum number of “smtp” processes.
>
> So I changed the smtp (or what I think it is) line as follows:
>
> smtp unix - - y - 5 smtp
>
> Is this the correct approach? I’m asking because it didn’t fully work. I delivered a newsletter today and I still got blocked by ISP.
>
> Thanks
> maxxer

You're right that you can't use *destination_concurrency_limit to limit
overall outgoing connection rate, but neither will limiting the number
of processes to 5 in master.cf work. You have a maximum of 5 processes
but nothing which stops each of them opening connections as fast as they
can. There are policy servers available for rate limiting, but I think
they also will not help since you want to slow things down, not reject
when over the limit. The only thing I can suggest is to slow down the
input rate of email.

This kind of outgoing email connection rate limiting seems incompatible
with the use you're making of the VPS service, unless you've got small
lists. Whereas in theory a single connection could deliver multiple
emails and emails with multiple recipients, you are not going to be able
to easily control how email is queued and delivered so you will probably
have to go as slow as 5 recipients per second.

John

Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Wietse Venema
In reply to this post by Lorenzo Milesi-2
Lorenzo Milesi:
> Hi.
>
> I need to limit the maximum number of outgoing SMTP connections done by Postfix for delivering messages.
> Our VPS provider is limiting to 5 conns/s, so I need Postfix not to open more than 5 connections to remote SMTP servers.

No, you need to RATE-LIMIT, not volume-limit.

smtp_destination_rate_delay = 1

is as fast as it will go, because Postfix does not support
rate delays < 1s.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Lorenzo Milesi-2
> smtp_destination_rate_delay = 1

I tried this, removed the limit on the number of processes, restarted postfix and flushed the queue but I still got a lot of connections throttled (blocked) by ISP’s firewall.
Shouldn’t I also add
smtp_destination_concurrency_limit = 5
?
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Wietse Venema
Lorenzo Milesi:
> > smtp_destination_rate_delay = 1
>
> I tried this, removed the limit on the number of processes, restarted postfix and flushed the queue but I still got a lot of connections throttled (blocked) by ISP?s firewall.
> Shouldn?t I also add
> smtp_destination_concurrency_limit = 5

You say that the ISP allow FIVE connection/second

With "smtp_destination_rate_delay = 1", Postfix makes no more than
ONE conection per second.

If the ISP keeps blocking, then either the ISP lies about their
policy, or you misunderstood their policy.

You can easily verify in Postfix logs that smtp_destination_rate_delay
works.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Benny Pedersen-2
In reply to this post by Lorenzo Milesi-2
Lorenzo Milesi skrev den 2019-01-03 14:19:
>> smtp_destination_rate_delay = 1
>
> I tried this, removed the limit on the number of processes, restarted
> postfix and flushed the queue but I still got a lot of connections
> throttled (blocked) by ISP’s firewall.
> Shouldn’t I also add
> smtp_destination_concurrency_limit = 5
> ?

show logs would be more helpfull now i think

and possible postconf -n ?

or does maillist now have crystall balls in 2019 ? :=)
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Lorenzo Milesi-2
In reply to this post by Wietse Venema
You can easily verify in Postfix logs that smtp_destination_rate_delay
works.

I must have done something wrong because it does not seem to behave correctly.

I’ve added this config to main.cf:

smtp_destination_concurrency_limit = 4
smtp_destination_rate_delay = 1s
libero_destination_concurrency_limit = 2
libero_destination_rate_delay = 6s
alice_destination_concurrency_limit = 1
alice_destination_rate_delay = 7s
turtle_destination_concurrency_limit = 2
turtle_destination_rate_delay = 6s

the first two are for this thread, while the following are custom transport for specific domains. in master.cf I’ve restored the standard smtp transport, with the additional ones above.
When I flush the queue for the exact same time (15:47:16) I’m getting 15 attempted connections:

Jan  3 15:47:16 www postfix/smtp[7684]: connect to mail.testowe24.pl[151.80.202.137]:25: Connection refused
Jan  3 15:47:16 www postfix/smtp[7684]: 8BC90602808: to=<[hidden email]>, relay=none, delay=19636, delays=19636/0.07/0.03/0, dsn=4.4.1, status=deferred (connect to mail.testowe24.pl[151.80.202.137]:25: Connection refused)
Jan  3 15:47:16 www postfix/smtp[7684]: 891FB602AD2: to=<[hidden email]>, relay=none, delay=19383, delays=19383/0.11/0.01/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=cbdolejki.pl type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7698]: connect to blertyd.eu[198.37.114.34]:25: Connection refused
Jan  3 15:47:16 www postfix/smtp[7698]: 40E52602A5A: to=<[hidden email]>, relay=none, delay=19389, delays=19389/0.03/0.33/0, dsn=4.4.1, status=deferred (connect to blertyd.eu[198.37.114.34]:25: Connection refused)
Jan  3 15:47:16 www postfix/smtp[7691]: 290A3602C20: host ksg.ovh[2001:41d0:8:6f14::1] said: 454 4.7.1 <[hidden email]>: Relay access denied (in reply to RCPT TO command)
Jan  3 15:47:16 www postfix/smtp[7705]: C6A14602B99: to=<[hidden email]>, relay=tepgames.com[212.129.45.37]:25, delay=19329, delays=19329/0.04/0.22/0, dsn=4.4.2, status=deferred (lost connection with tepgames.com[212.129.45.37] while receiving the initial server greeting)
Jan  3 15:47:16 www postfix/smtp[7690]: 84507602EA2: to=<[hidden email]>, relay=none, delay=19195, delays=19194/0.11/0.59/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=maastera.info type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7680]: 37357602E3E: to=<[hidden email]>, relay=none, delay=19197, delays=19197/0.04/0.72/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=ruks.info type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7681]: 3E102602E50: to=<[hidden email]>, relay=none, delay=19196, delays=19196/0.04/0.72/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=lupiko.info type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7698]: connect to gigsme.com[46.166.189.98]:25: Connection refused
Jan  3 15:47:16 www postfix/smtp[7698]: D49CC6029B8: to=<[hidden email]>, relay=none, delay=19504, delays=19504/0.02/0.16/0, dsn=4.4.1, status=deferred (connect to gigsme.com[46.166.189.98]:25: Connection refused)
Jan  3 15:47:16 www postfix/smtp[7706]: CF54E602725: to=<[hidden email]>, relay=smtp.aliceposta.it[82.57.200.133]:25, delay=19756, delays=19756/0.02/0.09/0.22, dsn=2.0.0, status=sent (250 <5B8F9E5B1C3DBF1D> Mail accepted)
Jan  3 15:47:16 www postfix/smtp[7686]: 883EF602DF3: to=<[hidden email]>, relay=none, delay=19196, delays=19195/0.08/0.7/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=nulki.info type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7684]: 850AB602A65: to=<[hidden email]>, relay=none, delay=19385, delays=19384/0.12/0.66/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=mailbox.co.pl type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7699]: E63A46029A5: to=<[hidden email]>, relay=none, delay=19505, delays=19504/0.02/0.56/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=przedszkole-pruszkow.pl type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7691]: 290A3602C20: to=<[hidden email]>, relay=ksg.ovh[37.59.52.20]:25, delay=19316, delays=19315/0.08/0.72/0.03, dsn=4.7.1, status=deferred (host ksg.ovh[37.59.52.20] said: 454 4.7.1 <[hidden email]>: Relay access denied (in reply to RCPT TO command))
Jan  3 15:47:16 www postfix/smtp[7700]: EF46B602D05: to=<[hidden email]>, relay=none, delay=19267, delays=19266/0.04/0.62/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=miagency.co.pl type=MX: Host not found, try again)
Jan  3 15:47:16 www postfix/smtp[7704]: C5BF2602947: to=<[hidden email]>, relay=none, delay=19498, delays=19498/0.03/0.61/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=emailvip.co.pl type=MX: Host not found, try again)

Did I do something wrong or misunderstood the behaviour?
thanks
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Lorenzo Milesi-2
In reply to this post by Benny Pedersen-2
Il giorno 3 gen 2019, alle ore 15:43, Benny Pedersen <[hidden email]> ha scritto:

and possible postconf -n ?
or does maillist now have crystall balls in 2019 ? :=)

oups, agree :)

Full config here:
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Wietse Venema
In reply to this post by Lorenzo Milesi-2
Lorenzo Milesi:
> Jan  3 15:47:16 www postfix/smtp[7684]: connect to mail.testowe24.pl[151.80.202.137]:25: Connection refused
> Jan  3 15:47:16 www postfix/smtp[7684]: 8BC90602808: to=<[hidden email]>, relay=none, delay=19636, delays=19636/0.07/0.03/0, dsn=4.4.1, status=deferred (connect to mail.testowe24.pl[151.80.202.137]:25: Connection refused)
> Jan  3 15:47:16 www postfix/smtp[7684]: 891FB602AD2: to=<[hidden email]>, relay=none, delay=19383, delays=19383/0.11/0.01/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=cbdolejki.pl type=MX: Host not found, try again)
> Jan  3 15:47:16 www postfix/smtp[7698]: connect to blertyd.eu[198.37.114.34]:25: Connection refused
> Jan  3 15:47:16 www postfix/smtp[7698]: 40E52602A5A: to=<[hidden email]>, relay=none, delay=19389, delays=19389/0.03/0.33/0, dsn=4.4.1, status=deferred (connect to blertyd.eu[198.37.114.34]:25: Connection refused)

The Postfix DELIVERY_rate_delay features rate-limit DELIVERIES.
None of the above attempts count towards the delivery rate limit,
because there are no email deliveries.

You need to rate-limit TCP handshakes. You will need to configure
traffic shaping inside your TCP/IP stack. The TCP stack is not part
of Postfix.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Viktor Dukhovni
In reply to this post by Lorenzo Milesi-2


> On Jan 3, 2019, at 5:23 AM, Lorenzo Milesi <[hidden email]> wrote:
>
> I need to limit the maximum number of outgoing SMTP connections done by Postfix for delivering messages.
> Our VPS provider is limiting to 5 conns/s, so I need Postfix not to open more than 5 connections to remote SMTP servers.

You're asking for a Postfix feature that does not exist.  For your
use-case to work, the Postfix queue manager would need to put all
mail in a single in memory queue, regardless of the destination
nexthop.  With:

   mumble_destination_recipient_limit = 1

Postfix create fragments the queue by (nexthop, recipient) rather
than just the nexthop.  You're looking for a feature that ignores
both the nexthop and the recipient aggregates all nexthops into a
single unified transport-wide queue.  No code for that exists.

Frankly, running a bulk mail MTA from a service that only allows
5 connections/sec is probably in violation of the service terms,
and in any case much too tight a straightjacket for Postfix,
which is written to be secure *and* performant.  There's only so
much support for extreme measures to make Postfix slow.

So for now my best advice is: https://dilbert.com/strip/1995-06-24

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Viktor Dukhovni
On Thu, Jan 03, 2019 at 02:57:34PM -0500, Viktor Dukhovni wrote:

> Postfix fragments the queue by (nexthop, recipient) rather
> than just the nexthop.  You're looking for a feature that ignores
> both the nexthop and the recipient aggregates all nexthops into a
> single unified transport-wide queue.  No code for that exists.

If you're willing run your own franken-postfix, hard coded to run
one queue per transport whenever the recipient limit > 1, you could
test the below.  I have no time to create a documented configurable
(mis)feature along these lines.

--- src/qmgr/qmgr_message.c
+++ src/qmgr/qmgr_message.c
@@ -1259,7 +1259,7 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message)
  * to decouple the per-destination recipient limit from the
  * per-destination concurrency.
  */
- vstring_strcpy(queue_name, STR(reply.nexthop));
+ vstring_strcpy(queue_name, "global_queue");
  if (strcmp(transport->name, MAIL_SERVICE_ERROR) != 0
     && strcmp(transport->name, MAIL_SERVICE_RETRY) != 0
     && transport->recipient_limit == 1) {

Note that with this, connection failures to any single destination,
will throttle all destinations.  Since you in any case  want rate
delays, see the note about

    http://www.postfix.org/postconf.5.html#transport_destination_concurrency_failed_cohort_limit

mentioned in:

    http://www.postfix.org/postconf.5.html#default_destination_rate_delay

Therefore:

    smtp_destination_rate_delay = 1s
    default_destination_concurrency_failed_cohort_limit = 10

And yet, this is all still a bad idea.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Benny Pedersen-2
In reply to this post by Viktor Dukhovni
Viktor Dukhovni skrev den 2019-01-03 20:57:

> So for now my best advice is: https://dilbert.com/strip/1995-06-24

dilbert is funny solution on the vps have limited dns resolvers, not a
postfix problem
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Wietse Venema
In reply to this post by Viktor Dukhovni
Viktor Dukhovni:

> On Thu, Jan 03, 2019 at 02:57:34PM -0500, Viktor Dukhovni wrote:
>
> > Postfix fragments the queue by (nexthop, recipient) rather
> > than just the nexthop.  You're looking for a feature that ignores
> > both the nexthop and the recipient aggregates all nexthops into a
> > single unified transport-wide queue.  No code for that exists.
>
> If you're willing run your own franken-postfix, hard coded to run
> one queue per transport whenever the recipient limit > 1, you could
> test the below.  I have no time to create a documented configurable
> (mis)feature along these lines.

He needs a SYN rate limiter.

The firewall blocks even when connections are not completed.
How do you propose to rate-limit SYN from within Postfix?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Viktor Dukhovni
On Thu, Jan 03, 2019 at 07:04:53PM -0500, Wietse Venema wrote:

> > If you're willing run your own franken-postfix, hard coded to run
> > one queue per transport whenever the recipient limit > 1, you could
> > test the below.  I have no time to create a documented configurable
> > (mis)feature along these lines.
>
> He needs a SYN rate limiter.

I was under the impression that there's no code in the queue manager
that distinguishes between successful and failed delivery attempts
when it comes to rate delays.  The feedback channel from the delivery
agent to the queue manager is intentionally minimal, all I see the
queue manager learning is:

    * Destination ok, apply positive feedback, keep going after
      any rate delay

    * Destination down, apply negative feedback and
      possibly throttle the queue.  Otherwise, keep going (seemingly
      also after any rate delay).

    * Delivery agent broken, throttle the transport.

The only visible side effect of a completed delivery without a crash
is a change in the queue's busy count, and a change in the queue's
concurrency window.  I don't see any way for the scheduler to
distinguish between completed deliveries and connection failure.

Likely I am missing some key insight, please pardon any confusion...

So, if destination down avoids rate delay, wouldn't that mean that
with the destination_concurrency_failed_cohort_limit > 1, throttling
may leave the transport ready for another delivery without incurring
a rate delay pause?

I don't think that'd be a feature, throttling may result from
connection failure, or a "[45]XX" code in the banner or EHLO reply,
or a lost connection, and I would expect that this a reason to skip
delaying the next delivery?

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Wietse Venema
Viktor Dukhovni:
> Likely I am missing some key insight, please pardon any confusion...

The ISP provides a budget of 5 connections/second, enforced at the
firewall level. The OP's logging shows ECONNREFUSED when the Postfix
scheduler rate limits to <1 delivery request per second. How can
that be?

For one delivery request, the Postfix SMTP client gets up to
five ($smtp_mx_address_limit) addresses from DNS, and burns through
those addresses in a split second as they fail with ECONNREFUSED.
Instantly, he has used up his entire 5 connections/second budget
with only one delivert request.

Obviously, the queue scheduler has no idea how many 'connections'
are being used up per delivery request. No amount of scheduler
tweaking will change that.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Viktor Dukhovni
> On Jan 4, 2019, at 7:23 AM, Wietse Venema <[hidden email]> wrote:
>
> For one delivery request, the Postfix SMTP client gets up to
> five ($smtp_mx_address_limit) addresses from DNS, and burns through
> those addresses in a split second as they fail with ECONNREFUSED.
> Instantly, he has used up his entire 5 connections/second budget
> with only one delivert request.
>
> Obviously, the queue scheduler has no idea how many 'connections'
> are being used up per delivery request. No amount of scheduler
> tweaking will change that.

Well for that we have smtp_mx_address_limit, which could be
reduced from 5 to 2 (or perhaps 3).  That does risk problems
delivering to sites whose primary MX has multiple IPs and is
fake, with the real MX at worse priority, but such sites clearly
don't really want to receive much mail... :-(

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Limiting global number of outgoing connections

Lorenzo Milesi-2
In reply to this post by Viktor Dukhovni
Thanks everyone for the help (and for the fun).

After “arguing” with support it ended up not even being 5 SYN/sec, but some random number decided by the firewall limiter, thus making any configuration tweak ineffective. 
I’ve agreed a different solution with the ISP, so now Postfix is happily delivering.

Thanks everyone again.

Il giorno 3 gen 2019, alle ore 20:57, Viktor Dukhovni <[hidden email]> ha scritto:

So for now my best advice is: https://dilbert.com/strip/1995-06-24