Connection Caching Per-Destination

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Connection Caching Per-Destination

Wietse Venema
Greg Sims:
> I know this is likely simplistic thinking -- but how about this in master.cf:
>
> outlook  unix  -       -       n       -       -       smtp
>   -o syslog_name=outlook
>   -o smtp_connection_cache_on_demand=yes
>   -o smtp_max_connections=8

There is no smtp_max_connections feature.

I suspect the real problem was that hundreds of domains were not
directed to the low-concurrency 'outlook' transport, and that
connection count 'overshoot' due to unused cached connections was
a red herring.

There is a rough idea of how to enforce strict connection counts
when connection caching is turned on. But it would not help in your
case, where the number of competing domains is 100x the number of
allowed concurrent connections. Under those conditions the feature
would behave as if connection reuse is turned off.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Connection Caching Per-Destination

Greg Sims
> I suspect the real problem was that hundreds of domains were not
> directed to the low-concurrency 'outlook' transport, and that
> connection count 'overshoot' due to unused cached connections was
> a red herring.

Please recall that I collected 383 email domains into
transport.outlook.regexp.  I confirmed that all traffic going to
outlook.com mx servers were going through the outlook: transport.  I
can not speak to the 'overshoot' issue.

I have a set of outlook.com associated logs where (1) the outlook:
transport was limited to 4 processes, (2) our email arrival rate was
500/minute and (3) outlook.com servers complained of MaxConnections.
I would be glad to share this set of logs in hopes of answering the
question: Is it rate limitations or connection limitations that causes
outlook to issue MaxConnections?  I can not share this log here due to
size and the email addresses it contains.  Please let me know if there
is an email address I should send these logs to.

PS.  I also have configured the following in main.cf:

smtpd_recipient_restrictions = check_recipient_mx_access
regexp:/etc/postfix/mx_access.regexp

as a way to get all outlook.com traffic to flow thru the outlook:
transport.  More on this later.

Thanks, Greg
www.RayStedman.org
Reply | Threaded
Open this post in threaded view
|

Re: Connection Caching Per-Destination

Wietse Venema
Greg Sims:
> > I suspect the real problem was that hundreds of domains were not
> > directed to the low-concurrency 'outlook' transport, and that
> > connection count 'overshoot' due to unused cached connections was
> > a red herring.
>
> Please recall that I collected 383 email domains into
> transport.outlook.regexp.  I confirmed that all traffic going to
> outlook.com mx servers were going through the outlook: transport.  I

This was after a bit of armtwisting from my end, and I recall this
was the first (and only) action that allowed you to significantly
increase the email volume.

> I have a set of outlook.com associated logs where (1) the outlook:
> transport was limited to 4 processes, (2) our email arrival rate was
> 500/minute and (3) outlook.com servers complained of MaxConnections.
> I would be glad to share this set of logs in hopes of answering the
> question: Is it rate limitations or connection limitations that causes
> outlook to issue MaxConnections?  I can not share this log here due to
> size and the email addresses it contains.  Please let me know if there
> is an email address I should send these logs to.

I decline. This thing has already cost way too much of my time.

> PS.  I also have configured the following in main.cf:
>
> smtpd_recipient_restrictions = check_recipient_mx_access
> regexp:/etc/postfix/mx_access.regexp
>
> as a way to get all outlook.com traffic to flow thru the outlook:
> transport.  More on this later.

This would avoid the need for hundreds of transport map entries,
and would avoid the need keep adding/removing entries as cusomers host
their email at outlook, or decide to take their business elsewhere.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Connection Caching Per-Destination

@lbutlr
On 02 Aug 2020, at 15:51, Wietse Venema <[hidden email]> wrote:
> This would avoid the need for hundreds of transport map entries,
> and would avoid the need keep adding/removing entries as cusomers host
> their email at outlook, or decide to take their business elsewhere.

Automatically adding the outlook domains doesn’t seem like it is hard to do, but removing domains that are not longer on outlook.com isa more difficult problem, and makes the regex solution untenable for active servers, I would think.

Either that or it takes much more parsing of the logs, and at some point that becomes very expensive.

--
You try to shape the world to what you want the world to be. Carving
your name a thousand times won't bring you back to me. Oh no, no
I might as well go and tell it to the trees. Go and tell it to
the trees, yeah.


Reply | Threaded
Open this post in threaded view
|

Re: Connection Caching Per-Destination

Wietse Venema
In reply to this post by Wietse Venema
Wietse Venema:
> There is a rough idea of how to enforce strict connection counts
> when connection caching is turned on. But it would not help in your
> case, where the number of competing domains is 100x the number of
> allowed concurrent connections. Under those conditions the feature
> would behave as if connection reuse is turned off.

An easier fix may be to change the scheduler and suspend round-robin
destination selection until a destination is blocked (concurrency
limit reached) or drained.

This would be less fair in the general case of mixed email flows,
but mass email is different enough that people can expect to need
some confguration for fastest delivery.

        Wietse
12