Outlook.com Max Connections

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Outlook.com Max Connections

Greg Sims
I have looked at a number of maillogs where we receive the 'exceeded
the maximum number of connections' error from the outlook servers.
The following is very telling.  The first nine are reformatted
'status=sent' records followed by a 'status=deferred' from outlook.  I
obscured the email address and the ip address in the deferred message.

  104 - Aug 21 01:35:37, r238,               hotmail.com, relay=
104.47.41.33, delay=    0.70, delays=  0.01 /   0.00 /   0.04 /   0.65
  105 - Aug 21 01:35:37, r235,                outlook.es, relay=
104.47.12.33, delay=    2.30, delays=  0.01 /   0.00 /   0.59 /   1.70
  106 - Aug 21 01:35:38, r236,               hotmail.com, relay=
104.47.41.33, delay=    0.38, delays=  0.01 /   0.00 /   0.04 /   0.33
  107 - Aug 21 01:35:38, r237,               hotmail.com, relay=
104.47.41.33, delay=    0.43, delays=  0.01 /   0.00 /   0.06 /   0.36
  108 - Aug 21 01:35:38, r238,               hotmail.com, relay=
104.47.41.33, delay=    0.43, delays=  0.01 /   0.00 /   0.04 /   0.38
  109 - Aug 21 01:35:39, r236,               hotmail.com, relay=
104.47.41.33, delay=    0.38, delays=  0.01 /   0.00 /   0.04 /   0.33
  110 - Aug 21 01:35:39, r238,               hotmail.com, relay=
104.47.41.33, delay=    0.52, delays=  0.01 /   0.00 /   0.04 /   0.47
  111 - Aug 21 01:35:39, r238,               hotmail.com, relay=
104.47.41.33, delay=    0.67, delays=  0.01 /   0.00 /   0.04 /   0.62
  112 - Aug 21 01:35:39, r238,                outlook.es, relay=
104.47.12.33, delay=    1.15, delays=  0.01 /   0.00 /   0.31 /   0.83

Aug 21 01:35:40 mail0.raystedman.org r238/smtp[174988]: 12D16CA9684:
to=<[hidden email]>,
relay=hotmail-com.olc.protection.outlook.com[104.47.41.33]:25,
delay=0.07, delays=0.01/0.01/0.04/0.02, dsn=4.7.652, status=deferred
(host hotmail-com.olc.protection.outlook.com[104.47.41.33] said: 451
4.7.652 The mail server [xxx.xxx.xxx.238] has exceeded the maximum
number of connections. (S3115)
[DM3NAM03FT010.eop-NAM03.prod.protection.outlook.com] (in reply to
MAIL FROM command))

In this case we have only sent 112 emails when we received the
deferred record.  You can also see that the r238 transport was
selected from our randmap a number of times -- which is a valid random
sequence.   This entire run sent 7.2K emails to outlook and received
only 8 Max Connection status=deferred records. I conclude that outlook
is actually complaining about the number of connections that are in
use at any point in time.

sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}

selects transports at random for delivering email.  Is it possible to
schedule a set of transports using a round-robin discipline? This
transport scheduling strategy may avoid the random sequence above
which caused outlook to complain.

Thanks, Greg
www.RayStedman.org
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Wietse Venema
Greg Sims:
> sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}
>
> selects transports at random for delivering email.  Is it possible to
> schedule a set of transports using a round-robin discipline? This

man 5 tcp_table
man 5 socketmap_table

and implement a server that responds in round-sobin order.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Viktor Dukhovni
In reply to this post by Greg Sims
On Sat, Aug 22, 2020 at 08:11:41AM -0700, Greg Sims wrote:

> I have looked at a number of maillogs where we receive the 'exceeded
> the maximum number of connections' error from the outlook servers.
> [...]
> I conclude that outlook is actually complaining about the number of
> connections that are in use at any point in time.

That's good to know.

> sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}
>
> selects transports at random for delivering email.  Is it possible to
> schedule a set of transports using a round-robin discipline? This
> transport scheduling strategy may avoid the random sequence above
> which caused outlook to complain.

To unconditionally avoid max connections per IP, ensure that each of the
IP-specific transports dedicated to outlook.com, hotmail.com, et. al.
have a transport *process limit* below the number that triggers the
limit, *and* avoid connection reuse, which can result in idle
connections in the connection cache adding to the live connection
count from active delivery agents.

Once no transport is capable of exceeding the limit, it won't matter
whether the scheduling is random or round-robin.

Given sufficiently high or variable message-delivery latency round-robin
can't help.  All the transports will be fully saturated at peak load,
with a backlog of messages in all their queues.  Or a batch of high
latency messages that happen to hit a particular transport will cause a
spike in its concurrency relative to the others, even with round-robin
selection.

So I wouldn't bother with working to implement round-robin, it is I
think a waste of time.  If not crossing the limit is important, make
sure you can't cross the limit regardless of the transport selection
order.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Wietse Venema
In reply to this post by Wietse Venema
Wietse Venema:

> Greg Sims:
> > sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}
> >
> > selects transports at random for delivering email.  Is it possible to
> > schedule a set of transports using a round-robin discipline? This
>
> man 5 tcp_table
> man 5 socketmap_table
>
> and implement a server that responds in round-sobin order.

However, if you want to limit the number of conections for each
Postfix SMTP client IP address, set the right process limit
in master.cf for those SMTP clients, and do "postfix reload".

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Greg Sims
I built a maillog post processor that looks for 'status=sent' for
realy=.*\.outlook.com and collects to=<.*@(.*)> which is the domain
portion of the email addresses. These domains are placed in a
frequency array and sorted from highest to lowest.  The highest
frequency is hotmail.com as you would expect.  I then create a
transport.regexp file that is used to direct outlook traffic to four
transports.  If a domain has a frequency of more than 250, email
addresses are spread across the available transports (see
transport.regexp below). I run this post processor from crontab every
24 hours creating a new transport.regexp.

main.cf:

sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}
transport_maps = regexp:/etc/postfix/transport.regexp

outlook5_destination_concurrency_failed_cohort_limit = 100
outlook5_destination_concurrency_positive_feedback = 1/3
outlook5_destination_concurrency_negative_feedback = 1/8
#
outlook6_destination_concurrency_failed_cohort_limit = 100
outlook6_destination_concurrency_positive_feedback = 1/3
outlook6_destination_concurrency_negative_feedback = 1/8
#
outlook7_destination_concurrency_failed_cohort_limit = 100
outlook7_destination_concurrency_positive_feedback = 1/3
outlook7_destination_concurrency_negative_feedback = 1/8
#
outlook8_destination_concurrency_failed_cohort_limit = 100
outlook8_destination_concurrency_positive_feedback = 1/3
outlook8_destination_concurrency_negative_feedback = 1/8


master.cf (outlook transports use the same ip addresses as the randmap
transports):

outlook5  unix  -       -       n       -       2       smtp
  -o smtp_connection_cache_on_demand=no
  -o smtp_bind_address=xxx.xxx.xxx.235
  -o smtp_helo_name=r235.raystedman.org
  -o syslog_name=outlook5
outlook6  unix  -       -       n       -       2       smtp
  -o smtp_connection_cache_on_demand=no
  -o smtp_bind_address=xxx.xxx.xxx.236
  -o smtp_helo_name=r236.raystedman.org
  -o syslog_name=outlook6
outlook7  unix  -       -       n       -       2       smtp
  -o smtp_connection_cache_on_demand=no
  -o smtp_bind_address=xxx.xxx.xxx.237
  -o smtp_helo_name=r237.raystedman.org
  -o syslog_name=outlook7
outlook8  unix  -       -       n       -       2       smtp
  -o smtp_connection_cache_on_demand=no
  -o smtp_bind_address=xxx.xxx.xxx.238
  -o smtp_helo_name=r238.raystedman.org
  -o syslog_name=outlook8


transport.regexp (output of the maillog post processor):

/[a-f,0-1]@hotmail\.com$/                 outlook5:
/[g-l,2-4]@hotmail\.com$/                 outlook6:
/[m-s,5-6]@hotmail\.com$/                 outlook7:
/[t-z,7-9]@hotmail\.com$/                 outlook8:
/@hotmail\.com$/                          outlook8:
/[a-f,0-1]@outlook\.com$/                 outlook7:
/[g-l,2-4]@outlook\.com$/                 outlook6:
/[m-s,5-6]@outlook\.com$/                 outlook5:
/[t-z,7-9]@outlook\.com$/                 outlook5:
/@outlook\.com$/                          outlook6:
/[a-f,0-1]@msn\.com$/                     outlook7:
/[g-l,2-4]@msn\.com$/                     outlook8:
/[m-s,5-6]@msn\.com$/                     outlook8:
/[t-z,7-9]@msn\.com$/                     outlook7:
/@msn\.com$/                              outlook6:
/@live\.com$/                             outlook5:
/@hotmail\.es$/                           outlook5:
/@hotmail\.co\.uk$/                       outlook6:
/@outlook\.es$/                           outlook7:
/@live\.com\.mx$/                         outlook8:
/@live\.com\.ar$/                         outlook8:
etc. for 422 lines


This design spreads outlook.com traffic across the four transports
uniformly and does not rely on smtpd_recipient_restrictions =
check_recipient_mx_access.  This implements a recommendation by Wietse
with some enhancements.  Thank you!

Here are the stats from this morning:
  * email arrival rate: 250/minute
  * outlook.com email sent: 7,088
  * MaxConnections: 6

Aug 24 01:42:29 mail0.raystedman.org outlook5/smtp[245369]:
6C585C8AD3E: to=<[hidden email]>,
relay=hotmail-com.olc.protection.outlook.com[104.47.32.33]:25,
delay=0.05, delays=0.01/0/0.03/0.01, dsn=4.7.652, status=deferred
(host hotmail-com.olc.protection.outlook.com[104.47.32.33] said: 451
4.7.652 The mail server [xxx.xxx.xxx.xxx.235] has exceeded the maximum
number of connections. (S3115)
[SN1NAM01FT015.eop-nam01.prod.protection.outlook.com] (in reply to
MAIL FROM command))

The MaxConnections messages we receive are spread across the four
transports -- which is good.

I will continue to monitor this in hopes MaxConnections goes to zero
-- but I do not believe it will.  We are only running 2 processes per
transport.  We also need to increase the email arrival rate to have an
effective email server.  Should we run 1 process per transport? What
additional levers do we have available?

Thanks you, Greg
www.RayStedman.org


Blessings, Greg
www.RayStedman.org


On Sat, Aug 22, 2020 at 1:36 PM Wietse Venema <[hidden email]> wrote:

>
> Wietse Venema:
> > Greg Sims:
> > > sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}
> > >
> > > selects transports at random for delivering email.  Is it possible to
> > > schedule a set of transports using a round-robin discipline? This
> >
> > man 5 tcp_table
> > man 5 socketmap_table
> >
> > and implement a server that responds in round-sobin order.
>
> However, if you want to limit the number of conections for each
> Postfix SMTP client IP address, set the right process limit
> in master.cf for those SMTP clients, and do "postfix reload".
>
>         Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Wietse Venema
Greg Sims:
> Here are the stats from this morning:
>   * email arrival rate: 250/minute
>   * outlook.com email sent: 7,088
>   * MaxConnections: 6

If this happens 6 times, why not let Postfix retry delivery later?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Tom Sommer
In reply to this post by Wietse Venema


On 2020-08-22 22:35, Wietse Venema wrote:

> Wietse Venema:
>> Greg Sims:
>> > sender_dependent_default_transport_maps = randmap:{r235,r236,r237,r238}
>> >
>> > selects transports at random for delivering email.  Is it possible to
>> > schedule a set of transports using a round-robin discipline? This
>>
>> man 5 tcp_table
>> man 5 socketmap_table
>>
>> and implement a server that responds in round-sobin order.
>
> However, if you want to limit the number of conections for each
> Postfix SMTP client IP address, set the right process limit
> in master.cf for those SMTP clients, and do "postfix reload".

The process limit would be an overall limit on the transport?

Is there a way to limit the number of concurrent connections for each
destination IP?

---
Tom
Reply | Threaded
Open this post in threaded view
|

Re: Outlook.com Max Connections

Viktor Dukhovni
> On Oct 13, 2020, at 7:33 AM, Tom Sommer <[hidden email]> wrote:
>
> The process limit would be an overall limit on the transport?
>
> Is there a way to limit the number of concurrent connections for each destination IP?

No.  Postfix has no mechanism for that.  The only entity with a
global view of the queue is the queue manager, and the queue
manager has no knowledge of which IP addresses a particular
nexthop will resolve to once smtp(8) tries to do a delivery.

Postfix schedules delivery of a message to a nexthop destination
(domain in most cases), there is no explicit scheduling of
connections, the smtp(8) delivery agent just tries one or more
MX host IPs in turn, shuffling equal-priority IPs at random,
but also trying to make sure that when both IPv4 and IPv6
addresses are available, neither some connections of each type
will be made before giving up on the destination as unreachable.

A subtantial redesign (unlikely to happen) of the queue manager
and/or smtp(8) delivery agent would be needed in order to schedule
connections, which means helper processes (and queues) for mapping
a nexthop to a set of IP addresses


One might therefore split the smtp(8) delivery agent into two
parts, with MX resolution and TLS policy lookup happening
separately from delivery, and the pending list of IP addresses
and recipients given to a connection manager process that would
orchestrate actual SMTP transactions against particular IP
addresses, with appropriate per-IP concurrency limits.

But this has implications for the queue manager, because now
a delivery attempt may be sitting for a while in a downstream
queue, waiting for a connection slot.  The resolution half of
the delivery agent can't report completion of the job, but
having it blocked waiting for a connection introduces potential
bottlenecks.  This design space has not been explored.

--
        Viktor.