Maximum simultaneous outbounds ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Maximum simultaneous outbounds ?

Ronald F. Guilmette-2


I got into a somehwat interesting discussion/argument with a fellow today
about how many different domains could reasonably be supported on a single
IPv4 address, generally speaking.

I pointed out that there's essentially no limits on how many different
domains a single instance of Apache, running on a single IPv4 address,
could support, other than maybe the general limits imposed by available
memory, maximum simultaneously open sockets supported by the underlying
OS, etc.  So in theory, one could put a million web sites on a single IPv4
address using Apache.  (It probably wouldn't work that well if all one
million of them were visited simultaneously, but I'm not even 100% sure
that that would necessarily cause a problem.)

Likeiwse, I'm not aware of any hard limit that, for example, BIND imposes
on the maximum number of zones that it can be an authoritative server for,
again using only a single IPv4 address.

Lastly, for email, I'm not aware of any hard limit that Postfix imposes
on the number of domains for which a single running Postfix instance
could -receive- email for on the inbound side.

But this other fellow I've been taking to offered an unexpected observation:
If a given Postfix installation was attempting to support, say, 1 million
unique domain names (correponding to 1 million unique customers) and if
just 11,000 of those were to all simultaneously attempt to send -outbound-
emails to six (6) different destinations apiece, then... this other fellow
asserted... all of the 65536 maximum available IPv4 port numbers would be
exhausted, and then havoc would ensue.

I did note my belief that Postfix is very robust in the face of most
unusual and/or uncommon conditions, and thus thus, even in such an
unusual secenario, Postfix would probably handle it just fine, and that
worst case, the later emails might just experience a bit of reasonable
delay while they waited for outbound ports to be freed up.

But this whole exchange got me to thinking... Does Postfix use each
outbound TCP port number in a manner where that port number is and remains
entirely and exclusively dedicated to being used for just -one- outbound
SMTP connection/session at a time?  If so, is there any compelling reason
why it MUST do so?

It's been awhile since I did any serious socket programming, but it is
my recollection that each connected socket is uniquely described by
a tuple consisting of the (a) source IP and (b) the source port and
(c) the destination IP and (d) the destination port and (e) some other
stuff I can't remember anymore but that is probably not important to my
question anyway.

As I understand it, the OS can keep everything straight and can send
all packets where they belong as long as each of these socket-describing
tuples is unique when consiedered as a whole and single unit.   Thus, one
single local port can support multiple simultaneous connections to other
IP addresses elsewhere as long as those remote IP addresses are all
different and unique, with respect to one another.

Anyway, the point that I am getting at is just this:  Does Postfix ever
make or have more than 65536 outbound TCP connections (from/on a single
IPv4 address) which are all simultaneously alive and active at any one
instant in time?  If not, could it theoretically do so?

Please understand.  I am *not* asking whether or not it would be the most
epic and colossal pain in the ass of all time to *rewirite* Postfix to
make it perform such a clever trick.  I accept the possibility/probability
that it might be.  That's not my question,  I am just asking whether or
not your garden variety run-of-the-mill IPv4 implementation of Berkeley
sockets could, in theory, support -some- application which opened and
connected -some- set of TCP sockets to -some- set of mutually unique
remote IPv4 addresses elsewhere and thus have more than 65536 of those
open at some given instant in time.

I know.  I know.  I should get the hell out of here and go pester some
UNIX/Linux sockets or kernel people for an answer to this quetsion, but
as the question is almost uniquely relevant to mail servers, I though
that I would start here.  If the answer turnes out to be "Sure!  Postfix
is already doing that every day of the week and twice on Sunday!" then
I don't really need to reveal my ignorance anyplace else.


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Viktor Dukhovni
> On Mar 3, 2019, at 2:56 AM, Ronald F. Guilmette <[hidden email]> wrote:
>
> But this other fellow I've been taking to offered an unexpected observation:
> If a given Postfix installation was attempting to support, say, 1 million
> unique domain names (correponding to 1 million unique customers) and if
> just 11,000 of those were to all simultaneously attempt to send -outbound-
> emails to six (6) different destinations apiece, then... this other fellow
> asserted... all of the 65536 maximum available IPv4 port numbers would be
> exhausted, and then havoc would ensue.

This mental model is deeply flawed.  Postfix has a queue manager, that
limits the concurrency per destination, and the active queue size.  And
a master(8) process that limits the process count per transport.  Postfix
also accepts messages at a finite rate, so 66,000 messages will not arrive
instantaneously.  Once the active queue is full further accepted messages
will accumulate in the incoming queue on disk, but will not consume network
resources or RAM.

It is of course possible to receive inbound messages faster than the
steady-state output rate, in which case the number of queued messages
will grow quite high.  And if this is allowed to continue indefinitely,
until the file system almost fills up.

But the port number exhaustion scenario is not even close.

  http://www.postfix.org/OVERVIEW.html#delivering
  http://www.pos

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Ronald F. Guilmette-2

In message <[hidden email]>,
Viktor Dukhovni <[hidden email]> wrote:

>> On Mar 3, 2019, at 2:56 AM, Ronald F. Guilmette
><[hidden email]> wrote:
>>
>> But this other fellow I've been taking to offered an unexpectedobservation:
>> If a given Postfix installation was attempting to support, say, 1 million
>> unique domain names (correponding to 1 million unique customers) and if
>> just 11,000 of those were to all simultaneously attempt to send -outbound-
>> emails to six (6) different destinations apiece, then... this other fellow
>> asserted... all of the 65536 maximum available IPv4 port numbers would be
>> exhausted, and then havoc would ensue.
>
>This mental model is deeply flawed.

Thank you for the response Vicktor, but could you please be more specific,
just so that I have it on the record?

Whose mental model is it that you are saying is "deeply flawed"?  Mine or
the other guy's?

>Postfix has a queue manager, that
>limits the concurrency per destination, and the active queue size.  And
>a master(8) process that limits the process count per transport. Postfix
>also accepts messages at a finite rate, so 66,000 messages will not arrive
>instantaneously.  Once the active queue is full further accepted messages
>will accumulate in the incoming queue on disk, but will not consume network
>resources or RAM.

Paraphrasing, it sounds to me like you just said that Postfix is designed
to behave well, and in fact does behave well, even under very high loads.

But I, for one, already knew that.  (And I suspect that most folks who use
Postfix at "big" places knew that already also.)

I still would like to know if the total number of outbound SMTP connections
which Postfix may have open, at any one given point in time, may or may not
exceed 65536.

(I admit that this is really rather entirely a matter of academic curiosity
on my part and that it may have little or no practical implications.  I
just have this running disagreement going about how many angels can dance
on the head of... I'm sorry... about how many domain names can, in practice
be hosted on a single IPv4 address.  I say "millions".  Others are telling
me that I'm delusional and need to seek immediate treatment. I am not yet
favorably inclined to acecpt their judgement on the matter.    The key point
of disagreement seens to be our differing evaluations about how many
simultaneous outbound SMTP a good quality... or best quality... SMTP server
could in practice support.)

>But the port number exhaustion scenario is not even close.

I'm not at all sure how to interpret that.

May I assume that your intent was to say that a hosting company could
tell all of its 1 million customers to use a single shared mail server
for all of their outbound needs, and that even though this might possibly
create a unsustainable load, the unsustainability would *not* become
evident, in the first instance, as an exhaustion of outbound IPv4 port
numbers?

Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Wietse Venema
In reply to this post by Ronald F. Guilmette-2
Ronald F. Guilmette:
> But this other fellow I've been taking to offered an unexpected observation:
> If a given Postfix installation was attempting to support, say, 1 million
> unique domain names (correponding to 1 million unique customers) and if
> just 11,000 of those were to all simultaneously attempt to send -outbound-
> emails to six (6) different destinations apiece, then... this other fellow
> asserted... all of the 65536 maximum available IPv4 port numbers would be
> exhausted, and then havoc would ensue.

As shipped, Postfix makes up to 100 parallel outbound connections,
200 if configured as an MX for remote domains. It also has limits
on the number and size of in-memory objects, and it stops accepting
new mail before the file system is full.

Postfix is in a different league than software that just runs the
system into the ground under load, and that requires a babysitter
to become unstuck.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Viktor Dukhovni
In reply to this post by Ronald F. Guilmette-2
On Sun, Mar 03, 2019 at 01:49:12AM -0800, Ronald F. Guilmette wrote:

> >> But this other fellow I've been taking to offered an unexpectedobservation:
> >> If a given Postfix installation was attempting to support, say, 1 million
> >> unique domain names (correponding to 1 million unique customers) and if
> >> just 11,000 of those were to all simultaneously attempt to send -outbound-
> >> emails to six (6) different destinations apiece, then... this other fellow
> >> asserted... all of the 65536 maximum available IPv4 port numbers would be
> >> exhausted, and then havoc would ensue.
> >
> >This mental model is deeply flawed.
>
> Thank you for the response Viktor, but could you please be more specific,
> just so that I have it on the record?
>
> Whose mental model is it that you are saying is "deeply flawed"?  Mine or
> the other guy's?

There's only one "mental model" under discussion of what happens
when Postfix is delivering email.  Namely, that no matter how many
messages arrive in quick succession, they'll all be "talking to the
network" (using an outbound TCP connection) at the same time.  This
mental model is deeply flawed.

I could also point out that TCP stacks can allow the same local
ephemeral port to be used for multiple TCP connections, provided
the 4-tuple (remote ip, remote port, local ip, local port) is unique.
There is no requirement that just the local ports of established
TCP connections be distinct.

> Paraphrasing, it sounds to me like you just said that Postfix is designed
> to behave well, and in fact does behave well, even under very high loads.

I tried to provide a more accurate model of how Postfix delivers
email, from which you or anyone else can reach your own conclusions.

> But I, for one, already knew that.  (And I suspect that most folks who use
> Postfix at "big" places knew that already also.)

Well, it seems that you only knew the empirical conclusions.  Had you
known how Postfix ensures performance under load, you'd have refuted
the other fellow's false scenario without coming to the list.

> I still would like to know if the total number of outbound SMTP connections
> which Postfix may have open, at any one given point in time, may or may not
> exceed 65536.

This is a silly question.  Typical message delivery latency can be
estimated at around 1s.  A hypothetical server running at a concurrency
of 64k connections would be pumping out 64k msgs/sec, but the Postfix
queue manager and the disk are very unlikely to go that fast.
Realistically, a single email server may be able to deliver at best
O(1000) msgs/sec.

At a hypothetical sustained 64k messages per second, a server would
be able to deliver around 5.6 billion messages a day.  That's not
a realistic load for a single machine, either inbound or outbound.

Real servers handle smaller loads with outbound concurrency limits
in the hundreds or a few thousand.  With Postfix brief input spikes
that exceed the output rate lead growth in the size of the queue
without unbounded demand for CPU and network.

There are also caps on concurrent incoming connections, and
sufficiently high input rates will reduce opportunities for new
connections, forcing some or most senders to defer delivery.  That's
what horizontal scaling is for, with anycast IPs to spread the load
geographically, and in-datacentre load-balancers to further spread
the load among multiple machines, ...

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Ronald F. Guilmette-2
In reply to this post by Wietse Venema

In message <[hidden email]>, you wrote:

>Postfix is in a different league than software that just runs the
>system into the ground under load, and that requires a babysitter
>to become unstuck.

Thanks for the clarification and the clarity.

You wouldn't happen to have the names of any products that fall
into that other category that you just described would you?

(It really irks me the way that some people demand lots and lots of
IPv4 addresses, which are in short supply, in order to accomplish
things that could be done with lots lots less of that particular
finite and limited resource.  But convincing some of these folks
of the error of their ways isn't easy, and I could use all of the
additional ammunition that I can lay hands on.)
Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Ronald F. Guilmette-2
In reply to this post by Viktor Dukhovni

In message <[hidden email]>, Viktor wrote:

>I could also point out that TCP stacks can allow the same local
>ephemeral port to be used for multiple TCP connections, provided
>the 4-tuple (remote ip, remote port, local ip, local port) is unique.
>There is no requirement that just the local ports of established
>TCP connections be distinct.

This answers my original and most fundamental question, and confirms
what I believed I already knew about the potential for simultaneous
local IPv4 port reuse.  So thanks for that.

>Well, it seems that you only knew the empirical conclusions.  Had you
>known how Postfix ensures performance under load, you'd have refuted
>the other fellow's false scenario without coming to the list.

Well, when arguing (e.g. on a mailing list) with someone who consistantly
drops down into the classic retorical "appeal to authority" mode (as
in: "I know, you don't, and you are an idiot, so STFU!') it is usually
best to get a pronouncement from a a different authority having a
different view, if the goal is to refute the false "appeal to authority"
being put forward.  So I came here.

I personally don't know off the top of my head any folks who are more
widely considered "authorities" on how mail servers can and should work
than you and Wietse.

>> I still would like to know if the total number of outbound SMTP connections
>> which Postfix may have open, at any one given point in time, may or may not
>> exceed 65536.
>
>This is a silly question.  Typical message delivery latency can be
>estimated at around 1s.  A hypothetical server running at a concurrency
>of 64k connections would be pumping out 64k msgs/sec, but the Postfix
>queue manager and the disk are very unlikely to go that fast.
>Realistically, a single email server may be able to deliver at best
>O(1000) msgs/sec.
>
>At a hypothetical sustained 64k messages per second, a server would
>be able to deliver around 5.6 billion messages a day.  That's not
>a realistic load for a single machine, either inbound or outbound.
>
>Real servers handle smaller loads with outbound concurrency limits
>in the hundreds or a few thousand.  With Postfix brief input spikes
>that exceed the output rate lead growth in the size of the queue
>without unbounded demand for CPU and network.
>
>There are also caps on concurrent incoming connections, and
>sufficiently high input rates will reduce opportunities for new
>connections, forcing some or most senders to defer delivery.  That's
>what horizontal scaling is for, with anycast IPs to spread the load
>geographically, and in-datacentre load-balancers to further spread
>the load among multiple machines, ...

Well, but see, this is precisly what the argument was/is about.  

As soon as you start talking about load balancers, you are also taking
about more than one IP address.

It was and is my contention that even great vast gobs of outbound email
can be handled on a single IPv4 address, *if* one is doing it "right".
And by "right" in this context, I mean having a great big pipe into the
machine in question, having the machine itself be something killer, like
fer instance a 32-core Ryzen or something, and having the "disk" be
something like a 1TB NVME stick, or maybe even... dare I say it?... Optane!

Basically, my central thesis in this other conversation that I'm having
elsewhere is that current usage norms when it comes to (finite and vanishing)
IPv4 addresses are, by and large, exceptionally wasteful and that allocation
policy should be adjusted accordingly.

My opponents in this debate have used and are using mutiple (mostly lame)
arguments for why they need lots and lots of IPv4 addreses.  I was able
to rather easily shoot down most of those (obviously lame) arguments on
my own, but when it came to this question of how many simultaneous outbound
mail sessions could dance on the head of a single IPv4 address, I had
to ask for some help.... which I believe I have now, mostly, gotten.
(Thank you.)


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Wietse Venema
In reply to this post by Ronald F. Guilmette-2
Ronald F. Guilmette:

>
> In message <[hidden email]>, you wrote:
>
> >Postfix is in a different league than software that just runs the
> >system into the ground under load, and that requires a babysitter
> >to become unstuck.
>
> Thanks for the clarification and the clarity.
>
> You wouldn't happen to have the names of any products that fall
> into that other category that you just described would you?

Let's say that Postfix was influenced by good and bad experiences
with other software.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

@lbutlr
In reply to this post by Ronald F. Guilmette-2
On Mar 3, 2019, at 16:17, Ronald F. Guilmette <[hidden email]> wrote:
> You wouldn't happen to have the names of any products that fall
> into that other category that you just described would you?

rsync done this to my system in the past.

--
My main job is trying to come up with new and innovative and effective ways to reject even more mail. I'm up to about 97% now.

Reply | Threaded
Open this post in threaded view
|

Re: Maximum simultaneous outbounds ?

Andrey Repin-2
In reply to this post by Ronald F. Guilmette-2
Greetings, Ronald F. Guilmette!

>>Postfix is in a different league than software that just runs the
>>system into the ground under load, and that requires a babysitter
>>to become unstuck.

> Thanks for the clarification and the clarity.

> You wouldn't happen to have the names of any products that fall
> into that other category that you just described would you?

> (It really irks me the way that some people demand lots and lots of
> IPv4 addresses, which are in short supply, in order to accomplish
> things that could be done with lots lots less of that particular
> finite and limited resource.  But convincing some of these folks
> of the error of their ways isn't easy, and I could use all of the
> additional ammunition that I can lay hands on.)

The main argument you should keep in mind is that connection limit is not
"64k", it is "64k from single local //port// to single remote //address//".
Which in itself is a hard to even imagine (not to say - reach) situation.

There's more realistic limits set in kernel, which will be reached long before
that, and which can still be configured to allow for more simultaneous
connections, than set by default.
Google "c10k problem" for pointers to possible solutions.


--
With best regards,
Andrey Repin
Monday, March 4, 2019 21:49:26

Sorry for my terrible english...