connection caching - limitations

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

connection caching - limitations

A. Schulze

Hallo,

that's my (legacy) setup:

a script generate 10k message files, same sender, different receiver.
they are injected using "sendmail -t -f sender < messagefile" in the local MTA
The MTA is configured to forward all messages to a central MSA.

This MSA require authentication and STARTTLS
All works fine, except speed.

The MTA uses one SMTP-Session per message. I think the reason is documented:
http://www.postfix.org/CONNECTION_CACHE_README.html#limitations, first bullet

as expected, even "smtp_connection_cache_destinations = static:all"  
doesn't help.

So what may be the strategy to speed up?

Thanks for ideas!
Andreas


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: connection caching - limitations

Wietse Venema
A. Schulze:

>
> Hallo,
>
> that's my (legacy) setup:
>
> a script generate 10k message files, same sender, different receiver.
> they are injected using "sendmail -t -f sender < messagefile" in the local MTA
> The MTA is configured to forward all messages to a central MSA.
>
> This MSA require authentication and STARTTLS
> All works fine, except speed.
>
> The MTA uses one SMTP-Session per message. I think the reason is documented:
> http://www.postfix.org/CONNECTION_CACHE_README.html#limitations, first bullet
>
> as expected, even "smtp_connection_cache_destinations = static:all"  
> doesn't help.
>
> So what may be the strategy to speed up?

Sending MULTI-RECIPIENT messages. Postfix by default groups up to
50 recipients per transaction.

Sending non-TLS email, until OpenSSL etc. support reuse on an
open connection.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: connection caching - limitations

Viktor Dukhovni
In reply to this post by A. Schulze

> On Apr 21, 2017, at 4:11 AM, A. Schulze <[hidden email]> wrote:
>
> A script generates 10k message files, same sender, different receiver.
> They are injected using "sendmail -t -f sender < messagefile" in the local MTA
> The MTA is configured to forward all messages to a central MSA.

Message injection via sendmail(1) is much less efficient than injection
via SMTP.  The message is synced to disk twice, and the pickup(8) service
can only process one message at a time, while SMTP inject can handle
multiple messages in parallel.

> This MSA require authentication and STARTTLS
> All works fine, except speed.

You've provided no information on where the performance bottleneck lies.
What are the averages of the delays=a/b/c/d log values?

> The MTA uses one SMTP-Session per message. I think the reason is documented:
> http://www.postfix.org/CONNECTION_CACHE_README.html#limitations, first bullet

Sure, and connection re-use is mainly helpful when some of the IP addresses
of the SMTP servers for the nexthop domain are non-responsive and connections
are timing out.  Otherwise, connection caching offers only marginal gains.
(If PTR records for your client's IP address are registered at a non-responsive
DNS server, that could be another source of per-connection overhead).

> as expected, even "smtp_connection_cache_destinations = static:all" doesn't help.
>
> So what may be the strategy to speed up?

First identify the origin of the delays.  Lack of connection reuse is rarely
the problem.

See:

        http://www.postfix.org/TUNING_README.html
        http://www.postfix.org/QSHAPE_README.html

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: connection caching - limitations

A. Schulze


Am 21.04.2017 um 16:08 schrieb Viktor Dukhovni:
> Message injection via sendmail(1) is much less efficient than injection
> via SMTP.  The message is synced to disk twice, and the pickup(8) service
> can only process one message at a time, while SMTP inject can handle
> multiple messages in parallel.

I'm aware of that. But (I think) submission is not the problem to me: The script take seconds only.
but then the messages are visible in the active queue. postfix start relaying to the MSA
one by one message up to $relay_destination_concurrency_limit (20) parallel.

> You've provided no information on where the performance bottleneck lies.
> What are the averages of the delays=a/b/c/d log values?
can look on Monday for these values....

> Sure, and connection re-use is mainly helpful when some of the IP addresses
> of the SMTP servers for the nexthop domain are non-responsive and connections
> are timing out.
that's not the case

> Otherwise, connection caching offers only marginal gains.
> (If PTR records for your client's IP address are registered at a non-responsive
> DNS server, that could be another source of per-connection overhead).
also not happen here

>> as expected, even "smtp_connection_cache_destinations = static:all" doesn't help.
>>
>> So what may be the strategy to speed up?
>
> First identify the origin of the delays.  Lack of connection reuse is rarely
> the problem.
>
> See:
>
> http://www.postfix.org/TUNING_README.html
> http://www.postfix.org/QSHAPE_README.html
>
OK, will read again, Thanks

One point to answer Wietse's suggestion: "Sending MULTI-RECIPIENT messages"
That's impossible because every message contain unique content for the
specific receiver. ( really 10k /different/ messages )

Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: connection caching - limitations

A. Schulze
In reply to this post by Viktor Dukhovni

Viktor Dukhovni:

> You've provided no information on where the performance bottleneck lies.
> What are the averages of the delays=a/b/c/d log values?

Thanks to Viktor for the reminder to "proof the performance bottleneck"
Today I send 5k messages and /measure/ the times.


   time for i in `seq 1 5000`; do sendmail -f $sender $recipient <  
msgfile; done
   real    3m34.281s
   user    0m13.120s
   sys     0m9.764s

first message
Apr 24 10:45:22 submitter postfix/pickup[14884]: 3wBKfp0Q7MzDYR:  
uid=12345 from=<$sender>
Apr 24 10:45:23 submitter postfix/smtp[17141]: 3wBKfp0Q7MzDYR:  
to=<$recipient>, relay=$MSA:25, delay=1.5, delays=0.05/0.01/0.49/1,  
dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3wBKfp49pbz3h1D)

last message
Apr 24 10:48:56 submitter postfix/pickup[29155]: 3wBKkw1tlJzGNV:  
uid=12345 from=<$sender>
Apr 24 10:51:43 submitter postfix/smtp[30768]: 3wBKkw1tlJzGNV:  
to=<$recipient>, relay=$MSA:25, delay=167, delays=0.03/165/0.41/1.2,  
dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3wBKp60c9kzXN9)

and some minutes later:
Apr 24 10:55:03 submitter postfix/scache[17300]: statistics: start  
interval Apr 24 10:45:23
Apr 24 10:55:03 submitter postfix/scache[17300]: statistics: domain  
lookup hits=0 miss=4995 success=0%


   the last message arrived ~4 minutes later at $recipient mailbox
   -> there is no problem (for me)

lesson learned: measure, don't suspect

Andreas

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: connection caching - limitations

Viktor Dukhovni

> On Apr 24, 2017, at 6:34 AM, A. Schulze <[hidden email]> wrote:
>
> Today I send 5k messages and /measure/ the times.

Look closely at the delays=a/b/c/d times.

>  time for i in `seq 1 5000`; do sendmail -f $sender $recipient < msgfile; done
>  real    3m34.281s
>  user    0m13.120s
>  sys     0m9.764s

That's 214 seconds, or 23 msgs/sec.  Are you writing straight through to disk,
or do you have a RAID controller with a battery-backed cache?  How big is the
"msgfile"?  What iostat tell you about your disk while this is happening?

> first message
> Apr 24 10:45:22 submitter postfix/pickup[14884]: 3wBKfp0Q7MzDYR: uid=12345 from=<$sender>
> Apr 24 10:45:23 submitter postfix/smtp[17141]: 3wBKfp0Q7MzDYR: to=<$recipient>, relay=$MSA:25, delay=1.5, delays=0.05/0.01/0.49/1, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3wBKfp49pbz3h1D)

The c/d numbers are very high for communication to a presumably nearby MSA.
What might be the cause of 400ms of latency setting up the connection?  What
might be the cause of 1000ms of latency to transfer the message?

At 1.5 seconds per delivery, and a concurrency of 20, your maximum throughput
is ~13.3 msgs/sec, which for 5000 msgs is ~375s or 6m15s.

> last message
> Apr 24 10:48:56 submitter postfix/pickup[29155]: 3wBKkw1tlJzGNV: uid=12345 from=<$sender>
> Apr 24 10:51:43 submitter postfix/smtp[30768]: 3wBKkw1tlJzGNV: to=<$recipient>, relay=$MSA:25, delay=167, delays=0.03/165/0.41/1.2, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3wBKp60c9kzXN9)

This message had a similar "c+d" of 1.6s.  The total time was 6m20s!  Which suggests that
the bottleneck is a high-latency MSA.  If the MSA has more capacity to offer and is willing
to accept more messages in parallel, you can raise the concurrency limit to compensate, but
it is generally more productive to see if you can reduce the latency first.

        Throughput = Concurrency / Latency

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: connection caching - limitations

Wietse Venema
Viktor Dukhovni:

> > last message
> > Apr 24 10:48:56 submitter postfix/pickup[29155]: 3wBKkw1tlJzGNV: uid=12345 from=<$sender>
> > Apr 24 10:51:43 submitter postfix/smtp[30768]: 3wBKkw1tlJzGNV: to=<$recipient>, relay=$MSA:25, delay=167, delays=0.03/165/0.41/1.2, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3wBKp60c9kzXN9)
>
> This message had a similar "c+d" of 1.6s.  The total time was
> 6m20s!  Which suggests that the bottleneck is a high-latency MSA.
> If the MSA has more capacity to offer and is willing to accept
> more messages in parallel, you can raise the concurrency limit to
> compensate, but it is generally more productive to see if you can
> reduce the latency first.
>
> Throughput = Concurrency / Latency

In other words, submit via SMTP instead of /usr/sbin/sendmail.
There are many open-source /usr/sbin/sendmail alternatives that
submit via SMTP. The Postfix sendmail command can't do that because
it would lose that is submitted while Postfix is down.

        Wietse
Loading...