SMTP connection reuse with TLS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SMTP connection reuse with TLS

mark burdett
Hi, I was curious if there are any plans for postfix to eventually
support SMTP connection reuse with STARTTLS.

We were using postfix to deliver bulk mail (email newsletters) to a mail
relay.  When TLS was disabled, Postfix was able to open up multiple
connections to the relay and reuse these connections for some period of
time, maintaining a high send rate with minimal RTT due to TCP connection.

After enabling TLS, postfix delivery was much slower, and packet capture
revealed the connection reset after each message was delivered.  Postfix
documentation confirms there is no connection reuse with TLS.
Unfortunately this dramatically slows down delivery to the relay because
of the RTT overhead of new TCP connections.

We switched to having our app do SMTP delivery directly to the relay
with connection reuse (using a standard SMTP library), rather than
delivering to the local postfix instance.

This was a reasonable work-around for us.  But we'd love to have postfix
on hand to queue and deliver mail to the relay, if it were possible to
optimize the STARTTLS support.

(In case you're curious why rapid delivery of bulk mail matters, even
bulk mail can be time sensitive: for example, advocacy organizations ask
subscribers to tweet or call their elected representative that morning.)

--mark B.


signature.asc (845 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Viktor Dukhovni
On Tue, Aug 01, 2017 at 02:41:52PM -0700, mark burdett wrote:

> Hi, I was curious if there are any plans for postfix to eventually support
> SMTP connection reuse with STARTTLS.

This requires a complex outbound TLS proxy to cache the connections
in process, and handle peer authentication.  Some of the work has
already been done on the inbound side to enable TLS in postscreen,
but much work remains, as outbound TLS is much more complex.  This
is not likely to happen in the near term.

> After enabling TLS, postfix delivery was much slower, and packet capture
> revealed the connection reset after each message was delivered.  Postfix
> documentation confirms there is no connection reuse with TLS. Unfortunately
> this dramatically slows down delivery to the relay because of the RTT
> overhead of new TCP connections.

Increased latency can be amortized with increased concurrency.
Just open more connections and the overall throughput rate will
remain the same.

        Throughput = Concurrency / Latency

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Benny Pedersen-2
In reply to this post by mark burdett
Where is logs ?

And lastly

postconf  -nf
postconf -Mf

from both servers, with that its more chance of more help
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

mark burdett
In reply to this post by Viktor Dukhovni
On 08/01/2017 03:32 PM, Viktor Dukhovni wrote:

> On Tue, Aug 01, 2017 at 02:41:52PM -0700, mark burdett wrote:
>
>> Hi, I was curious if there are any plans for postfix to eventually support
>> SMTP connection reuse with STARTTLS.
>
> This requires a complex outbound TLS proxy to cache the connections
> in process, and handle peer authentication.  Some of the work has
> already been done on the inbound side to enable TLS in postscreen,
> but much work remains, as outbound TLS is much more complex.  This
> is not likely to happen in the near term.
Thanks for the explanation! It does sound tricky, and would need plenty
of regression testing.

>> After enabling TLS, postfix delivery was much slower, and packet capture
>> revealed the connection reset after each message was delivered.  Postfix
>> documentation confirms there is no connection reuse with TLS. Unfortunately
>> this dramatically slows down delivery to the relay because of the RTT
>> overhead of new TCP connections.
>
> Increased latency can be amortized with increased concurrency.
> Just open more connections and the overall throughput rate will
> remain the same.
>
> Throughput = Concurrency / Latency
That's true, as a work-around.  Unfortunately we're talking about not
just opening a new TCP connection but also reestablishing TLS, which
means yet more RTT and CPU.  So the increased concurrency will be
significant and will require upping limits on the client and server
side.  I'm guessing most folks in this scenario will prefer to code a
lightweight SMTP worker that can reuse connections, running at a lower
concurrency.

--mark B.


signature.asc (845 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Marco Pizzoli


On Wed, Aug 2, 2017 at 6:57 PM, mark burdett <[hidden email]> wrote:


That's true, as a work-around.  Unfortunately we're talking about not just opening a new TCP connection but also reestablishing TLS, which means yet more RTT and CPU.  So the increased concurrency will be significant and will require upping limits on the client and server side.  I'm guessing most folks in this scenario will prefer to code a lightweight SMTP worker that can reuse connections, running at a lower concurrency.

--mark B.

Don't take for granted that you can't optimize things. I have an important experience to share...
I faced the same issue when enabling the SSL connection to the remote systems. My 220K email sending was lasting 2/3 hours more than without SSL.
Now I can send 100K emails in 1h, with a single Postfix instance and a single sending IP address. On a virtual Linux machine on VMware.

Have a look at:
- smtp_tls_session_cache_database <-- this is the most important thing. I suggest lmdb as the backing store
- if you are on Linux on virtual, also to RNGD/Haveged (the second being the best for speed)
- making use of the most recent openssl major version you can afford to link postifx to
- loading jemalloc as the memory allocator for all postfix processes

Provided that the hard drive won't be an issue, you will see a considerable difference in the performance.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Viktor Dukhovni
In reply to this post by mark burdett
On Wed, Aug 02, 2017 at 09:57:43AM -0700, mark burdett wrote:

> > Increased latency can be amortized with increased concurrency.
> > Just open more connections and the overall throughput rate will
> > remain the same.
> >
> > Throughput = Concurrency / Latency
>
> That's true, as a work-around.  Unfortunately we're talking about not just
> opening a new TCP connection but also reestablishing TLS, which means yet
> more RTT and CPU.  So the increased concurrency will be significant and will
> require upping limits on the client and server side.

Yes, instead of the default destination concurrency limit of 20,
you'll need a limit of 50 or 100, but this should not be prohibitive.
With TLS session caching (ideally the upstream relay supports TLS
session resumption) the TLS handshake can be quite lightweight and
the total CPU is about the same, determined largely by the cost of
encrypting the message payload.  So for a given throughput the CPU
is the same, with the extra concurrency just amortizing latency,
with little impact on CPU.

Of course if the other end does not support TLS session resumption
there will be some extra CPU overhead, but modern CPUs are plenty
fast, and can do thousands of RSA handshakes per second.

> I'm guessing most folks in this scenario will prefer to code a lightweight
> SMTP worker that can reuse connections, running at a lower concurrency.

Your call.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Viktor Dukhovni
In reply to this post by Marco Pizzoli
On Wed, Aug 02, 2017 at 07:11:23PM +0200, Marco Pizzoli wrote:

> Have a look at:
> - smtp_tls_session_cache_database <-- this is the most important thing. I
> suggest lmdb as the backing store

Yes, but Berkeley DB also works well enough in practice.

> - if you are on Linux on virtual, also to RNGD/Haveged (the second being
> the best for speed)

I don't think this is good advice.  Use the default entropy source:

        tls_random_source = dev:/dev/urandom

and let the kernel take care of entropy.

> - loading jemalloc as the memory allocator for all postfix processes

This is unlikely to be a bottleneck for SMTP.  The default malloc should
be just fine.  The only real tuning required for a dedicated upstream
is:

    - Enable the client-side SMTP TLS session cache

    - Make sure the upstream server supports session resumption, ideally
      via session tickets, but a remote cache is also ok.

    - Increase concurrency as required for the larger TLS
      round-trip delay.  If the average message size is large
      enough, the latency increase will be small, and perhaps no
      tuning is required.  If the average message size is small,
      a multiple of 2 or a bit more may be appropriate.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Marco Pizzoli


On Wed, Aug 2, 2017 at 7:44 PM, Viktor Dukhovni <[hidden email]> wrote:
On Wed, Aug 02, 2017 at 07:11:23PM +0200, Marco Pizzoli wrote:

> Have a look at:
> - smtp_tls_session_cache_database <-- this is the most important thing. I
> suggest lmdb as the backing store

Yes, but Berkeley DB also works well enough in practice.

I believe you. But my experience comparing the two in OpenLDAP is strongly toward lmdb.

> - if you are on Linux on virtual, also to RNGD/Haveged (the second being
> the best for speed)

I don't think this is good advice.  Use the default entropy source:

        tls_random_source = dev:/dev/urandom

and let the kernel take care of entropy.

I did not say to change the default entropy source configured in postfix.
I only meant to "help" the /dev/random source by making use of (one of) those daemons. In virtual machines the entropy is really low. In my vmware machines I am at 300, compared to the maximum possible of 4000.
Those daemons really do help. During one of my bulk-sendings I saw the rngd daemon popping up within "top", so to say it needed to do its job.


> - loading jemalloc as the memory allocator for all postfix processes

This is unlikely to be a bottleneck for SMTP.  The default malloc should
be just fine. 

Could be, but jemalloc (and tcmalloc) give also benefit for memory fragmentation, apart for the pure allocation speed.
So in constrained memory systems more than in those with huge memory they make you at least not-losing-something in the long term. Or at least they should.
 
The only real tuning required for a dedicated upstream
is:

    - Enable the client-side SMTP TLS session cache

Confirm
 
    - Make sure the upstream server supports session resumption, ideally
      via session tickets, but a remote cache is also ok.

Confirm. Indeed I don't see all of them supporting it, unfortunately.
 
    - Increase concurrency as required for the larger TLS
      round-trip delay.  If the average message size is large
      enough, the latency increase will be small, and perhaps no
      tuning is required.  If the average message size is small,
      a multiple of 2 or a bit more may be appropriate.

On this latest point I have to in part disagree... My experience makes me doubtful on just extending the concurrent connections towards the remote systems.
I actually need to limit the concurrent connection number to my destinations in order to not be deferred due to the "too many connections from your IP".

 

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SMTP connection reuse with TLS

Viktor Dukhovni
On Wed, Aug 02, 2017 at 08:03:14PM +0200, Marco Pizzoli wrote:

> > Yes, but Berkeley DB also works well enough in practice.
> >
>
> I believe you. But my experience comparing the two in OpenLDAP is strongly
> toward lmdb.

The Postfix SMTP cache is a very different use-case.  The main
incentive to use LMDB is suboptimal Berkeley DB licensing.  For
Postfix, Berkeley DB is likely to perform about the same as LMDB
(perhaps slightly better, or slightly worse, but in neither case
a real bottleneck).

> Could be, but jemalloc (and tcmalloc) give also benefit for memory
> fragmentation, apart for the pure allocation speed.

> >     - Increase concurrency as required for the larger TLS
> >       round-trip delay.  If the average message size is large
> >       enough, the latency increase will be small, and perhaps no
> >       tuning is required.  If the average message size is small,
> >       a multiple of 2 or a bit more may be appropriate.
>
> On this latest point I have to *in part* disagree... My experience makes me
> doubtful on just extending the concurrent connections towards the remote
> systems.

When sending to 3rd party MX hosts, indeed yes, but here the OP is
using a single dedicated usptream smarthost relay, so larger
concurrency should not be a problem.

--
        Viktor.
Loading...