tlsmgr high io load because of session cache

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

tlsmgr high io load because of session cache

Matthias Schneider
Hello,

I had a very high I/O load on process tlsmgr because the smtp_scache and
smtpd_scache files are written to often (smtp_scache.db ~70mb) .

data_directory = /var/lib/postfix
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

moving /var/lib/postfix to a tmpfs filesystem solved my problem for now,
but i am looking for a better solution.
I noticed that postfix also supports memcache as lookup table
(http://www.postfix.org/DATABASE_README.html)
is this also supported for smtp_tls_session_cache_database ? Can anyone
show me a config example?

Thanks!
Matthias Schneider

Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Wietse Venema
Matthias Schneider:

> Hello,
>
> I had a very high I/O load on process tlsmgr because the smtp_scache and
> smtpd_scache files are written to often (smtp_scache.db ~70mb) .
>
> data_directory = /var/lib/postfix
> smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
> smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
>
> moving /var/lib/postfix to a tmpfs filesystem solved my problem for now,
> but i am looking for a better solution.
> I noticed that postfix also supports memcache as lookup table
> (http://www.postfix.org/DATABASE_README.html)
> is this also supported for smtp_tls_session_cache_database ? Can anyone
> show me a config example?

memcache should work just fine. Specify memcache:/configfile instead
of btree:/pathname. The contents of the configfile are documented
in memcache_table(5). There is no need to change the default ttl
of 3600 seconds.

In main.cf, set smtpd_tls_session_cache_timeout=0 and
smtp_tls_session_cache_timeout=0. Expiration is done in the memcache
server.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Viktor Dukhovni
On Fri, Nov 14, 2014 at 10:10:52AM -0500, Wietse Venema wrote:

> Matthias Schneider:
> > Hello,
> >
> > I had a very high I/O load on process tlsmgr because the smtp_scache and
> > smtpd_scache files are written to often (smtp_scache.db ~70mb) .
> >
> > data_directory = /var/lib/postfix
> > smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
> > smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
> >
> > moving /var/lib/postfix to a tmpfs filesystem solved my problem for now,
> > but i am looking for a better solution.
> > I noticed that postfix also supports memcache as lookup table
> > (http://www.postfix.org/DATABASE_README.html)
> > is this also supported for smtp_tls_session_cache_database ? Can anyone
> > show me a config example?
>
> memcache should work just fine. Specify memcache:/configfile instead
> of btree:/pathname. The contents of the configfile are documented
> in memcache_table(5). There is no need to change the default ttl
> of 3600 seconds.
>
> In main.cf, set smtpd_tls_session_cache_timeout=0 and
> smtp_tls_session_cache_timeout=0. Expiration is done in the memcache
> server.

IMPORTANT NOTE: Setting the timeout to zero, disables session
caching in the SMTP server.  Short lifetimes also bound the session
validity at the SSL library layer, this is NOT just a database
timeout).

1.  Just disable the disk based SMTP server session cache,

        # Only issue session tickets, let the client do the
        # caching
        #
        smtpd_tls_session_cache_database =

    This requires Postfix >= 2.10 and OpenSSL >= 1.0.0.  You
    might find that entirely solves the problem.

2.  You can reduce the session cache lifetime on the client, which
    should reduce the file size.

    smtpd_tls_session_cache_

3.  DO NOT use TCP to offload the SMTP session cache to remote
    memcache servers.  The session cache contains sensitive session
    master keys that would enable an attacker to decrypt your TLSs
    traffic.  When have some time, I'll add code to tlsmgr(8) to
    encrypt cache entries before storing them into the database
    and to decrypt them when they are read back.  Using the same
    key rollover code as for SMTP server session ticket keys.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Matthias Schneider
> On Fri, Nov 14, 2014 at 10:10:52AM -0500, Wietse Venema wrote:
>
>> Matthias Schneider:
>>> Hello,
>>>
>>> I had a very high I/O load on process tlsmgr because the smtp_scache and
>>> smtpd_scache files are written to often (smtp_scache.db ~70mb) .
>>>
>>> data_directory = /var/lib/postfix
>>> smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
>>> smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
>>>
>>> moving /var/lib/postfix to a tmpfs filesystem solved my problem for now,
>>> but i am looking for a better solution.
>>> I noticed that postfix also supports memcache as lookup table
>>> (http://www.postfix.org/DATABASE_README.html)
>>> is this also supported for smtp_tls_session_cache_database ? Can anyone
>>> show me a config example?
>> memcache should work just fine. Specify memcache:/configfile instead
>> of btree:/pathname. The contents of the configfile are documented
>> in memcache_table(5). There is no need to change the default ttl
>> of 3600 seconds.
>>
>> In main.cf, set smtpd_tls_session_cache_timeout=0 and
>> smtp_tls_session_cache_timeout=0. Expiration is done in the memcache
>> server.
> IMPORTANT NOTE: Setting the timeout to zero, disables session
> caching in the SMTP server.  Short lifetimes also bound the session
> validity at the SSL library layer, this is NOT just a database
> timeout).
>
> 1.  Just disable the disk based SMTP server session cache,
>
> # Only issue session tickets, let the client do the
> # caching
> #
> smtpd_tls_session_cache_database =
>
>      This requires Postfix >= 2.10 and OpenSSL >= 1.0.0.  You
>      might find that entirely solves the problem.
>
> 2.  You can reduce the session cache lifetime on the client, which
>      should reduce the file size.
>
>      smtpd_tls_session_cache_
>
> 3.  DO NOT use TCP to offload the SMTP session cache to remote
>      memcache servers.  The session cache contains sensitive session
>      master keys that would enable an attacker to decrypt your TLSs
>      traffic.  When have some time, I'll add code to tlsmgr(8) to
>      encrypt cache entries before storing them into the database
>      and to decrypt them when they are read back.  Using the same
>      key rollover code as for SMTP server session ticket keys.
>


Hello Viktor,

since my server is more smtp client than smtpd i have to tune the
smtp_tls_session_cache_database setting, do you recommend to set
smtp_tls_session_cache_database to empty or using a memcache server for
performance increase?

I am running the latest postfix 2.11.3 version and openssl >1.0.1

I'll use Unix socket or ::1 for security.

Thank you!
Matthias
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Viktor Dukhovni
On Fri, Nov 14, 2014 at 05:22:34PM +0100, Matthias Schneider wrote:

> Hello Viktor,
>
> since my server is more smtp client than smtpd i have to tune the
> smtp_tls_session_cache_database setting, do you recommend to set
> smtp_tls_session_cache_database to empty or using a memcache server for
> performance increase?

Your real problem is that you're sending *a lot* to some servers
that promise session caching, but fail to actually cache sessions.

If that's the vast majority of your client connecions, you can
indeed simply disable TLS session caching.  Otherwise, you
can configure a clone SMTP transport with:

    nocache unix ... smtp
        -o smtp_tls_session_cache_database=

and route mail for certain nexthop domains via that transport.

Do determine which domains support TLS session caching, you can
use posttls-finger (included with 2.11.3 source, make sure to build
with -DUSE_TLS):

    $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache gmail.com
    posttls-finger: looking for session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 in memory cache
    posttls-finger: save session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 to memory cache
    posttls-finger: Untrusted TLS connection established to gmail-smtp-in.l.google.com[173.194.68.26]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
    posttls-finger: Reconnecting after 2 seconds
    posttls-finger: looking for session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 in memory cache
    posttls-finger: reloaded session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 from memory cache
    posttls-finger: gmail-smtp-in.l.google.com[173.194.68.26]:25: Reusing old session
    posttls-finger: Untrusted TLS connection established to gmail-smtp-in.l.google.com[173.194.68.26]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
    posttls-finger: Maximum reconnect count reached.

    $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache postfix.org
    posttls-finger: looking for session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A in memory cache
    posttls-finger: save session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A to memory cache
    posttls-finger: Anonymous TLS connection established to mail.cloud9.net[168.100.1.7]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
    posttls-finger: Server is anonymous
    posttls-finger: Reconnecting after 2 seconds
    posttls-finger: looking for session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A in memory cache
    posttls-finger: reloaded session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A from memory cache
    posttls-finger: mail.cloud9.net[168.100.1.7]:25: Reusing old session
    posttls-finger: Anonymous TLS connection established to mail.cloud9.net[168.100.1.7]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
    posttls-finger: Maximum reconnect count reached.

So gmail.com and postfix.org offer and actually reuses sessions,  On the
other hand, storing hotmail, AOL or Yahoo sessions is just a waste
of I/O, since they are rarely if ever reusable.

    $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache hotmail.com
    posttls-finger: looking for session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 in memory cache
    posttls-finger: save session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 to memory cache
    posttls-finger: Untrusted TLS connection established to mx4.hotmail.com[207.46.8.199]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)
    posttls-finger: Reconnecting after 2 seconds
    posttls-finger: looking for session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E in memory cache
    posttls-finger: save session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E to memory cache
    posttls-finger: Untrusted TLS connection established to mx4.hotmail.com[207.46.8.199]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)
    posttls-finger: Maximum reconnect count reached.

    $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache aol.com
    posttls-finger: looking for session [64.12.91.196]:25&3A40B308D4D0C919F0578116E0DFF7530391F4D4118A674626484D27CD0BE2B0 in memory cache
    posttls-finger: save session [64.12.91.196]:25&3A40B308D4D0C919F0578116E0DFF7530391F4D4118A674626484D27CD0BE2B0 to memory cache
    posttls-finger: Anonymous TLS connection established to mailin-03.mx.aol.com[64.12.91.196]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
    posttls-finger: Server is anonymous
    posttls-finger: Reconnecting after 2 seconds
    posttls-finger: looking for session [64.12.91.196]:25&8B1DE87019CFE68E0285ED2F0C20C7C6BE314DDC39B043FC6ADD3D96B5A5A9FA in memory cache
    posttls-finger: save session [64.12.91.196]:25&8B1DE87019CFE68E0285ED2F0C20C7C6BE314DDC39B043FC6ADD3D96B5A5A9FA to memory cache
    posttls-finger: Anonymous TLS connection established to mailin-03.mx.aol.com[64.12.91.196]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
    posttls-finger: Maximum reconnect count reached.

    $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache yahoo.com
    posttls-finger: looking for session [98.136.217.203]:25&A53B68A7BDE12105B8A974681A2BE9A8A2F4A1530E5B7D4413CD919A848327D2 in memory cache
    posttls-finger: save session [98.136.217.203]:25&A53B68A7BDE12105B8A974681A2BE9A8A2F4A1530E5B7D4413CD919A848327D2 to memory cache
    posttls-finger: Untrusted TLS connection established to mta7.am0.yahoodns.net[98.136.217.203]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
    posttls-finger: Reconnecting after 2 seconds
    posttls-finger: looking for session [98.136.217.203]:25&2DF2C714B51AFEFD66564D31C2E2C5A06420388866B4809A03F1CF6EB4642FBA in memory cache
    posttls-finger: save session [98.136.217.203]:25&2DF2C714B51AFEFD66564D31C2E2C5A06420388866B4809A03F1CF6EB4642FBA to memory cache
    posttls-finger: Untrusted TLS connection established to mta7.am0.yahoodns.net[98.136.217.203]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
    posttls-finger: Maximum reconnect count reached.

Large sites are only likely to have a working session cache if they
support session tickets.  Google wrote the RFC on TLS session
tickets.  Not surprisingly, they have a working implementation.

The polite thing to do is to leave caching enabled by default (works
with destinations that do it right).  And disable it just for the
few largest that do it wrong (offer sessions but don't generally
resume them).  This requires a transport table entry for each such
"disabled" domain and a master.cf entry as above.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Matthias Schneider
> On Fri, Nov 14, 2014 at 05:22:34PM +0100, Matthias Schneider wrote:
>
>> Hello Viktor,
>>
>> since my server is more smtp client than smtpd i have to tune the
>> smtp_tls_session_cache_database setting, do you recommend to set
>> smtp_tls_session_cache_database to empty or using a memcache server for
>> performance increase?
> Your real problem is that you're sending *a lot* to some servers
> that promise session caching, but fail to actually cache sessions.
>
> If that's the vast majority of your client connecions, you can
> indeed simply disable TLS session caching.  Otherwise, you
> can configure a clone SMTP transport with:
>
>      nocache unix ... smtp
> -o smtp_tls_session_cache_database=
>
> and route mail for certain nexthop domains via that transport.
>
> Do determine which domains support TLS session caching, you can
> use posttls-finger (included with 2.11.3 source, make sure to build
> with -DUSE_TLS):
>
>      $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache gmail.com
>      posttls-finger: looking for session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 in memory cache
>      posttls-finger: save session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 to memory cache
>      posttls-finger: Untrusted TLS connection established to gmail-smtp-in.l.google.com[173.194.68.26]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
>      posttls-finger: Reconnecting after 2 seconds
>      posttls-finger: looking for session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 in memory cache
>      posttls-finger: reloaded session [173.194.68.26]:25&02EBBDE489A3B055A64609B3919EC8E59AF25682182DD7E82D4BAF3FDF388585 from memory cache
>      posttls-finger: gmail-smtp-in.l.google.com[173.194.68.26]:25: Reusing old session
>      posttls-finger: Untrusted TLS connection established to gmail-smtp-in.l.google.com[173.194.68.26]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
>      posttls-finger: Maximum reconnect count reached.
>
>      $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache postfix.org
>      posttls-finger: looking for session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A in memory cache
>      posttls-finger: save session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A to memory cache
>      posttls-finger: Anonymous TLS connection established to mail.cloud9.net[168.100.1.7]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
>      posttls-finger: Server is anonymous
>      posttls-finger: Reconnecting after 2 seconds
>      posttls-finger: looking for session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A in memory cache
>      posttls-finger: reloaded session [168.100.1.7]:25&6BED8D59EAC9E109BE27219B31481DBD79420A11FF7FFD9BF75E55B7E80DE14A from memory cache
>      posttls-finger: mail.cloud9.net[168.100.1.7]:25: Reusing old session
>      posttls-finger: Anonymous TLS connection established to mail.cloud9.net[168.100.1.7]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
>      posttls-finger: Maximum reconnect count reached.
>
> So gmail.com and postfix.org offer and actually reuses sessions,  On the
> other hand, storing hotmail, AOL or Yahoo sessions is just a waste
> of I/O, since they are rarely if ever reusable.
>
>      $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache hotmail.com
>      posttls-finger: looking for session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 in memory cache
>      posttls-finger: save session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 to memory cache
>      posttls-finger: Untrusted TLS connection established to mx4.hotmail.com[207.46.8.199]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)
>      posttls-finger: Reconnecting after 2 seconds
>      posttls-finger: looking for session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E in memory cache
>      posttls-finger: save session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E to memory cache
>      posttls-finger: Untrusted TLS connection established to mx4.hotmail.com[207.46.8.199]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)
>      posttls-finger: Maximum reconnect count reached.
>
>      $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache aol.com
>      posttls-finger: looking for session [64.12.91.196]:25&3A40B308D4D0C919F0578116E0DFF7530391F4D4118A674626484D27CD0BE2B0 in memory cache
>      posttls-finger: save session [64.12.91.196]:25&3A40B308D4D0C919F0578116E0DFF7530391F4D4118A674626484D27CD0BE2B0 to memory cache
>      posttls-finger: Anonymous TLS connection established to mailin-03.mx.aol.com[64.12.91.196]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
>      posttls-finger: Server is anonymous
>      posttls-finger: Reconnecting after 2 seconds
>      posttls-finger: looking for session [64.12.91.196]:25&8B1DE87019CFE68E0285ED2F0C20C7C6BE314DDC39B043FC6ADD3D96B5A5A9FA in memory cache
>      posttls-finger: save session [64.12.91.196]:25&8B1DE87019CFE68E0285ED2F0C20C7C6BE314DDC39B043FC6ADD3D96B5A5A9FA to memory cache
>      posttls-finger: Anonymous TLS connection established to mailin-03.mx.aol.com[64.12.91.196]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
>      posttls-finger: Maximum reconnect count reached.
>
>      $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache yahoo.com
>      posttls-finger: looking for session [98.136.217.203]:25&A53B68A7BDE12105B8A974681A2BE9A8A2F4A1530E5B7D4413CD919A848327D2 in memory cache
>      posttls-finger: save session [98.136.217.203]:25&A53B68A7BDE12105B8A974681A2BE9A8A2F4A1530E5B7D4413CD919A848327D2 to memory cache
>      posttls-finger: Untrusted TLS connection established to mta7.am0.yahoodns.net[98.136.217.203]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
>      posttls-finger: Reconnecting after 2 seconds
>      posttls-finger: looking for session [98.136.217.203]:25&2DF2C714B51AFEFD66564D31C2E2C5A06420388866B4809A03F1CF6EB4642FBA in memory cache
>      posttls-finger: save session [98.136.217.203]:25&2DF2C714B51AFEFD66564D31C2E2C5A06420388866B4809A03F1CF6EB4642FBA to memory cache
>      posttls-finger: Untrusted TLS connection established to mta7.am0.yahoodns.net[98.136.217.203]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
>      posttls-finger: Maximum reconnect count reached.
>
> Large sites are only likely to have a working session cache if they
> support session tickets.  Google wrote the RFC on TLS session
> tickets.  Not surprisingly, they have a working implementation.
>
> The polite thing to do is to leave caching enabled by default (works
> with destinations that do it right).  And disable it just for the
> few largest that do it wrong (offer sessions but don't generally
> resume them).  This requires a transport table entry for each such
> "disabled" domain and a master.cf entry as above.
>
Thank you Viktor, I'll have a try.
It would be great if there would be some kind of TLS debugging to log
successful and not successful TLS session reusing.
To have a statistic here and discovering bad destination domains.
(feature request :-) )

Matthias
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Viktor Dukhovni
In reply to this post by Viktor Dukhovni
On Fri, Nov 14, 2014 at 05:20:14PM +0000, Viktor Dukhovni wrote:

> So gmail.com and postfix.org offer and actually reuses sessions,  On the
> other hand, storing hotmail, AOL or Yahoo sessions is just a waste
> of I/O, since they are rarely if ever reusable.
>
>     $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache hotmail.com
>     posttls-finger: looking for session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 in memory cache
>     posttls-finger: save session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 to memory cache
>     posttls-finger: Reconnecting after 2 seconds
>     posttls-finger: looking for session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E in memory cache
>     posttls-finger: save session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E to memory cache
>     posttls-finger: Untrusted TLS connection established to mx4.hotmail.com[207.46.8.199]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)

Note that the cache lookup key is different for the second connection.

To work with split caches for hosts behind load-balancers, Postfix
includes the server name from the remote EHLO response in the lookup
key, that way a small number of hosts behind a single IP address
each get their own cache entry.

This does not work very well when the load-balancer front-ends
hundreds or thousands of hosts, you almost rarely connect to
the same host again, while caching lots of discrete sessions.

The gmail "cloud" has a fixed EHLO server name:

    posttls-finger: < 250-mx.google.com at your service

which works well with Postfix, because they have a unified session
cache with session tickets and shared keys across the cloud.  While
consecutive hotmail connections (to the same IP) may yield:

    posttls-finger: < 250-BAY004-MC2F21.hotmail.com (3.20.0.138) Hello [192.0.2.1]
    posttls-finger: < 250-BAY004-MC2F35.hotmail.com (3.20.0.138) Hello [192.0.2.1]

which results in a different cache slot.  I'd have to write new
code to determine whether suppressing the EHLO response session
lookup key "salt" would lead to better cache utilization for the
various large "cloud provider" domains.

Determining this dynamically, would require keeping some statistics
on cache re-use by IP, and switching from salted to unsalted to
off, if cache re-use is sufficiently poor.  The code for that would
require some care.  Not promising anything any time soon.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Viktor Dukhovni
In reply to this post by Matthias Schneider
On Fri, Nov 14, 2014 at 06:41:52PM +0100, Matthias Schneider wrote:

> It would be great if there would be some kind of TLS debugging to log
> successful and not successful TLS session reusing.

The present TLS log levels are too coarse.  You'd get the data in
question at log level 2, but so much other logging along with it,
that your system performance would degrade considerably under
logging I/O pressure.

So for now, you'll need to tune by hand for a few large receiving
domains.  As for hotmail, it seems unlikely that "unsalted" sessions
would work better, they don't support session tickets:

    posttls-finger: SSL_connect:before/connect initialization
    posttls-finger: SSL_connect:SSLv2/v3 write client hello A
    posttls-finger: SSL_connect:SSLv3 read server hello A
    posttls-finger: SSL_connect:SSLv3 read server certificate A
    posttls-finger: SSL_connect:SSLv3 read server key exchange A
    posttls-finger: SSL_connect:SSLv3 read server done A
    posttls-finger: SSL_connect:SSLv3 write client key exchange A
    posttls-finger: SSL_connect:SSLv3 write change cipher spec A
    posttls-finger: SSL_connect:SSLv3 write finished A
    posttls-finger: SSL_connect:SSLv3 flush data
    posttls-finger: SSL_connect:SSLv3 read finished A

so are unlikely to have a unified cross-server cache.  Compare with:

    posttls-finger: SSL_connect:before/connect initialization
    posttls-finger: SSL_connect:SSLv2/v3 write client hello A
    posttls-finger: SSL_connect:SSLv3 read server hello A
    posttls-finger: SSL_connect:SSLv3 read server certificate A
    posttls-finger: SSL_connect:SSLv3 read server key exchange A
    posttls-finger: SSL_connect:SSLv3 read server done A
    posttls-finger: SSL_connect:SSLv3 write client key exchange A
    posttls-finger: SSL_connect:SSLv3 write change cipher spec A
    posttls-finger: SSL_connect:SSLv3 write finished A
    posttls-finger: SSL_connect:SSLv3 flush data
 -> posttls-finger: SSL_connect:SSLv3 read server session ticket A
    posttls-finger: SSL_connect:SSLv3 read finished A

for Gmail (these messages are from "posttls-finger -Ldebug").

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Viktor Dukhovni
On Fri, Nov 14, 2014 at 05:58:04PM +0000, Viktor Dukhovni wrote:

> So for now, you'll need to tune by hand for a few large receiving
> domains.  As for hotmail, it seems unlikely that "unsalted" sessions
> would work better, they don't support session tickets:
>
>     posttls-finger: SSL_connect:before/connect initialization
>     posttls-finger: SSL_connect:SSLv2/v3 write client hello A
>     posttls-finger: SSL_connect:SSLv3 read server hello A
>     posttls-finger: SSL_connect:SSLv3 read server certificate A
>     posttls-finger: SSL_connect:SSLv3 read server key exchange A
>     posttls-finger: SSL_connect:SSLv3 read server done A
>     posttls-finger: SSL_connect:SSLv3 write client key exchange A
>     posttls-finger: SSL_connect:SSLv3 write change cipher spec A
>     posttls-finger: SSL_connect:SSLv3 write finished A
>     posttls-finger: SSL_connect:SSLv3 flush data
>     posttls-finger: SSL_connect:SSLv3 read finished A
>
> so are unlikely to have a unified cross-server cache.  Compare with:

The situation may be more promising for Yahoo:

    posttls-finger: SSL_connect:before/connect initialization
    posttls-finger: SSL_connect:SSLv2/v3 write client hello A
    posttls-finger: SSL_connect:SSLv3 read server hello A
    posttls-finger: SSL_connect:SSLv3 read server certificate A
    posttls-finger: SSL_connect:SSLv3 read server key exchange A
    posttls-finger: SSL_connect:SSLv3 read server done A
    posttls-finger: SSL_connect:SSLv3 write client key exchange A
    posttls-finger: SSL_connect:SSLv3 write change cipher spec A
    posttls-finger: SSL_connect:SSLv3 write finished A
    posttls-finger: SSL_connect:SSLv3 flush data
    posttls-finger: SSL_connect:SSLv3 read server session ticket A
    posttls-finger: SSL_connect:SSLv3 read finished A

Here session reuse would perhaps work better without the "salt",
but I don't have command-line code at hand to find out.  (However,
you could test witp smtp_reply_filter):

    http://www.postfix.org/postconf.5.html#smtp_reply_filter

    Suitable PCRE table:

        /^(250-mta)\d+(\.mail\..*\.yahoo\.com[ \t\r\n].*)/ $1-N$2

Bash example:

    $ postmap -q \
        "$(printf "250-mta1377.mail.ne1.yahoo.com\r\n250-PIPELINING\r\n250-SIZE 41943040\r\n250 8BITMIME\r\n")" \
        pcre:<(echo '/^(250-mta)\d+(\.mail\..*\.yahoo\.com[ \t\r\n].*)/ $1-N$2')
    250-mta-N.mail.ne1.yahoo.com
    250-PIPELINING
    250-SIZE 41943040
    250 8BITMIME

That would lead to a lot fewer cache entries for Yahoo, whether
they end up re-used or not.  One per data-centre, rather than one
per MTA.

A similar mapping for the hotmail MTA names, could also reduce I/O
load by re-cycling a smaller number of cache entries, rather than
constantly writing new ones.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: tlsmgr high io load because of session cache

Emmanuel Fusté-2
In reply to this post by Viktor Dukhovni
Le 14/11/2014 18:47, Viktor Dukhovni a écrit :

> On Fri, Nov 14, 2014 at 05:20:14PM +0000, Viktor Dukhovni wrote:
>
>> So gmail.com and postfix.org offer and actually reuses sessions,  On the
>> other hand, storing hotmail, AOL or Yahoo sessions is just a waste
>> of I/O, since they are rarely if ever reusable.
>>
>>      $ posttls-finger -c -r 2 -m 1 -lmay -Lsummary,cache hotmail.com
>>      posttls-finger: looking for session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 in memory cache
>>      posttls-finger: save session [207.46.8.199]:25&B3615E5BC0C51EF280EB79AC8C2D83BB5062B2BE73D21E5CD2AE6E5577D99934 to memory cache
>>      posttls-finger: Reconnecting after 2 seconds
>>      posttls-finger: looking for session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E in memory cache
>>      posttls-finger: save session [207.46.8.199]:25&7364A28B331EC120944E55777F8A2AF16784CDC5840C1BA6EF5FE028C66F993E to memory cache
>>      posttls-finger: Untrusted TLS connection established to mx4.hotmail.com[207.46.8.199]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)
> Note that the cache lookup key is different for the second connection.
>
> To work with split caches for hosts behind load-balancers, Postfix
> includes the server name from the remote EHLO response in the lookup
> key, that way a small number of hosts behind a single IP address
> each get their own cache entry.
>
> This does not work very well when the load-balancer front-ends
> hundreds or thousands of hosts, you almost rarely connect to
> the same host again, while caching lots of discrete sessions.
>
> The gmail "cloud" has a fixed EHLO server name:
>
>      posttls-finger: < 250-mx.google.com at your service
>
> which works well with Postfix, because they have a unified session
> cache with session tickets and shared keys across the cloud.  While
> consecutive hotmail connections (to the same IP) may yield:
>
>      posttls-finger: < 250-BAY004-MC2F21.hotmail.com (3.20.0.138) Hello [192.0.2.1]
>      posttls-finger: < 250-BAY004-MC2F35.hotmail.com (3.20.0.138) Hello [192.0.2.1]
>
> which results in a different cache slot.  I'd have to write new
> code to determine whether suppressing the EHLO response session
> lookup key "salt" would lead to better cache utilization for the
> various large "cloud provider" domains.
>
> Determining this dynamically, would require keeping some statistics
> on cache re-use by IP, and switching from salted to unsalted to
> off, if cache re-use is sufficiently poor.  The code for that would
> require some care.  Not promising anything any time soon.
>
Hello Viktor,

Side questions about session cache.
If I have some postfix smtpd physical servers behind a load-balancer,
how could I setup them to be TLS client session cache friendly ?
- same EHLO
- same keys (and certificate ?).
- shared memcached session cache
and the same questions but to be session ticket friendly: no need of the
shared memcached, but for the other things ?

Emmanuel.