MDB_MAP_FULL: Environment mapsize limit reached

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

MDB_MAP_FULL: Environment mapsize limit reached

A. Schulze

Hello,

running postfix-3.4.7 on Debian 10 I found the following warning in my logs:

postfix/tlsmgr[705]: warning:  
lmdb:/var/lib/postfix/smtp_tls_session_cache is unavailable. open  
database /var/lib/postfix/smtp_tls_session_cache.lmdb: MDB_MAP_FULL:  
Environment mapsize limit reached

on two moderate loaded servers there where thousands of them, each  
generated by the same tlsmgr PID
Messages got delivered without other trouble. I guess, simply reusing  
TLS sessions did not happen.
The database files /var/lib/postfix/smtp_tls_session_cache.lmdb had  
sizes of ~20 and ~32 MB.

I've other instances with smtp_tls_session_cache of 400 MB but there I  
use old BDB format and postfix-3.4.6.
I've also other instances where also LMDB is used for  
smtp_tls_session_cache and the files are ~40 MB large.
But there, postfix-3.4.5 is installed.

smtp_tls_connection_reuse is explicit set to "yes" and lmdb_map_size  
is at it's default ( ~16 MB ) on all of my instances.

For now "rm /var/lib/postfix/smtp_tls_session_cache.lmdb; postfix  
reload" made the warning go away.

Wild guess: there where some code changes for tls_fast_shutdown_enable  
between 3.4.5 and 3.4.7
(20190615 and 20190723)

Andreas


Reply | Threaded
Open this post in threaded view
|

Re: MDB_MAP_FULL: Environment mapsize limit reached

Wietse Venema
A. Schulze:
>
> Hello,
>
> running postfix-3.4.7 on Debian 10 I found the following warning in my logs:
>
> postfix/tlsmgr[705]: warning:  
> lmdb:/var/lib/postfix/smtp_tls_session_cache is unavailable. open  
> database /var/lib/postfix/smtp_tls_session_cache.lmdb: MDB_MAP_FULL:  
> Environment mapsize limit reached

Postfix has a bunch of worarounds for LMDB, that were working by
the time that LMDB support was implemented, but LMDB may have been
'improved', and it may be time to review those workaroundso

In the meantime, you can set a larger limit.

> Wild guess: there where some code changes for tls_fast_shutdown_enable  
> between 3.4.5 and 3.4.7
> (20190615 and 20190723)

Instead of speculating you could set tls_fast_shutdown_enable to
revert the old behavior and see if it makes a difference.

So I'll wait for that to be confirmed. No need to drop everyting
I am doing.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: MDB_MAP_FULL: Environment mapsize limit reached

Wietse Venema
Wietse Venema:

> A. Schulze:
> >
> > Hello,
> >
> > running postfix-3.4.7 on Debian 10 I found the following warning in my logs:
> >
> > postfix/tlsmgr[705]: warning:  
> > lmdb:/var/lib/postfix/smtp_tls_session_cache is unavailable. open  
> > database /var/lib/postfix/smtp_tls_session_cache.lmdb: MDB_MAP_FULL:  
> > Environment mapsize limit reached

Here's an example of adding entries incrementally to an LMDB table,
similar to what tlsmgr does as it adds entries to a TLS session
cache:

    % touch /tmp/main.cf
    % postconf -c /tmp lmdb_map_size=1000
    % rm /tmp/size-test*
    % sed 's/.*/& &/' /usr/share/dict/words | /usr/bin/time postmap -fic /tmp lmdb:/tmp/size-test
          216.47 real         3.50 user        13.21 sys
    % ls -l /tmp/size-test*
    -rw-r--r--  1 wietse  wheel  8257536 Jan  9 15:20 /tmp/size-test.lmdb

With "-i", when postmap runs into a 'map full' error, it increases
the mapsize limit and tries to add the entry again. On my system
with lmdb-0.9.11 that works perfectly, extending the table from
~1000 to 8257536.

If this test fails for you then that is an indication that
LMDB has changed.

One difference is that my test adds small items (a word from a
dictionary), while you are adding larger items (TLS session
information).

        Wietse

> Postfix has a bunch of worarounds for LMDB, that were working by
> the time that LMDB support was implemented, but LMDB may have been
> 'improved', and it may be time to review those workaroundso
>
> In the meantime, you can set a larger limit.
>
> > Wild guess: there where some code changes for tls_fast_shutdown_enable  
> > between 3.4.5 and 3.4.7
> > (20190615 and 20190723)
>
> Instead of speculating you could set tls_fast_shutdown_enable to
> revert the old behavior and see if it makes a difference.
>
> So I'll wait for that to be confirmed. No need to drop everyting
> I am doing.
>
> Wietse
>
Reply | Threaded
Open this post in threaded view
|

Re: MDB_MAP_FULL: Environment mapsize limit reached

Wietse Venema
Wietse Venema:

> Wietse Venema:
> > A. Schulze:
> > >
> > > Hello,
> > >
> > > running postfix-3.4.7 on Debian 10 I found the following warning in my logs:
> > >
> > > postfix/tlsmgr[705]: warning:  
> > > lmdb:/var/lib/postfix/smtp_tls_session_cache is unavailable. open  
> > > database /var/lib/postfix/smtp_tls_session_cache.lmdb: MDB_MAP_FULL:  
> > > Environment mapsize limit reached
>
> Here's an example of adding entries incrementally to an LMDB table,
> similar to what tlsmgr does as it adds entries to a TLS session
> cache:
>
>     % touch /tmp/main.cf
>     % postconf -c /tmp lmdb_map_size=1000
>     % rm /tmp/size-test*
>     % sed 's/.*/& &/' /usr/share/dict/words | /usr/bin/time postmap -fic /tmp lmdb:/tmp/size-test
>  216.47 real         3.50 user        13.21 sys
>     % ls -l /tmp/size-test*
>     -rw-r--r--  1 wietse  wheel  8257536 Jan  9 15:20 /tmp/size-test.lmdb
>
> With "-i", when postmap runs into a 'map full' error, it increases
> the mapsize limit and tries to add the entry again. On my system
> with lmdb-0.9.11 that works perfectly, extending the table from
> ~1000 to 8257536.
>
> If this test fails for you then that is an indication that
> LMDB has changed.

This test completes fine with LMDB 0.9.23 (built on Fedora 30).
Starting with a small LMDB size limit, "postmap -i" keeps raising
the limimt as it adds items to the LMDB table. This is the same
Postfix library code path that is used with adding items to the TLS
session cache, only the sizes of the items may differ.

I suspect that you run Postfix daemons with a low memmory limit.
Don't do that with memory-mapped databases.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: MDB_MAP_FULL: Environment mapsize limit reached

Wietse Venema
Wietse Venema:
> This test completes fine with LMDB 0.9.23 (built on Fedora 30).
> Starting with a small LMDB size limit, "postmap -i" keeps raising
> the limimt as it adds items to the LMDB table. This is the same
> Postfix library code path that is used with adding items to the TLS
> session cache, only the sizes of the items may differ.
>
> I suspect that you run Postfix daemons with a low memmory limit.
> Don't do that with memory-mapped databases.

or it could be af file size limit issue - like the
message file size limit.

        Wietse