Ignoring a failing dictionary ?

[Ganael Laplanche]

Hello,

Maybe I am missing something but I cannot find a way to silently ignore a
failing dictionary in a chain, such as:

virtual_alias_maps =    ldap:/etc/postfix/myldap.cf,
                        hash:/etc/postfix/mymap

where I would like my hash map to be checked even if ldap is failing.

Is there such a mechanism in Postfix (ala 'unavail=continue' in Linux'
nsswitch) ? Maybe a specific meta lookup table (such as pipemap or unionmap)
could be used here, but I cannot find a suitable one for that specific
purpose.

Thanks in advance,
Best regards,

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes

[Wietse Venema]

Ganael Laplanche:
> Hello,
>
> Maybe I am missing something but I cannot find a way to silently ignore a
> failing dictionary in a chain, such as:

There is no such feature.

> virtual_alias_maps =    ldap:/etc/postfix/myldap.cf,
>                         hash:/etc/postfix/mymap

Ignoring errors would result in misdelivery of email. You may have
expectations that it is OK for software to randomly misdeliver
email, but that is not how Postfix works.

If LDAP cannot handle many concurrent connections, use proxymap
like everyone does with mysql and the like, or hide it under
a memcache_table.

http://www.postfix.org/proxymap.8.html

    virtual_alias_maps = proxy:ldap:/etc/postfix/myldap.cf,
                         hash:/etc/postfix/mymap

http://www.postfix.org/memcache_table.5.html

    virtual_alias_maps = memcache:/etc/postfix/memcache-aliases.cf

        Wietse

[Ganael Laplanche]

On Tuesday, January 19, 2021 1:59:42 PM CET Wietse Venema wrote:

Hello Wietse,

Thanks for your reply,

> > Ignoring errors would result in misdelivery of email. You may have
> expectations that it is OK for software to randomly misdeliver
> email, but that is not how Postfix works.

Well, I don't expect mail to be misdelivered, of course :)

Misdelivery would not happen if next dictionary(ies) have similar contents
(and this is sysadmin's work to ensure it is the case). Having such a
possibility could allow several tries on different remote backends before
finally falling back on a local one.

> If LDAP cannot handle many concurrent connections, use proxymap
> like everyone does with mysql and the like, or hide it under
> a memcache_table. [...]

Proxymap won't help here as our concern here is not related to LDAP server
overload.

Let me explain : our LDAP servers are populated by 3rd party tools (and team)
that might (in theory) fail and disable accounts by mistake. Of course, this
is not Postfix' problem *but* we would like to avoid such a situation where
many accounts have disappeared from the directory and where we would refuse
mail by mistake.

Our idea was to use our LDAP directory as a first dictionary to quickly handle
address change and new accounts but always have a local fallback (a verified
dump produced every week) as a last resort to continue accepting mail for
accounts that would have been disabled by mistake (in fact, for any disabled
account, for 7 days).

That chain seems good but it would be nice if we could use our local dump
(last dictionary in conf) when LDAP is failing in the chain, because it has
verified contents (+ disabled accounts) that match our directory.

With current available Postfix options, I don't know how to handle such a
configuration (but maybe we just shouldn't do that :p).

> http://www.postfix.org/memcache_table.5.html

Maybe memcache with a *very* long TTL could be used here, but I was looking
for a pseudo-dictionay such as unionmap (maybe something like 'noretrymap')
that would ignore DICT_ERR_RETRY from failing backends and get the first
useable result from child dicts.

Would there be another option ?

Thanks again for your help,
Best regards,

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes

[John Stoffel-2]

>>>>> "Ganael" == Ganael Laplanche <[hidden email]> writes:

Ganael> On Tuesday, January 19, 2021 1:59:42 PM CET Wietse Venema wrote:
Ganael> Hello Wietse,

Ganael> Thanks for your reply,

>> > Ignoring errors would result in misdelivery of email. You may have
>> expectations that it is OK for software to randomly misdeliver
>> email, but that is not how Postfix works.

Ganael> Well, I don't expect mail to be misdelivered, of course :)

Ganael> Misdelivery would not happen if next dictionary(ies) have similar contents
Ganael> (and this is sysadmin's work to ensure it is the case). Having such a
Ganael> possibility could allow several tries on different remote backends before
Ganael> finally falling back on a local one.

>> If LDAP cannot handle many concurrent connections, use proxymap
>> like everyone does with mysql and the like, or hide it under
>> a memcache_table. [...]

Ganael> Proxymap won't help here as our concern here is not related to LDAP server
Ganael> overload.

Ganael> Let me explain : our LDAP servers are populated by 3rd party
Ganael> tools (and team) that might (in theory) fail and disable
Ganael> accounts by mistake. Of course, this is not Postfix' problem
Ganael> *but* we would like to avoid such a situation where many
Ganael> accounts have disappeared from the directory and where we
Ganael> would refuse mail by mistake.

So why not run your own LDAP servers, which pull from those upstream
LDAP servers, and then you can do your own retention rules as you
like?

This way if you don't find <address> in upstream, you can start your 7
day countdown clock and keep accepting email as you want.  If the name
comes back (hopefully not for a different new user!!!) you can just
push/restore the email then.  

Ganael> Our idea was to use our LDAP directory as a first dictionary
Ganael> to quickly handle address change and new accounts but always
Ganael> have a local fallback (a verified dump produced every week) as
Ganael> a last resort to continue accepting mail for accounts that
Ganael> would have been disabled by mistake (in fact, for any disabled
Ganael> account, for 7 days).

You've got some funky requirements, you must have been burned before
by this other group making random changes which lost email.  Not fun.  

[Viktor Dukhovni]

On Tue, Jan 19, 2021 at 03:03:49PM +0100, Ganael Laplanche wrote:

> > http://www.postfix.org/memcache_table.5.html
>
> Maybe memcache with a *very* long TTL could be used here, but I was looking
> for a pseudo-dictionay such as unionmap (maybe something like 'noretrymap')
> that would ignore DICT_ERR_RETRY from failing backends and get the first
> useable result from child dicts.
>
> Would there be another option ?

The only solution that comes to mind is "socketmap".  You can write your
own (for reasonable performance, not single-threaded) server that keeps
track of whether LDAP is producing acceptable answers, and if not, falls
back to stale local sources).

You'll need an LDAP connection pool, and some way to determine whether
the LDAP replies are acceptable (that's the hard part).

--
    Viktor.

[Wietse Venema]

Ganael Laplanche:
> On Tuesday, January 19, 2021 1:59:42 PM CET Wietse Venema wrote:
>
> Hello Wietse,
>
> Thanks for your reply,
>
> > > Ignoring errors would result in misdelivery of email. You may have
> > expectations that it is OK for software to randomly misdeliver
> > email, but that is not how Postfix works.

As explained later, the problem is not with LDAP lookup ERRORS,
it is with LDAP returning a "not found" response (i.e. NOT an error).

That should not be a problem with the proposed configuration:

    virtual_alias_maps = ldap:..., hash:...

If ldap returns "not found", Postfix will search the hash map.
It has always worked that way.

        Wietse

[Curtis Maurand]

> On Jan 19, 2021, at 10:00 AM, Viktor Dukhovni <[hidden email]> wrote:
>
> On Tue, Jan 19, 2021 at 03:03:49PM +0100, Ganael Laplanche wrote:
>
>>> http://www.postfix.org/memcache_table.5.html
>>
>> Maybe memcache with a *very* long TTL could be used here, but I was looking
>> for a pseudo-dictionay such as unionmap (maybe something like 'noretrymap')
>> that would ignore DICT_ERR_RETRY from failing backends and get the first
>> useable result from child dicts.
>>
>> Would there be another option ?
>
> The only solution that comes to mind is "socketmap".  You can write your
> own (for reasonable performance, not single-threaded) server that keeps
> track of whether LDAP is producing acceptable answers, and if not, falls
> back to stale local sources).
>
> You'll need an LDAP connection pool, and some way to determine whether
> the LDAP replies are acceptable (that's the hard part).
>
> --
>    Viktor.

I’ve been watching this thread and all the contortions you are truung to do to get postfix to do something it’s not designed to do.  You can change postfix parameters on the fly.  Please pardon me if you think this might be a stupid question, but LDAP has excellent replication capabilities.  How about multiple LDAP servers in a high availability setup? instead of continual replication, replicate once per week.  If you’re  virtualized, make sure the LDAP guests reside on different hosts.  See Linux heartbeat setups.  that doesn’t satisfy your needing to read from backup data, though.

another idea would be to script it.  write a daemon ( if you need it to check more than once per minute) that checks the ldap server periodically to see that it’s returning correct information. if it is not, the script can issue a postconf command to change ldap servers on the fly.

by that matter a script could simply repopulate that section of its database from the dump file you created and that can also be done on the fly.  importing the ldif file doesn’t take much time, even for large files.  alternatively, you could get the ldap server to reverse its replication direction and simply repopulate the master from the backup server.  the reconciliation will only take a few seconds.

you could also build a quick webapp for your team that could restore a single or multiple accounts from the slave server back to the master with the accidental deletion.

there are ways to automate this in such a way that downtime or rejected mail time is minimal.  there is also the matter of restoring the mailbox, too.

Ther is also btrfs or zfs with snapshot capabilities.  stop the ldap server, restore the database from the snapshot, start the server.  that can also be automated and have it happen in seconds.

—Curtis

Sent from my iPhone

[Ganael Laplanche]

On Tuesday, January 19, 2021 4:40:23 PM CET Curtis Maurand wrote:

Hello,

Thanks to all of you for your answers. I have grouped my replies below:

John> So why not run your own LDAP servers, which pull from those upstream
John> LDAP servers, and then you can do your own retention rules as you
John> like?

We would probably have to avoid OpenLDAP replication mechanism to defer
deletions specifically, which is not something we would like to do
(OpenLDAP does it better than us). We also would like to avoid having to
manage a complex ETL infra to handle that.

What could be done indeed is to defer a full replication (see also Curtis'
suggestion below).

John> You've got some funky requirements, you must have been burned before
John> by this other group making random changes which lost email.  Not fun.

Fortunately, that has not happened yet (the team does a very good job indeed
:)), but we are aware of that risk and prefer anticipating it :)

Viktor> The only solution that comes to mind is "socketmap".  You can write
Viktor> your own (for reasonable performance, not single-threaded) server that
Viktor> keeps track of whether LDAP is producing acceptable answers, and if
Viktor> not, falls back to stale local sources).

Socketmap could be an option but that would require additional development
while Postfix has nearly all the needed things to handle that case. Maybe this
could be easier implemented through a specific dictionary, as I was first
looking for? We still have to think about that but I doubt specific
development is the direction we will take here.

Wietse> As explained later, the problem is not with LDAP lookup ERRORS,
Wietse> it is with LDAP returning a "not found" response (i.e. NOT an error).
Wietse>
Wietse> That should not be a problem with the proposed configuration:
Wietse>
Wietse>     virtual_alias_maps = ldap:..., hash:...

Right, that configuration works and is enough for handling disappearing
addresses. The problem is when LDAP becomes unavailable; while that situation
is not normal and should be treated as a temporary error in most cases, we do
not care in our precise case because everything needed is supposed to be
present in further maps.

As you noticed, I am trying to solve two problems at once :

1) address deletions by mistake (LDAP returning 'not found'); this is OK with
the above configuration

2) as an "improvement" (if ignoring failures can be called that way) to speed
up delivery, do not fail when LDAP is unavailable as we have everything needed
in further hash map

But maybe 2) is stupid. Maybe we should just continue temporarily reject mail
if LDAP is failing (anyway, this is a critical situation for our IT
infrastructure) and keep that configuration. I first thought Postfix would
offer an option to skip a failing backend (provided the sysadmin knows what he
is doing), that's why I started this thread, but maybe that would just be a
bad idea.

Curtis> How about multiple LDAP servers in a high availability setup? instead
Curtis> of continual replication, replicate once per week. [...]

This is the same kind of suggestion John made: using different LDAP servers
with different timing/contents (+ a rollback mechanism if errors occur). The
idea of replicating once a week is interesting. We could first search a main
'up-to-date' server and then fallback to another one with deferred contents.
The drawback is in that case, we would loose our local file fallback that I
find interesting because it cannot supposedly fail (or our machine would have
a serious problem) and that would not solve the main LDAP server being
unavailable (and resulting deferred mail).

Changing LDAP servers (maps) on the fly is interesting, I'll think about that.

Well, no definitive solution, but thanks for all those ideas/feedbacks! That
will help us in our thinking :)

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes

[Jaroslaw Rafa]

Dnia 20.01.2021 o godz. 15:26:39 Ganael Laplanche pisze:
>
> 2) as an "improvement" (if ignoring failures can be called that way) to speed
> up delivery, do not fail when LDAP is unavailable as we have everything needed
> in further hash map

So just try to create some simple "proxy" to your LDAP server that does only
one thing: if LDAP is available, just return the response from LDAP; if not,
just returns "not found". And use that proxy in Postfix in place of your
actual LDAP server. This will require probably some development, but it
shouldn't be very much.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Ganael Laplanche]

On Wednesday, January 20, 2021 3:41:02 PM CET Jaroslaw Rafa wrote:

> So just try to create some simple "proxy" to your LDAP server that does only
> one thing: if LDAP is available, just return the response from LDAP; if
> not, just returns "not found". And use that proxy in Postfix in place of
> your actual LDAP server. This will require probably some development, but
> it shouldn't be very much.

Yes, that would probably be an idea and this is what Viktor was suggesting (I
suppose) whan he was writing about socketmap.

Thanks!

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes

[natan]

Hi
Or use  two ldap - master- slave and use  haproxy like

defaults
 mode tcp
 timeout connect 10s
 timeout server  5500s
 timeout client  5000s
 log /dev/log local5

frontend ldap-389
mode tcp
bind 127.0.0.1:389
option socket-stats
option tcplog
option tcpka
timeout client 500s
default_backend ldap-389-origin

backend ldap-389-origin
server ldap-master xxx.xxx.xxx.xxx:389 check fall 1 rise 2s
server ldap-slave yyy.yyy.yyy.yyy:389 backup check fall 1 rise 2s

mode tcp
balance roundrobin
stick-table type ip size 200k expire 30m
option tcpka
option tcp-check
tcp-check connect port 389
tcp-check send-binary 300c0201 # LDAP bind request "<ROOT>" simple
tcp-check send-binary 01 # message ID
tcp-check send-binary 6007 # protocol Op
tcp-check send-binary 0201 # bind request
tcp-check send-binary 03 # LDAP v3
tcp-check send-binary 04008000 # name, simple authentication
tcp-check expect binary 0a0100 # bind response + result code: success
tcp-check send-binary 30050201034200 # unbind request


On 20.01.2021 15:41, Jaroslaw Rafa wrote:
> Dnia 20.01.2021 o godz. 15:26:39 Ganael Laplanche pisze:
>> 2) as an "improvement" (if ignoring failures can be called that way) to speed
>> up delivery, do not fail when LDAP is unavailable as we have everything needed
>> in further hash map
> So just try to create some simple "proxy" to your LDAP server that does only
> one thing: if LDAP is available, just return the response from LDAP; if not,
> just returns "not found". And use that proxy in Postfix in place of your
> actual LDAP server. This will require probably some development, but it
> shouldn't be very much.

--

[Wietse Venema]

Ganael Laplanche:

> Wietse> As explained later, the problem is not with LDAP lookup ERRORS,
> Wietse> it is with LDAP returning a "not found" response (i.e. NOT an error).
> Wietse>
> Wietse> That should not be a problem with the proposed configuration:
> Wietse>
> Wietse>     virtual_alias_maps = ldap:..., hash:...
>
> Right, that configuration works and is enough for handling disappearing
> addresses. The problem is when LDAP becomes unavailable; while that situation
> is not normal and should be treated as a temporary error in most cases, we do
> not care in our precise case because everything needed is supposed to be
> present in further maps.
>
> As you noticed, I am trying to solve two problems at once :
>
> 1) address deletions by mistake (LDAP returning 'not found'); this is OK with
> the above configuration
>
> 2) as an "improvement" (if ignoring failures can be called that way) to speed
> up delivery, do not fail when LDAP is unavailable as we have everything neede
> in further hash map

It's not going to happen.

Instead, query the hash map BEFORE ldap, and dump ldap periodically
(hourly?) to hash. 'New' users will still be found most of the time.

Just do it carefully.

1) dump ldap > /etc/postfix/alias-from-ldap-new

2) if there are no errors

    postmap hash:/etc/postfix/alias-from-ldap-new

3) if there are no errors

    mv  /etc/postfix/alias-from-ldap-new.db  /etc/postfix/alias-from-ldap.db

4) optional: if there are no errors

    postfix reload

This should be done woth cron and make. For 'make" examples, see
http://www.postfix.org/DATABASE_README.html#safe_db

        Wietse

[Jaroslaw Rafa]

Dnia 20.01.2021 o godz. 15:48:29 Ganael Laplanche pisze:
> > So just try to create some simple "proxy" to your LDAP server that does only
> > one thing: if LDAP is available, just return the response from LDAP; if
> > not, just returns "not found". And use that proxy in Postfix in place of
> > your actual LDAP server. This will require probably some development, but
> > it shouldn't be very much.
>
> Yes, that would probably be an idea and this is what Viktor was suggesting (I
> suppose) whan he was writing about socketmap.

There's no need to use socketmap if your proxy would be an actual proxy, ie.
it would speak LDAP protocol to the connecting clients. You'll need to just
replace your original LDAP server with the proxy in your configuration.

Searching the web for "LDAP proxy", this is one of the first results I
found: https://ldaptor.readthedocs.io/en/latest/cookbook/ldap-proxy.html .
It's a Python library for LDAP, and this documentation presents some example
code for creating LDAP proxies using this library.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Curtis Maurand]

Sent from my iPhone

> On Jan 20, 2021, at 10:27 AM, Jaroslaw Rafa <[hidden email]> wrote:
>
> Dnia 20.01.2021 o godz. 15:48:29 Ganael Laplanche pisze:
>>> So just try to create some simple "proxy" to your LDAP server that does only
>>> one thing: if LDAP is available, just return the response from LDAP; if
>>> not, just returns "not found". And use that proxy in Postfix in place of
>>> your actual LDAP server. This will require probably some development, but
>>> it shouldn't be very much.
>>
>> Yes, that would probably be an idea and this is what Viktor was suggesting (I
>> suppose) whan he was writing about socketmap.
>
> There's no need to use socketmap if your proxy would be an actual proxy, ie.
> it would speak LDAP protocol to the connecting clients. You'll need to just
> replace your original LDAP server with the proxy in your configuration.
>
> Searching the web for "LDAP proxy", this is one of the first results I
> found: https://ldaptor.readthedocs.io/en/latest/cookbook/ldap-proxy.html .
> It's a Python library for LDAP, and this documentation presents some example
> code for creating LDAP proxies using this library.

A thought would be to have a separate process that checks for ldap availability.  if not, issue a postconf command to change the setting on the fly.  when the ldap server comes back, issue a new postconf command to reverse the process to go back to the ldap server.


> --
> Regards,
>   Jaroslaw Rafa
>   [hidden email]
> --
> "In a million years, when kids go to school, they're gonna know: once there
> was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Ganael Laplanche]

On Wednesday, January 20, 2021 6:23:22 PM CET Curtis Maurand wrote:

Hello,

Natan> Or use  two ldap - master- slave and use  haproxy like
Natan> [...]
Natan> tcp-check send-binary 04008000 # name, simple authentication
Natan> tcp-check expect binary 0a0100 # bind response + result code: success
Natan> tcp-check send-binary 30050201034200 # unbind request

Interesting, thanks. Making LDAP not failing is one way of handling the
problem, right :)

Jaroslaw> Searching the web for "LDAP proxy", this is one of the first results
Jaroslaw> I found:
Jaroslaw> https://ldaptor.readthedocs.io/en/latest/cookbook/ldap-proxy.html .
Jaroslaw> It's a Python library for LDAP, and this documentation presents some
Jaroslaw> example code for creating LDAP proxies using this library.

I didn't know Ldaptor, thanks for the link. That would be quite easy to
develop indeed.

Curtis> A thought would be to have a separate process that checks for ldap
Curtis> availability.  if not, issue a postconf command to change the setting
Curtis> on the fly.  when the ldap server comes back, issue a new postconf
Curtis> command to reverse the process to go back to the ldap server.

Yes, I understand the watchdog idea :)

Wietse> Instead, query the hash map BEFORE ldap, and dump ldap periodically
Wietse> (hourly?) to hash. 'New' users will still be found most of the time.
Wietse> Just do it carefully.

Hmmmm... If we put the dump before, we will loose our 7-days window to react.
What could be done maybe is have 2 hash maps and not use LDAP at all : 1 file
generated every hour and our 7-days old dump as a second choice. But this is
not perfect neither as we will have a 1-hour lag regarding new info coming
from LDAP.

Well, I don't know what solution will finally be chosen but all that gives us
plenty to think about. Thanks to you all!

Best regards,

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes

[Viktor Dukhovni]

On Thu, Jan 21, 2021 at 09:58:30AM +0100, Ganael Laplanche wrote:
> On Wednesday, January 20, 2021 6:23:22 PM CET Curtis Maurand wrote:

> Natan> Or use  two ldap - master- slave and use  haproxy like
> Natan> [...]
> Natan> tcp-check send-binary 04008000 # name, simple authentication
> Natan> tcp-check expect binary 0a0100 # bind response + result code: success
> Natan> tcp-check send-binary 30050201034200 # unbind request
>
> Interesting, thanks. Making LDAP not failing is one way of handling the
> problem, right :)

Postfix already rebuilds LDAP connections on error and retries the
search:

    rc = search_st(dict_ldap->ld, vstring_str(base), dict_ldap->scope,
                   vstring_str(query), dict_ldap->result_attributes->argv,
                   dict_ldap->timeout, &res);

    if (rc == LDAP_SERVER_DOWN) {
        if (msg_verbose)
            msg_info("%s: Lost connection for LDAP source %s, reopening",
                     myname, dict_ldap->parser->name);

        dict_ldap_unbind(dict_ldap->ld);
        dict_ldap->ld = DICT_LDAP_CONN(dict_ldap)->conn_ld = 0;
        dict_ldap_connect(dict_ldap);

        /*
         * if dict_ldap_connect() set dict_ldap->dict.error, abort.
         */
        if (dict_ldap->dict.error)
            return (0);

        rc = search_st(dict_ldap->ld, vstring_str(base), dict_ldap->scope,
                     vstring_str(query), dict_ldap->result_attributes->argv,
                       dict_ldap->timeout, &res);

    }

If there's more than one LDAP server, and the one being used crashes,
the new connection will use a different server.

> Jaroslaw> Searching the web for "LDAP proxy", this is one of the first results
> Jaroslaw> I found:
> Jaroslaw> https://ldaptor.readthedocs.io/en/latest/cookbook/ldap-proxy.html .
> Jaroslaw> It's a Python library for LDAP, and this documentation presents some
> Jaroslaw> example code for creating LDAP proxies using this library.
>
> I didn't know Ldaptor, thanks for the link. That would be quite easy to
> develop indeed.
>
> Curtis> A thought would be to have a separate process that checks for ldap
> Curtis> availability.  if not, issue a postconf command to change the setting
> Curtis> on the fly.  when the ldap server comes back, issue a new postconf
> Curtis> command to reverse the process to go back to the ldap server.
>
> Yes, I understand the watchdog idea :)

Postfix already (as a matter of best-practice) supports proxymap(8)
between the smtpd(8), cleanup(8), ... and the LDAP server, just specify
the table as "proxy:ldap:..." instead of "ldap:..."

> Wietse> Instead, query the hash map BEFORE ldap, and dump ldap periodically
> Wietse> (hourly?) to hash. 'New' users will still be found most of the time.
> Wietse> Just do it carefully.
>
> Hmmmm... If we put the dump before, we will loose our 7-days window to react.
> What could be done maybe is have 2 hash maps and not use LDAP at all : 1 file
> generated every hour and our 7-days old dump as a second choice. But this is
> not perfect neither as we will have a 1-hour lag regarding new info coming
> from LDAP.
>
> Well, I don't know what solution will finally be chosen but all that gives us
> plenty to think about. Thanks to you all!

A hash map after LDAP sounds reasonable, as a means to limit the amount
of lost email (1 hour of new user accounts) should LDAP start lying by
reporting "not found" for accounts that should exist.  For the real
LDAP use "proxy:" and configure multiple servers.

--
    Viktor.

[Jaroslaw Rafa]

Dnia 21.01.2021 o godz. 11:15:49 Viktor Dukhovni pisze:
>
> Postfix already (as a matter of best-practice) supports proxymap(8)
> between the smtpd(8), cleanup(8), ... and the LDAP server, just specify
> the table as "proxy:ldap:..." instead of "ldap:..."

But I was thinking about a very specific LDAP proxy, whose purpose is to
return artificial "not found" result if LDAP connection fails.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[John Stoffel-2]

>>>>> "Ganael" == Ganael Laplanche <[hidden email]> writes:

Ganael> Hmmmm... If we put the dump before, we will loose our 7-days
Ganael> window to react.  What could be done maybe is have 2 hash maps
Ganael> and not use LDAP at all : 1 file generated every hour and our
Ganael> 7-days old dump as a second choice. But this is not perfect
Ganael> neither as we will have a 1-hour lag regarding new info coming
Ganael> from LDAP.

So why not populate a new OU from your master production OU, and use
that for all lookups.  The process would then be that when you delete
from the primary OU, it's starts a 7 day count down on the secondary
to finish the deletion.

But other updates/changes would be immediately (or every five minutes
or whatever) propagted to the aliases OU which you do the lookups
against.

So this would give you A) a way to keep email flowing for 7 days, and
B) easy way to recover from accidents.

All you would need to do is change which OU postfix works against.

This lets you use LDAP replication, load sharing, scaling, etc.
Without hacving to muck about with a completely seperate process on
the postfix side to catch changes, because you need to watch you LDAP
OU for new/changed entries and replicate them to the hash table
reliably.

And then the cleanup as well.  Keep it all in LDAP if you can.

John

[Ganael Laplanche]

On Thursday, January 21, 2021 5:15:49 PM CET Viktor Dukhovni wrote:

Hello Viktor,

> Postfix already rebuilds LDAP connections on error and retries the
> search:
> [...]
>
> If there's more than one LDAP server, and the one being used crashes,
> the new connection will use a different server.

That can be useful to limit LDAP problems, thanks.

> A hash map after LDAP sounds reasonable, as a means to limit the amount
> of lost email (1 hour of new user accounts) should LDAP start lying by
> reporting "not found" for accounts that should exist.  For the real
> LDAP use "proxy:" and configure multiple servers.

Well, that was my original idea, and I think it could've been OK, *but*...

...we've shared all suggestions from this thread yesterday with my team. The
main problem we are facing is that we're not willing to defer mail if LDAP is
down ; and even with several LDAP servers, we can never be sure this won't
happen.

With that in mind, it seems the easiest way to have something similar to our
original design would be to have two hash maps chained : the first being a
LDAP dump made every hour or so, and the second one being made and checked
every week. The only downside is that we will have a small lag, but that may
be acceptable.

Well, again, thanks to you all for sharing your ideas :)

Best regards,

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes

[Ganael Laplanche]

On Thursday, January 21, 2021 11:01:44 PM CET John Stoffel wrote:

Hello John,

> So why not populate a new OU from your master production OU, and use
> that for all lookups.  The process would then be that when you delete
> from the primary OU, it's starts a 7 day count down on the secondary
> to finish the deletion.
> [...]

That would have been an idea too, thanks.

> And then the cleanup as well.  Keep it all in LDAP if you can.

Well, it's finally been decided here to swicth to local hash maps instead to
avoid potential mail delaying (see my previous answer).

Thanks anyway for your suggestion :)

Best regards,

--
Ganael Laplanche <[hidden email]>
Unix Systems Engineer @CentraleSupelec Rennes