dict_ldap_lookup questions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

dict_ldap_lookup questions

Gomes, Rich

I’ve started to see a lot of these errors, albeit intermittently:

 

warning: dict_ldap_lookup: Search error -5: Timed out

 

Followed by these:

 

NOQUEUE: reject: RCPT from unknown[x.x.x.x]: 451 4.3.0 <[hidden email]>: Temporary lookup failure

 

 

This server is functioning as an internal relay (no local users) and is performing LDAP lookups for internally routed domains.

 

Is this simply a matter of increasing the timeout? Currently I do not have one specified.

 

Also, where is the cache settings on the LDAP query?

I assume it would not be looking up [hidden email] each time it received an email for it within a specified time-frame.

 

 

Thanks,

Rich

Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni

> On Feb 10, 2017, at 12:01 PM, Gomes, Rich <[hidden email]> wrote:
>
> warning: dict_ldap_lookup: Search error -5: Timed out

You've probably neglected to index the appropriate attributes.
Fix the indexing, and the timeouts will likely go away.

> Is this simply a matter of increasing the timeout?

No your LDAP queries should be fast.  Fix the underlying issue.

> Also, where is the cache settings on the LDAP query?

There is no cache.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
Ok, thank you.

Can you point me in the right direction for indexing?
All I can find is adding this line to the config:
result_attribute = memberaddr  (this was in an expanding groups thread)





-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 12:09 PM
To: Postfix users <[hidden email]>
Subject: Re: dict_ldap_lookup questions


> On Feb 10, 2017, at 12:01 PM, Gomes, Rich <[hidden email]> wrote:
>
> warning: dict_ldap_lookup: Search error -5: Timed out

You've probably neglected to index the appropriate attributes.
Fix the indexing, and the timeouts will likely go away.

> Is this simply a matter of increasing the timeout?

No your LDAP queries should be fast.  Fix the underlying issue.

> Also, where is the cache settings on the LDAP query?

There is no cache.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni
On Fri, Feb 10, 2017 at 05:21:18PM +0000, Gomes, Rich wrote:

> Can you point me in the right direction for indexing?
> All I can find is adding this line to the config:
> result_attribute = memberaddr  (this was in an expanding groups thread)

LDAP data indexing is something that happens in the LDAP database on
the LDAP server.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
It's going against MS AD, I am sure indexing is configured correctly there.


What can I do on my postfix server to alleviate this issue?


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 12:33 PM
To: [hidden email]
Subject: Re: dict_ldap_lookup questions

On Fri, Feb 10, 2017 at 05:21:18PM +0000, Gomes, Rich wrote:

> Can you point me in the right direction for indexing?
> All I can find is adding this line to the config:
> result_attribute = memberaddr  (this was in an expanding groups
> thread)

LDAP data indexing is something that happens in the LDAP database on the LDAP server.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni
On Fri, Feb 10, 2017 at 05:37:36PM +0000, Gomes, Rich wrote:

> It's going against MS AD, I am sure indexing is configured correctly there.

That rather depends on what query you're sending, and how AD is
configured.  Your confidence does not inspire confidence. :-(

> What can I do on my postfix server to alleviate this issue?

    * Use queries that are implemented efficiently on the AD side.

    * Use LDAP servers that are not already struggling with processing
      other queries.

    * As appropriate specify the "domain" attribute in the LDAP table
      definitions to avoid looking for data for domains that you don't
      use in LDAP.

    * Post your Postfix LDAP table definition (sans passwords).

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
Confident because I was part of their setup.   ;  )

* Use queries that are implemented efficiently on the AD side.

        Done

    * Use LDAP servers that are not already struggling with processing
      other queries.

        This is a load balanced pool of (hardware load balanced, not round robin DNS)

    * As appropriate specify the "domain" attribute in the LDAP table
      definitions to avoid looking for data for domains that you don't
      use in LDAP.
       
        Done

    * Post your Postfix LDAP table definition (sans passwords).
       
        # Directory settings
domain = first.com, second.com, third.com, fourth.com, fifth.com, sixth.com
server_host = pool.internal.domain.com
search_base = dc=internal, dc=domain, dc=com
version = 3

# User Binding
bind = yes

bind_dn = CN=serviceaccount,OU=northamerica,DC=internal,DC=domain,DC=com
bind_pw = randompassword

# Filter
query_filter = (&(objectclass=person)(proxyAddresses=smtp:%s))
leaf_result_attribute = proxyAddresses



Thanks for the assistance



Rich




-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 12:49 PM
To: [hidden email]
Subject: Re: dict_ldap_lookup questions

On Fri, Feb 10, 2017 at 05:37:36PM +0000, Gomes, Rich wrote:

> It's going against MS AD, I am sure indexing is configured correctly there.

That rather depends on what query you're sending, and how AD is configured.  Your confidence does not inspire confidence. :-(

> What can I do on my postfix server to alleviate this issue?

    * Use queries that are implemented efficiently on the AD side.

    * Use LDAP servers that are not already struggling with processing
      other queries.

    * As appropriate specify the "domain" attribute in the LDAP table
      definitions to avoid looking for data for domains that you don't
      use in LDAP.

    * Post your Postfix LDAP table definition (sans passwords).

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni

> On Feb 10, 2017, at 1:15 PM, Gomes, Rich <[hidden email]> wrote:
>
> domain = first.com, second.com, third.com, fourth.com, fifth.com, sixth.com
> server_host = pool.internal.domain.com
> search_base = dc=internal, dc=domain, dc=com
> version = 3
>
> # Filter
> query_filter = (&(objectclass=person)(proxyAddresses=smtp:%s))
> leaf_result_attribute = proxyAddresses

The query filter looks fine.  So query performance should fine,
provided you use "proxy:ldap:..." instead of "ldap:..." some
servers don't like having thousands of connections and using
"proxy:" pools requests from multiple smtpd(8) servers over
a single connection in proxyread(8).

Separately, your result attribute is odd.  I know of no Postfix
table that expects multiple "smtp:<address>" address values.
Also you're not using any "speciail_result_attribute" fiels,
so "leaf_result_attribute" should just be "result_attribute".
For object existence use:

        query_filter = (&(proxyAddresses=smtp:%s)(objectclass=person))
        result_attribute = mail

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
The reason the query is setup like that is we have several internal domains and a user may have an alias for one or all of them depending on their employment history.
Since it is working as expected, I'd rather leave it as is, unless you feel it may be a contributor to the issue I am seeing.





-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 1:38 PM
To: Postfix users <[hidden email]>
Subject: Re: dict_ldap_lookup questions


> On Feb 10, 2017, at 1:15 PM, Gomes, Rich <[hidden email]> wrote:
>
> domain = first.com, second.com, third.com, fourth.com, fifth.com,
> sixth.com server_host = pool.internal.domain.com search_base =
> dc=internal, dc=domain, dc=com version = 3
>
> # Filter
> query_filter = (&(objectclass=person)(proxyAddresses=smtp:%s))
> leaf_result_attribute = proxyAddresses

The query filter looks fine.  So query performance should fine, provided you use "proxy:ldap:..." instead of "ldap:..." some servers don't like having thousands of connections and using "proxy:" pools requests from multiple smtpd(8) servers over a single connection in proxyread(8).

Separately, your result attribute is odd.  I know of no Postfix table that expects multiple "smtp:<address>" address values.
Also you're not using any "speciail_result_attribute" fiels, so "leaf_result_attribute" should just be "result_attribute".
For object existence use:

        query_filter = (&(proxyAddresses=smtp:%s)(objectclass=person))
        result_attribute = mail

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni

> On Feb 10, 2017, at 2:27 PM, Gomes, Rich <[hidden email]> wrote:
>
> The reason the query is setup like that is we have several internal
> domains and a user may have an alias for one or all of them depending
> on their employment history.

You've failed to understand my response.  The "proxyAddresses" attribute
is multi-valued, and returns results of the form "smtp:<rfc822address>".
Nothing in Postfix can uses such results, so you're better off returning
a single-valued attribute such as "mail".

> Since it is working as expected, I'd rather leave it as is, unless you
> feel it may be a contributor to the issue I am seeing.

The primary recommendation is to use "proxy:ldap:" rather than "ldap:".
You've not yet explained what you're using LDAP for.  Is this a
relay_recipient_maps table?  Some other table that ignores the RHS value?
Have you tested lookup latency with:
       
        $ domain=example.com # Replace with actual domain
        $ i=0; while (( i < 1024 )); do
                echo "$i-probe@$domain"
                i=$(( i + 1 ))
          done | time postmap -q - ldap:/table/file.cf

The idea is to establish a single connection and then test ~1000
queries for distinct addresses (for a domain that matches the
domain= constraints in the table definition).  The actual addresses
need not exist in LDAP.

Report your results.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
I am using ldap:

I will try using it as proxy:ldap: instead as well as your script suggestion

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 2:43 PM
To: Postfix users <[hidden email]>
Subject: Re: dict_ldap_lookup questions


> On Feb 10, 2017, at 2:27 PM, Gomes, Rich <[hidden email]> wrote:
>
> The reason the query is setup like that is we have several internal
> domains and a user may have an alias for one or all of them depending
> on their employment history.

You've failed to understand my response.  The "proxyAddresses" attribute is multi-valued, and returns results of the form "smtp:<rfc822address>".
Nothing in Postfix can uses such results, so you're better off returning a single-valued attribute such as "mail".

> Since it is working as expected, I'd rather leave it as is, unless you
> feel it may be a contributor to the issue I am seeing.

The primary recommendation is to use "proxy:ldap:" rather than "ldap:".
You've not yet explained what you're using LDAP for.  Is this a relay_recipient_maps table?  Some other table that ignores the RHS value?
Have you tested lookup latency with:
       
        $ domain=example.com # Replace with actual domain
        $ i=0; while (( i < 1024 )); do
                echo "$i-probe@$domain"
                i=$(( i + 1 ))
          done | time postmap -q - ldap:/table/file.cf

The idea is to establish a single connection and then test ~1000 queries for distinct addresses (for a domain that matches the domain= constraints in the table definition).  The actual addresses need not exist in LDAP.

Report your results.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
In the test script for latency below, does table/file.cf refer to the ldapconfig file I am using?


        $ domain=example.com # Replace with actual domain
        $ i=0; while (( i < 1024 )); do
                echo "$i-probe@$domain"
                i=$(( i + 1 ))
          done | time postmap -q - ldap:/table/file.cf





-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Gomes, Rich
Sent: Friday, February 10, 2017 2:49 PM
To: Postfix users <[hidden email]>
Subject: RE: dict_ldap_lookup questions

I am using ldap:

I will try using it as proxy:ldap: instead as well as your script suggestion

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 2:43 PM
To: Postfix users <[hidden email]>
Subject: Re: dict_ldap_lookup questions


> On Feb 10, 2017, at 2:27 PM, Gomes, Rich <[hidden email]> wrote:
>
> The reason the query is setup like that is we have several internal
> domains and a user may have an alias for one or all of them depending
> on their employment history.

You've failed to understand my response.  The "proxyAddresses" attribute is multi-valued, and returns results of the form "smtp:<rfc822address>".
Nothing in Postfix can uses such results, so you're better off returning a single-valued attribute such as "mail".

> Since it is working as expected, I'd rather leave it as is, unless you
> feel it may be a contributor to the issue I am seeing.

The primary recommendation is to use "proxy:ldap:" rather than "ldap:".
You've not yet explained what you're using LDAP for.  Is this a relay_recipient_maps table?  Some other table that ignores the RHS value?
Have you tested lookup latency with:
       
        $ domain=example.com # Replace with actual domain
        $ i=0; while (( i < 1024 )); do
                echo "$i-probe@$domain"
                i=$(( i + 1 ))
          done | time postmap -q - ldap:/table/file.cf

The idea is to establish a single connection and then test ~1000 queries for distinct addresses (for a domain that matches the domain= constraints in the table definition).  The actual addresses need not exist in LDAP.

Report your results.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni

> On Feb 13, 2017, at 1:38 PM, Gomes, Rich <[hidden email]> wrote:
>
> In the test script for latency below, does table/file.cf refer to the ldapconfig file I am using?
>
> $ domain=example.com # Replace with actual domain
> $ i=0; while (( i < 1024 )); do
> echo "$i-probe@$domain"
> i=$(( i + 1 ))
>  done | time postmap -q - ldap:/table/file.cf

Yes.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

RE: dict_ldap_lookup questions

Gomes, Rich
In reply to this post by Viktor Dukhovni
Here is from a Test machine with very low mail traffic and the suggested config changes:

real    0m51.42s
user    0m0.05s
sys     0m0.04s


And here is from Prod with a high volume of traffic and the original configuration:

real    1m24.74s
user    0m0.05s
sys     0m0.06s


Still trying to get the application folks to simulate the same volume of traffic in Test so we can definitively see if the changes made a difference.





-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Dukhovni
Sent: Friday, February 10, 2017 2:43 PM
To: Postfix users <[hidden email]>
Subject: Re: dict_ldap_lookup questions


> On Feb 10, 2017, at 2:27 PM, Gomes, Rich <[hidden email]> wrote:
>
> The reason the query is setup like that is we have several internal
> domains and a user may have an alias for one or all of them depending
> on their employment history.

You've failed to understand my response.  The "proxyAddresses" attribute is multi-valued, and returns results of the form "smtp:<rfc822address>".
Nothing in Postfix can uses such results, so you're better off returning a single-valued attribute such as "mail".

> Since it is working as expected, I'd rather leave it as is, unless you
> feel it may be a contributor to the issue I am seeing.

The primary recommendation is to use "proxy:ldap:" rather than "ldap:".
You've not yet explained what you're using LDAP for.  Is this a relay_recipient_maps table?  Some other table that ignores the RHS value?
Have you tested lookup latency with:
       
        $ domain=example.com # Replace with actual domain
        $ i=0; while (( i < 1024 )); do
                echo "$i-probe@$domain"
                i=$(( i + 1 ))
          done | time postmap -q - ldap:/table/file.cf

The idea is to establish a single connection and then test ~1000 queries for distinct addresses (for a domain that matches the domain= constraints in the table definition).  The actual addresses need not exist in LDAP.

Report your results.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: dict_ldap_lookup questions

Viktor Dukhovni

> On Feb 14, 2017, at 2:55 PM, Gomes, Rich <[hidden email]> wrote:
>
> Here is from a Test machine with very low mail traffic and the suggested config changes:
>
> real    0m51.42s
> user    0m0.05s
> sys     0m0.04s

50ms per query is a rather high lookup latency for LDAP.  Around ten years
back I was seeing ~2ms per query.  What's the network latency between the
MTA and the LDAP servers?  Are they too busy serving lookups from internal
interactive users?  Perhaps replica servers exclusively for the MTAs are
needed to isolate the MTAs from high user load and vice versa.

In any case it seems the LDAP server is quite slow.  Test with other filters,
e.g. 'mail=%s' or something similar that definitely hits an indexed attribute.
Perhaps authorization is slow, are there access controls that may be expensive
to evaluate, ...

> And here is from Prod with a high volume of traffic and the original configuration:
>
> real    1m24.74s
> user    0m0.05s
> sys     0m0.06s

The difference between 51ms and 85ms is not dramatic.  A performant LDAP
service would be around 1ms (~150km speed of light round-trip time).
Unless your LDAP servers are in a distant city, the performance you're
seeing is much too low.

--
        Viktor.