Lookup tables

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Lookup tables

jack-2
Hi,

In the online documentation for access tables
(http://www.postfix.org/access.5.html), it says:

              Subnetworks  are  matched  by  repeatedly  truncating
              the  last ".octet" from the remote IPv4 host address
              string until a  match is found in the access table, or
              until further truncation is not possible.

This is supposedly subject only to the restriction that the table is an
indexed file "such as DB or DBM".

I have the following client_access table:
5.188.9         REJECT WebShield Network trying to hack Dovecot
2018-05-10 - test
5.188.9.1         REJECT WebShield Network trying to hack Dovecot 2018-05-10

I compile the table to create client_access.db:
# postmap client_access

I then try:
# postmap -q 5.188.9.2 client_access
[no output]

# postmap -q 5.188.9.1 client_access
REJECT WebShield Network trying to hack Dovecot 2018-05-10

The behaviour of postmap seems to be at odds with the documentation;
specfically, it does not seem to be possible to match an address against
an address-prefix in the table. Am I misunderstanding the docs, or do
they need fixing?

I haven't tried any of the other indexed lookup types; is there some
other table type that works properly? Do I need to test them all to see
if they comply with the docs?

Thanks,
--
Jack.

Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

jack-2
Sorry - I should have said:

Postfix 2.11.3, running on Debian Jessie.

Also, I ran these tests using postmap when it became apparent to me that
postfix itself was not matching address prefixes in hash tables.

On 14/05/2018 11:18, jack wrote:

> Hi,
>
> In the online documentation for access tables
> (http://www.postfix.org/access.5.html), it says:
>
>               Subnetworks  are  matched  by  repeatedly  truncating
>               the  last ".octet" from the remote IPv4 host address
>               string until a  match is found in the access table, or
>               until further truncation is not possible.
>
> This is supposedly subject only to the restriction that the table is an
> indexed file "such as DB or DBM".
>
> I have the following client_access table:
> 5.188.9         REJECT WebShield Network trying to hack Dovecot
> 2018-05-10 - test
> 5.188.9.1         REJECT WebShield Network trying to hack Dovecot 2018-05-10
>
> I compile the table to create client_access.db:
> # postmap client_access
>
> I then try:
> # postmap -q 5.188.9.2 client_access
> [no output]
>
> # postmap -q 5.188.9.1 client_access
> REJECT WebShield Network trying to hack Dovecot 2018-05-10
>
> The behaviour of postmap seems to be at odds with the documentation;
> specfically, it does not seem to be possible to match an address against
> an address-prefix in the table. Am I misunderstanding the docs, or do
> they need fixing?
>
> I haven't tried any of the other indexed lookup types; is there some
> other table type that works properly? Do I need to test them all to see
> if they comply with the docs?
>
> Thanks,
>
Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

Mike Guelfi
In reply to this post by jack-2
postmap is a lookup management tool; doing a query on an IP in a  
subnet isn't going to succeed.

You probably just forgot to enable client_access or reload postfix

What does this return?
# postconf smtpd_client_restrictions

Default is:
smtpd_client_restrictions =

enabled would be:
smtpd_client_restrictions = check_client_access hash:/path/to/client_access

Quoting jack <[hidden email]>:

> Hi,
>
> In the online documentation for access tables
> (http://www.postfix.org/access.5.html), it says:
>
>               Subnetworks  are  matched  by  repeatedly  truncating
>               the  last ".octet" from the remote IPv4 host address
>               string until a  match is found in the access table, or
>               until further truncation is not possible.
>
> This is supposedly subject only to the restriction that the table is an
> indexed file "such as DB or DBM".
>
> I have the following client_access table:
> 5.188.9         REJECT WebShield Network trying to hack Dovecot
> 2018-05-10 - test
> 5.188.9.1         REJECT WebShield Network trying to hack Dovecot 2018-05-10
>
> I compile the table to create client_access.db:
> # postmap client_access
>
> I then try:
> # postmap -q 5.188.9.2 client_access
> [no output]
>
> # postmap -q 5.188.9.1 client_access
> REJECT WebShield Network trying to hack Dovecot 2018-05-10
>
> The behaviour of postmap seems to be at odds with the documentation;
> specfically, it does not seem to be possible to match an address against
> an address-prefix in the table. Am I misunderstanding the docs, or do
> they need fixing?
>
> I haven't tried any of the other indexed lookup types; is there some
> other table type that works properly? Do I need to test them all to see
> if they comply with the docs?
>
> Thanks,
> --
> Jack.


Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

Wietse Venema
In reply to this post by jack-2
jack:

> Hi,
>
> In the online documentation for access tables
> (http://www.postfix.org/access.5.html), it says:
>
>               Subnetworks  are  matched  by  repeatedly  truncating
>               the  last ".octet" from the remote IPv4 host address
>               string until a  match is found in the access table, or
>               until further truncation is not possible.
>
> This is supposedly subject only to the restriction that the table is an
> indexed file "such as DB or DBM".

Postfix will query hash (btreem, dbm, lmdb, ldap, etc.) table
multiple times, first with the full IP address and then with prefixes
of the IP address. With your example of 5.188.9.2 the queries would
be:

    5.188.9.2
    5.188.9

There would be more queries if there is no match.

(with cidr, pcre, and regexp tables there would be only one lookup).

> I have the following client_access table:
> 5.188.9         REJECT WebShield Network trying to hack Dovecot
> 2018-05-10 - test
> 5.188.9.1         REJECT WebShield Network trying to hack Dovecot 2018-05-10
>
> I compile the table to create client_access.db:
> # postmap client_access
>
> I then try:
> # postmap -q 5.188.9.2 client_access
> [no output]

The postmap command does not make all the queries that I described
above. You will have to do that instead.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

jack-2
In reply to this post by Mike Guelfi
# postconf smtpd_client_restrictions
smtpd_client_restrictions = reject_unknown_reverse_client_hostname,
check_client_access hash:/etc/postfix/client_access,
permit_sasl_authenticated

On 14/05/2018 12:00, Mike Guelfi wrote:

> postmap is a lookup management tool; doing a query on an IP in a subnet
> isn't going to succeed.
>
> You probably just forgot to enable client_access or reload postfix
>
> What does this return?
> # postconf smtpd_client_restrictions
>
> Default is:
> smtpd_client_restrictions =
>
> enabled would be:
> smtpd_client_restrictions = check_client_access hash:/path/to/client_access
>
> Quoting jack <[hidden email]>:
>
>> Hi,
>>
>> In the online documentation for access tables
>> (http://www.postfix.org/access.5.html), it says:
>>
>>               Subnetworks  are  matched  by  repeatedly  truncating
>>               the  last ".octet" from the remote IPv4 host address
>>               string until a  match is found in the access table, or
>>               until further truncation is not possible.
>>
>> This is supposedly subject only to the restriction that the table is an
>> indexed file "such as DB or DBM".
>>
>> I have the following client_access table:
>> 5.188.9         REJECT WebShield Network trying to hack Dovecot
>> 2018-05-10 - test
>> 5.188.9.1         REJECT WebShield Network trying to hack Dovecot
>> 2018-05-10
>>
>> I compile the table to create client_access.db:
>> # postmap client_access
>>
>> I then try:
>> # postmap -q 5.188.9.2 client_access
>> [no output]
>>
>> # postmap -q 5.188.9.1 client_access
>> REJECT WebShield Network trying to hack Dovecot 2018-05-10
>>
>> The behaviour of postmap seems to be at odds with the documentation;
>> specfically, it does not seem to be possible to match an address against
>> an address-prefix in the table. Am I misunderstanding the docs, or do
>> they need fixing?
>>
>> I haven't tried any of the other indexed lookup types; is there some
>> other table type that works properly? Do I need to test them all to see
>> if they comply with the docs?
>>
>> Thanks,
>> --
>> Jack.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

jack-2
In reply to this post by Wietse Venema
Mike, I had:

# postconf smtpd_client_restrictions
smtpd_client_restrictions = reject_unknown_reverse_client_hostname,
check_client_access hash:/etc/postfix/client_access,
permit_sasl_authenticated

On 14/05/2018 12:13, Wietse Venema wrote:

> jack:
>> Hi,
>>
>> In the online documentation for access tables
>> (http://www.postfix.org/access.5.html), it says:
>>
>>               Subnetworks  are  matched  by  repeatedly  truncating
>>               the  last ".octet" from the remote IPv4 host address
>>               string until a  match is found in the access table, or
>>               until further truncation is not possible.
>>
>> This is supposedly subject only to the restriction that the table is an
>> indexed file "such as DB or DBM".
>
> Postfix will query hash (btreem, dbm, lmdb, ldap, etc.) table
> multiple times, first with the full IP address and then with prefixes
> of the IP address. With your example of 5.188.9.2 the queries would
> be:
>
>     5.188.9.2
>     5.188.9
>
> There would be more queries if there is no match.

Aaaah. Light dawns. So the prefix match should be working in postfix,
even if it doesn't work in postmap. That's not what I thought I
observed; but I didn't test postfix thoroughly, because it was easier to
test postmap. Oh well!
>
> (with cidr, pcre, and regexp tables there would be only one lookup).

Non-indexed tables will no doubt be less efficient, beyond some
threshold dataset size. And those table-types are memory-resident, AIUI,
so there would be a memory-hit for large tables.

Anyway, thanks for clearing this up.

--
Jack.
Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

jack-2
In reply to this post by Wietse Venema
On 14/05/2018 12:13, Wietse Venema wrote:

>
> Postfix will query hash (btreem, dbm, lmdb, ldap, etc.) table
> multiple times, first with the full IP address and then with prefixes
> of the IP address. With your example of 5.188.9.2 the queries would
> be:
>
>     5.188.9.2
>     5.188.9
>
> There would be more queries if there is no match.

So I have been unable to discover that information anywhere on the
postfix.org site. Also, I have a suspicion that I may have already asked
this question, many years ago, received an answer, and forgotten it.

I suspect it must be an FAQ; perhaps the docs should include it? It's
not obvious, at least not to me. Perhaps it's already in the docs, and I
just failed to find it?

Thanks again,

--
Jack.
Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

Viktor Dukhovni


> On May 14, 2018, at 1:54 PM, jack <[hidden email]> wrote:
>
>> Postfix will query hash (btreem, dbm, lmdb, ldap, etc.) table
>> multiple times, first with the full IP address and then with prefixes
>> of the IP address. With your example of 5.188.9.2 the queries would
>> be:
>>
>>    5.188.9.2
>>    5.188.9
>>
>> There would be more queries if there is no match.
>
> So I have been unable to discover that information anywhere on the
> postfix.org site. Also, I have a suspicion that I may have already asked
> this question, many years ago, received an answer, and forgotten it.

Look for "HOST NAME/ADDRESS PATTERNS" in http://www.postfix.org/access.5.html

The http://www.postfix.org/postconf.5.html#check_client_access docs point
you at access(5), so this is not exactly hiding...

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

jack-2

On 14/05/2018 18:58, Viktor Dukhovni wrote:
>
>
> Look for "HOST NAME/ADDRESS PATTERNS" in http://www.postfix.org/access.5.html
>
> The http://www.postfix.org/postconf.5.html#check_client_access docs point
> you at access(5), so this is not exactly hiding...

Thank you Viktor, I am familiar with those pages.

The fact which I think may be undocumented is that postfix (but not
postmap) performs iterative prefix queries, when (and only when?) the
table-type is indexed.

I really hope I haven't come across as critical; I love this program.
I'm only making a helpful observation about the discoverability of the
info that Wietse gave me. Perhaps a page with info about the hash/DB
table-type (like there is about the LDAP table-type) might help. Or not.

--
Jack.
Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

Viktor Dukhovni


> On May 14, 2018, at 2:46 PM, jack <[hidden email]> wrote:
>
> The fact which I think may be undocumented is that postfix (but not
> postmap) performs iterative prefix queries, when (and only when?) the
> table-type is indexed.

This is basic logic, support for multi-line header/body lookups
notwithstanding, postmap does not know the syntax of the lookup
key or purpose of the tables, you hand it a key and it looks up
the value.  It couldn't possibly implement access(5) semantics,
because it has no idea it is doing access(5) lookups, or which
of the key types (email address, hostname, ip address, ...) you're
giving it.

Perhaps this (logically inescapable) limitation should be explicitly
called out in the postmap(1) manpage, since, despite it being clearly
impossible, users nevertheless seem to expect postmap(1) to be
omniscient.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Lookup tables

Noel Jones-2
In reply to this post by jack-2
On 5/14/2018 1:46 PM, jack wrote:

>
> On 14/05/2018 18:58, Viktor Dukhovni wrote:
>>
>>
>> Look for "HOST NAME/ADDRESS PATTERNS" in
>> http://www.postfix.org/access.5.html
>>
>> The http://www.postfix.org/postconf.5.html#check_client_access
>> docs point
>> you at access(5), so this is not exactly hiding...
>
> Thank you Viktor, I am familiar with those pages.
>
> The fact which I think may be undocumented is that postfix (but not
> postmap) performs iterative prefix queries, when (and only when?) the
> table-type is indexed.

You're looking at this backwards. Postfix ACCESS TABLES
(check_*_access) do iterative lookups in indexed files. This is a
feature of access tables, not general postfix indexed file behavior.
 This behavior is documented in access(5), which is where it belongs.

Each postfix feature that does anything other that a single verbatim
lookup is documented within that feature's man page lookup pattern
section.  Features do not document things they DON'T do, since an
exhaustive list isn't possible.

That said, it's not unreasonable to include a note on the postmap
page since this seems to trip up many novice users, and part of the
mission is to make postfix usable for novice users.

... Of course, the hard part is crafting a note that's brief and
doesn't add to the confusion.


My first iteration, probably needs improvement.

  -q key Search the specified ...
   ...
     Note: This performs a single search of the key as supplied.
Iterative search of sub-keys is not supported.




  -- Noel Jones