Feature request for postscreen: "defer"

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Feature request for postscreen: "defer"

Christian Rößner
> Am 13.09.2016 um 18:09 schrieb Wietse Venema <[hidden email]>:
>
> Christian Ro??ner:
>> Is there some chance that postscreen could be extended to also have "defer"?
>
> That is a good question, but you might want to ask that in a thread
> that isn't about socketmaps.

You are totally right. I created a new thread for this.

The idea is to give postscreen a "defer" option. At connect time, dynamic services can work with the IP address of a connecting client. In some cases, this can result in whitelisting, blacklisting or no decision. But a fourth decision: "defer" might be interesting in cases, where the risk of a false-positive decision is too big.

Having this in postscreen reduces load on external DNS queries, if you also use dnsblog.

Thanks in advance

Christian
--
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, https://www.roessner-network-solutions.com


smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request for postscreen: "defer"

Wietse Venema
Christian Ro??ner:

> > Am 13.09.2016 um 18:09 schrieb Wietse Venema <[hidden email]>:
> >
> > Christian Ro??ner:
> >> Is there some chance that postscreen could be extended to also have "defer"?
> >
> > That is a good question, but you might want to ask that in a thread
> > that isn't about socketmaps.
>
> You are totally right. I created a new thread for this.
>
> The idea is to give postscreen a "defer" option. At connect time,
> dynamic services can work with the IP address of a connecting
> client. In some cases, this can result in whitelisting, blacklisting
> or no decision. But a fourth decision: "defer" might be interesting
> in cases, where the risk of a false-positive decision is too big.
>
> Having this in postscreen reduces load on external DNS queries,
> if you also use dnsblog.

Unlike DNS lookups, the access map lookup is a blocking operation,
and if your tcp map takes 80ms to complete (a typical trans-atlantic
query), then you can handle only 12 connections per second, and
make postsceen the largest performance bottleneck on the system.

        Wietse
Reply | Threaded
Open this post in threaded view
|

postscreen-policy (was: Feature request for postscreen: "defer")

Wietse Venema
Wietse Venema:
> Unlike DNS lookups, the access map lookup is a blocking operation,
> and if your tcp map takes 80ms to complete (a typical trans-atlantic
> query), then you can handle only 12 connections per second, and
> make postsceen the largest performance bottleneck on the system.

After starting work on postscreen by the middle of 2009, I soon
realized that I might have to add some postscreen-policy interface
for things that are too complex or that take too much time compared
to a quick access map lookup. Perhaps the time has come.

Basically this would be a very small subset of the SMTP server
policy protocol with just the network 5-tuple (source/destination
address/port, protocol, client concurrency), enough to do some
simple reputation work.

Perhaps it also makes sense for postscreen to make a postscreen-policy
call based on the information that it has collected with its dummy
SMTP engine.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: postscreen-policy (was: Feature request for postscreen: "defer")

Patrick Ben Koetter-2
* Wietse Venema <[hidden email]>:

> Wietse Venema:
> > Unlike DNS lookups, the access map lookup is a blocking operation,
> > and if your tcp map takes 80ms to complete (a typical trans-atlantic
> > query), then you can handle only 12 connections per second, and
> > make postsceen the largest performance bottleneck on the system.
>
> After starting work on postscreen by the middle of 2009, I soon
> realized that I might have to add some postscreen-policy interface
> for things that are too complex or that take too much time compared
> to a quick access map lookup. Perhaps the time has come.
>
> Basically this would be a very small subset of the SMTP server
> policy protocol with just the network 5-tuple (source/destination
> address/port, protocol, client concurrency), enough to do some
> simple reputation work.
>
> Perhaps it also makes sense for postscreen to make a postscreen-policy
> call based on the information that it has collected with its dummy
> SMTP engine.

That's great news! The reason Christian is using tcp tables is that there's no
postscreen API to call external policy services at the moment. If there was
he/we would be eager to use that instead.

p@rick

--
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
 
Reply | Threaded
Open this post in threaded view
|

Re: postscreen-policy (was: Feature request for postscreen: "defer")

Wietse Venema
Patrick Ben Koetter:

> * Wietse Venema <[hidden email]>:
> > Wietse Venema:
> > > Unlike DNS lookups, the access map lookup is a blocking operation,
> > > and if your tcp map takes 80ms to complete (a typical trans-atlantic
> > > query), then you can handle only 12 connections per second, and
> > > make postsceen the largest performance bottleneck on the system.
> >
> > After starting work on postscreen by the middle of 2009, I soon
> > realized that I might have to add some postscreen-policy interface
> > for things that are too complex or that take too much time compared
> > to a quick access map lookup. Perhaps the time has come.
> >
> > Basically this would be a very small subset of the SMTP server
> > policy protocol with just the network 5-tuple (source/destination
> > address/port, protocol, client concurrency), enough to do some
> > simple reputation work.
> >
> > Perhaps it also makes sense for postscreen to make a postscreen-policy
> > call based on the information that it has collected with its dummy
> > SMTP engine.
>
> That's great news! The reason Christian is using tcp tables is that there's no
> postscreen API to call external policy services at the moment. If there was
> he/we would be eager to use that instead.

Yes, I wanted the discussion to end on an optimistic note. Something to
work on in the train.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: postscreen-policy

allenc
In reply to this post by Wietse Venema

On 13/09/16 20:01, Wietse Venema wrote:
> Wietse Venema:
>> Unlike DNS lookups, the access map lookup is a blocking operation,
>> and if your tcp map takes 80ms to complete (a typical trans-atlantic
>> query), then you can handle only 12 connections per second, and
>> make postsceen the largest performance bottleneck on the system.
> After starting work on postscreen by the middle of 2009, I soon
> realized that I might have to add some postscreen-policy interface
> for things that are too complex or that take too much time compared
> to a quick access map lookup. Perhaps the time has come.
From a slightly different angle, I would like to see some sort of
"probationary" status, where the temporary whitelist is not activated.

I frequently receive a well-behaved connection, earning  PASS NEW,
immediately followed by abusive behaviour  (see the log extract below).
   In this instance, my fail-2-ban lookalike kicked in, and blocked
several thousand packets, before the IPTABLES counters were reset.

The zz.countries.nerd.dk entry tells me the host was located in China -
a spam hot-spot for me. (One demerit point, but not enough to blacklist)

Allen C


Sep 13 15:23:14 geronimo postfix/postscreen[9767]: CONNECT from
[202.106.74.102]:2600 to [192.168.150.12]:25
Sep 13 15:23:14 geronimo postfix/dnsblog[9769]: addr 202.106.74.102
listed by domain zz.countries.nerd.dk as 127.0.0.156
Sep 13 15:23:20 geronimo postfix/postscreen[9767]: PASS NEW
[202.106.74.102]:2600
Sep 13 15:23:21 geronimo postfix/smtpd[9777]: connect from
unknown[202.106.74.102]
Sep 13 15:23:24 geronimo postfix/smtpd[9777]: disconnect from
unknown[202.106.74.102] ehlo=1 quit=1 commands=2
Sep 13 15:23:27 geronimo postfix/postscreen[9767]: CONNECT from
[202.106.74.102]:4225 to [192.168.150.12]:25
Sep 13 15:23:29 geronimo postfix/postscreen[9767]: CONNECT from
[202.106.74.102]:1202 to [192.168.150.12]:25
Sep 13 15:23:31 geronimo postfix/postscreen[9767]: CONNECT from
[202.106.74.102]:2070 to [192.168.150.12]:25
Sep 13 15:23:31 geronimo postfix/postscreen[9767]: NOQUEUE: reject:
CONNECT from [202.106.74.102]:2070: too many connections
Sep 13 15:23:31 geronimo postfix/postscreen[9767]: DISCONNECT
[202.106.74.102]:2070
Sep 13 15:23:33 geronimo postfix/postscreen[9767]: PASS OLD
[202.106.74.102]:4225
Sep 13 15:23:33 geronimo postfix/postscreen[9767]: CONNECT from
[202.106.74.102]:2478 to [192.168.150.12]:25
Sep 13 15:23:33 geronimo postfix/postscreen[9767]: NOQUEUE: reject:
CONNECT from [202.106.74.102]:2478: too many connections
Sep 13 15:23:33 geronimo postfix/postscreen[9767]: DISCONNECT
[202.106.74.102]:2478
Sep 13 15:23:33 geronimo postfix/smtpd[9777]: connect from
unknown[202.106.74.102]
Sep 13 15:23:34 geronimo postfix/postscreen[9767]: CONNECT from
[202.106.74.102]:1393 to [192.168.150.12]:25
Sep 13 15:23:34 geronimo postfix/postscreen[9767]: PASS OLD
[202.106.74.102]:1393
Sep 13 15:23:34 geronimo postfix/smtpd[9786]: connect from
unknown[202.106.74.102]
Sep 13 15:23:35 geronimo postfix/postscreen[9767]: PASS OLD
[202.106.74.102]:1202
Sep 13 15:23:35 geronimo postfix/smtpd[9788]: connect from
unknown[202.106.74.102]
Sep 13 15:23:35 geronimo postfix/smtpd[9788]: warning: Connection
concurrency limit exceeded: 3 from unknown[202.106.74.102] for service smtpd
Sep 13 15:23:35 geronimo postfix/smtpd[9788]: disconnect from
unknown[202.106.74.102] commands=0/0
......
Etc, etc, until the IP address is blocked.




Reply | Threaded
Open this post in threaded view
|

Re: Feature request for postscreen: "defer"

Christian Rößner
In reply to this post by Wietse Venema
> Am 13.09.2016 um 19:00 schrieb Wietse Venema <[hidden email]>:

>
> Christian Ro??ner:
>>> Am 13.09.2016 um 18:09 schrieb Wietse Venema <[hidden email]>:
>>>
>>> Christian Ro??ner:
>>>> Is there some chance that postscreen could be extended to also have "defer"?
>>>
>>> That is a good question, but you might want to ask that in a thread
>>> that isn't about socketmaps.
>>
>> You are totally right. I created a new thread for this.
>>
>> The idea is to give postscreen a "defer" option. At connect time,
>> dynamic services can work with the IP address of a connecting
>> client. In some cases, this can result in whitelisting, blacklisting
>> or no decision. But a fourth decision: "defer" might be interesting
>> in cases, where the risk of a false-positive decision is too big.
>>
>> Having this in postscreen reduces load on external DNS queries,
>> if you also use dnsblog.
>
> Unlike DNS lookups, the access map lookup is a blocking operation,
> and if your tcp map takes 80ms to complete (a typical trans-atlantic
> query), then you can handle only 12 connections per second, and
> make postsceen the largest performance bottleneck on the system.
Good point. I will think about moving the tcp-map to "smtpd".

Thank you very much for clarifying the performance impact

Christian
--
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, https://www.roessner-network-solutions.com


smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request for postscreen: "defer"

Christian Rößner
> Am 14.09.2016 um 07:50 schrieb Christian Rößner <[hidden email]>:

>
>> Am 13.09.2016 um 19:00 schrieb Wietse Venema <[hidden email]>:
>>
>> Christian Ro??ner:
>>>> Am 13.09.2016 um 18:09 schrieb Wietse Venema <[hidden email]>:
>>>>
>>>> Christian Ro??ner:
>>>>> Is there some chance that postscreen could be extended to also have "defer"?
>>>>
>>>> That is a good question, but you might want to ask that in a thread
>>>> that isn't about socketmaps.
>>>
>>> You are totally right. I created a new thread for this.
>>>
>>> The idea is to give postscreen a "defer" option. At connect time,
>>> dynamic services can work with the IP address of a connecting
>>> client. In some cases, this can result in whitelisting, blacklisting
>>> or no decision. But a fourth decision: "defer" might be interesting
>>> in cases, where the risk of a false-positive decision is too big.
>>>
>>> Having this in postscreen reduces load on external DNS queries,
>>> if you also use dnsblog.
>>
>> Unlike DNS lookups, the access map lookup is a blocking operation,
>> and if your tcp map takes 80ms to complete (a typical trans-atlantic
>> query), then you can handle only 12 connections per second, and
>> make postsceen the largest performance bottleneck on the system.
>
> Good point. I will think about moving the tcp-map to "smtpd".
>
> Thank you very much for clarifying the performance impact
Ah... Just read about the postscreen-policy idea. :-)
--
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, https://www.roessner-network-solutions.com



smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: postscreen-policy (was: Feature request for postscreen: "defer")

Patrick Ben Koetter-2
In reply to this post by Wietse Venema
* Wietse Venema <[hidden email]>:

> Patrick Ben Koetter:
> > * Wietse Venema <[hidden email]>:
> > > Wietse Venema:
> > > > Unlike DNS lookups, the access map lookup is a blocking operation,
> > > > and if your tcp map takes 80ms to complete (a typical trans-atlantic
> > > > query), then you can handle only 12 connections per second, and
> > > > make postsceen the largest performance bottleneck on the system.
> > >
> > > After starting work on postscreen by the middle of 2009, I soon
> > > realized that I might have to add some postscreen-policy interface
> > > for things that are too complex or that take too much time compared
> > > to a quick access map lookup. Perhaps the time has come.
> > >
> > > Basically this would be a very small subset of the SMTP server
> > > policy protocol with just the network 5-tuple (source/destination
> > > address/port, protocol, client concurrency), enough to do some
> > > simple reputation work.

Seems like you had fleshed out a simliar idea a few years before, too:
https://www.mail-archive.com/postfix-devel@.../msg00258.html


> > > Perhaps it also makes sense for postscreen to make a postscreen-policy
> > > call based on the information that it has collected with its dummy
> > > SMTP engine.
> >
> > That's great news! The reason Christian is using tcp tables is that there's no
> > postscreen API to call external policy services at the moment. If there was
> > he/we would be eager to use that instead.
>
> Yes, I wanted the discussion to end on an optimistic note. Something to
> work on in the train.

I was just perusing the Change Log for the upcoming Postfix 3.3 release
looking for a note referring to a postscreen policy delegation protocol.

Did I miss the note? Did you loose interest? Missed the train? ;)

p@rick


--
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein
 
Reply | Threaded
Open this post in threaded view
|

Re: postscreen-policy (was: Feature request for postscreen: "defer")

Wietse Venema
Patrick Ben Koetter:

> * Wietse Venema <[hidden email]>:
> > Patrick Ben Koetter:
> > > * Wietse Venema <[hidden email]>:
> > > > Wietse Venema:
> > > > > Unlike DNS lookups, the access map lookup is a blocking operation,
> > > > > and if your tcp map takes 80ms to complete (a typical trans-atlantic
> > > > > query), then you can handle only 12 connections per second, and
> > > > > make postsceen the largest performance bottleneck on the system.
> > > >
> > > > After starting work on postscreen by the middle of 2009, I soon
> > > > realized that I might have to add some postscreen-policy interface
> > > > for things that are too complex or that take too much time compared
> > > > to a quick access map lookup. Perhaps the time has come.
> > > >
> > > > Basically this would be a very small subset of the SMTP server
> > > > policy protocol with just the network 5-tuple (source/destination
> > > > address/port, protocol, client concurrency), enough to do some
> > > > simple reputation work.
>
> Seems like you had fleshed out a simliar idea a few years before, too:
> https://www.mail-archive.com/postfix-devel@.../msg00258.html
>
>
> > > > Perhaps it also makes sense for postscreen to make a postscreen-policy
> > > > call based on the information that it has collected with its dummy
> > > > SMTP engine.
> > >
> > > That's great news! The reason Christian is using tcp tables is that there's no
> > > postscreen API to call external policy services at the moment. If there was
> > > he/we would be eager to use that instead.
> >
> > Yes, I wanted the discussion to end on an optimistic note. Something to
> > work on in the train.
>
> I was just perusing the Change Log for the upcoming Postfix 3.3 release
> looking for a note referring to a postscreen policy delegation protocol.
>
> Did I miss the note? Did you loose interest? Missed the train? ;)

Lack of time. It's no more complex than the way that postscreen
communicates with dnsblog processes. Maybe in the Postfix 3.4 cycle.

        Wietse