Three trivial filtering questions

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Three trivial filtering questions

Ronald F. Guilmette-2


Does reject_non_fqdn_helo_hostname, when placed in the
smtpd_helo_restrictions, permit clients to HELO/EHLO
with a square-bracket enclosed dotted quad IPv4 address?

If so, is the dotted quad checked to see that it properly
represents the actual IP address of the actual current client?

Also, I have just added all of the following to my
smtpd_recipient_restrictions:

        reject_rhsbl_reverse_client multi.surbl.org
        reject_rhsbl_reverse_client multi.uribl.com
        reject_rhsbl_reverse_client dbl.spamhaus.org
        reject_rhsbl_sender multi.surbl.org
        reject_rhsbl_sender multi.uribl.com
        reject_rhsbl_sender dbl.spamhaus.org
        reject_rhsbl_helo multi.surbl.org
        reject_rhsbl_helo multi.uribl.com
        reject_rhsbl_helo dbl.spamhaus.org

For the time being, and mostly just to see how effective these filters
are on their own, I have these listed in my smtpd_recipient_restrictions
*prior to* several subsequent reject_rbl_client clauses.  Oddly however,
in spite of the ordering, it is appearing to me as if perhaps the above
RHS filters are either not actually being applied or else are being applied
_after_ the subsequent reject_rbl_client filters.  Certainly, some spam
that I believe should have been rejected on the basis of one or another
of the above RHS filters I am instead seeing (in my maillog file) being
rejected instead by one or another of the subsequent reject_rbl_client
filters.   What could I be doing wrong?

I am running 2.10.0.


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Noel Jones-2
On 8/4/2013 8:06 PM, Ronald F. Guilmette wrote:
> Does reject_non_fqdn_helo_hostname, when placed in the
> smtpd_helo_restrictions, permit clients to HELO/EHLO
> with a square-bracket enclosed dotted quad IPv4 address?

Yes.

>
> If so, is the dotted quad checked to see that it properly
> represents the actual IP address of the actual current client?

No.


>
> Also, I have just added all of the following to my
> smtpd_recipient_restrictions:
>
>         reject_rhsbl_reverse_client multi.surbl.org
>         reject_rhsbl_reverse_client multi.uribl.com
>         reject_rhsbl_reverse_client dbl.spamhaus.org
>         reject_rhsbl_sender multi.surbl.org
>         reject_rhsbl_sender multi.uribl.com
>         reject_rhsbl_sender dbl.spamhaus.org
>         reject_rhsbl_helo multi.surbl.org
>         reject_rhsbl_helo multi.uribl.com
>         reject_rhsbl_helo dbl.spamhaus.org
>
> For the time being, and mostly just to see how effective these filters
> are on their own, I have these listed in my smtpd_recipient_restrictions
> *prior to* several subsequent reject_rbl_client clauses.  Oddly however,
> in spite of the ordering, it is appearing to me as if perhaps the above
> RHS filters are either not actually being applied or else are being applied
> _after_ the subsequent reject_rbl_client filters.  Certainly, some spam
> that I believe should have been rejected on the basis of one or another
> of the above RHS filters I am instead seeing (in my maillog file) being
> rejected instead by one or another of the subsequent reject_rbl_client
> filters.   What could I be doing wrong?


Doing RBL client checks in postscreen?




  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2

In message <[hidden email]>,
Noel Jones <[hidden email]> wrote:

>On 8/4/2013 8:06 PM, Ronald F. Guilmette wrote:
>> Does reject_non_fqdn_helo_hostname, when placed in the
>> smtpd_helo_restrictions, permit clients to HELO/EHLO
>> with a square-bracket enclosed dotted quad IPv4 address?
>
>Yes.

The documentatation should probably be adjusted to make that more clear.
Right now it reads:

     Reject the request when the HELO or EHLO hostname is not in fully-
     qualified domain form, as required by the RFC.

>> If so, is the dotted quad checked to see that it properly
>> represents the actual IP address of the actual current client?
>
>No.

Is there any restriction verb that would cause a HELO/EHLO which specifies
a square-bracketed dotted quad IPv4 address to be rejected when & if the
dotted quad does not match the actual current client IP address?

Would reject_unknown_helo_hostname do it?  If not maybe a new restriction
verb would be useful to perform this exact check.

>> Certainly, some spam
>> that I believe should have been rejected on the basis of one or another
>> of the above RHS filters I am instead seeing (in my maillog file) being
>> rejected instead by one or another of the subsequent reject_rbl_client
>> filters.   What could I be doing wrong?
>
>
>Doing RBL client checks in postscreen?

I am not using postscreen at the present time.

Do I need to use that if I want to perform RHSBL checks?


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Stan Hoeppner
In reply to this post by Noel Jones-2
On 8/4/2013 9:54 PM, Noel Jones wrote:
> On 8/4/2013 8:06 PM, Ronald F. Guilmette wrote:
...

>> Also, I have just added all of the following to my
>> smtpd_recipient_restrictions:
>>
>>         reject_rhsbl_reverse_client multi.surbl.org
>>         reject_rhsbl_reverse_client multi.uribl.com
>>         reject_rhsbl_reverse_client dbl.spamhaus.org
>>         reject_rhsbl_sender multi.surbl.org
>>         reject_rhsbl_sender multi.uribl.com
>>         reject_rhsbl_sender dbl.spamhaus.org
>>         reject_rhsbl_helo multi.surbl.org
>>         reject_rhsbl_helo multi.uribl.com
>>         reject_rhsbl_helo dbl.spamhaus.org
>>
>> For the time being, and mostly just to see how effective these filters
>> are on their own, I have these listed in my smtpd_recipient_restrictions
>> *prior to* several subsequent reject_rbl_client clauses.  Oddly however,
>> in spite of the ordering, it is appearing to me as if perhaps the above
>> RHS filters are either not actually being applied or else are being applied
>> _after_ the subsequent reject_rbl_client filters.  Certainly, some spam
>> that I believe should have been rejected on the basis of one or another
>> of the above RHS filters I am instead seeing (in my maillog file) being
>> rejected instead by one or another of the subsequent reject_rbl_client
>> filters.   What could I be doing wrong?
>
> Doing RBL client checks in postscreen?

That would be one cause.  Another could be having duplicate
reject_rbl_client statements in smtpd_client_restrictions.

Ron, putting all of your restriction statements under
smtpd_recipient_restrictions and removing the others, i.e.

smtpd_client_restrictions
smtpd_helo_restrictions
smtpd_sender_restrictions

is beneficial especially when troubleshooting things of this nature,
where evaluation order matters.  Since smtpd_delay_reject is enabled by
default you can put all restrictions in smtpd_recipient_restrictions.

http://www.postfix.org/postconf.5.html#smtpd_delay_reject

In your current case, if you have a reject_rbl_client statement in
smtpd_client_restrictions it will evaluate first, and first match wins.
 So the rejection occurs before evaluation reaches say reject_rhsbl_helo
dbl.spamhaus.org in your smtpd_recipient_restrictions.

This may not be what's happening in this case, but putting all
restrictions under smtpd_recipient_restrictions is good practice
regardless.  It also makes complex whitelisting setups much easier, etc,
etc.

--
Stan



Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Stan Hoeppner
In reply to this post by Ronald F. Guilmette-2
On 8/4/2013 10:13 PM, Ronald F. Guilmette wrote:

> In message <[hidden email]>,
> Noel Jones <[hidden email]> wrote:
>
>> On 8/4/2013 8:06 PM, Ronald F. Guilmette wrote:
>>> Does reject_non_fqdn_helo_hostname, when placed in the
>>> smtpd_helo_restrictions, permit clients to HELO/EHLO
>>> with a square-bracket enclosed dotted quad IPv4 address?
>>
>> Yes.
>
> The documentatation should probably be adjusted to make that more clear.
> Right now it reads:
>
>      Reject the request when the HELO or EHLO hostname is not in fully-
>      qualified domain form, as required by the RFC.

You'll find elsewhere in the Postfix documentation and in RFC that
address literals must be accepted.

>>> If so, is the dotted quad checked to see that it properly
>>> represents the actual IP address of the actual current client?
>>
>> No.
>
> Is there any restriction verb that would cause a HELO/EHLO which specifies
> a square-bracketed dotted quad IPv4 address to be rejected when & if the
> dotted quad does not match the actual current client IP address?

No.  You would need to use a policy daemon for something like this.

> Would reject_unknown_helo_hostname do it?  

No.  This rejects when no DNS A or MX record exists for the HELO string.
 This test is skipped for address literals.

> If not maybe a new restriction
> verb would be useful to perform this exact check.

Maybe you should explain why you're having a problem rejecting spamware
that HELO's with an IP literal.  If rejecting based on a HELO string is
your last line of defense you're in trouble Ron.  Surely a spamfighter
of your experience isn't pinning his hopes on HELO. ;)

If your IP literal HELO problem is indeed bot ware, then using
Postscreen will stop these clients, before they have a chance to HELO.

>>> Certainly, some spam
>>> that I believe should have been rejected on the basis of one or another
>>> of the above RHS filters I am instead seeing (in my maillog file) being
>>> rejected instead by one or another of the subsequent reject_rbl_client
>>> filters.   What could I be doing wrong?
>>
>> Doing RBL client checks in postscreen?
>
> I am not using postscreen at the present time.
>
> Do I need to use that if I want to perform RHSBL checks?

No, they are independent of one another.  But if you want to easily stop
bots Postscreen is the way to go.  DNSBL checks in Postscreen are
feature creep, they are not required.  I.e. you can setup default
Postscreen to stop bots and still do your reject_rbl_* checks in
smtpd_client_restrictions.

With your current setup and described problem you could simply remark
all of your reject_rbl_client statements temporarily and see if your
reject_rhsbl_* statements catch anything.

--
Stan

Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Stan Hoeppner
In reply to this post by Ronald F. Guilmette-2
On 8/4/2013 10:13 PM, Ronald F. Guilmette wrote:

> Do I need to use that if I want to perform RHSBL checks?

BTW, if you want to maximize potential hits on RHSBLs just short of
doing body checks, you may want to give Sahil Tandon's TCP server based
RHSBL header checker a spin.  It grabs domains from headers and checks
them against the 3 most popular RHSBLs: DBL, SURBL, and URIBL.  You can
add or subtract from the default server list if you choose.

http://www.hardwarefreak.com/checkdbl.pl.txt

He may have a newer version somewhere.  This is the last version I used.
 I quit using it because it wasn't stopping much with my small mail
stream, after all of my smtpd restrictions.  With larger flows it may be
useful for catching some that may otherwise make it to your content
filter, or make it through entirely if you don't use one.

--
Stan

Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2
In reply to this post by Stan Hoeppner

In message <[hidden email]>,
Stan Hoeppner <[hidden email]> wrote:

>> Doing RBL client checks in postscreen?
>
>That would be one cause.

As I mentioned, I am not using postscreen at the present time.

>Another could be having duplicate
>reject_rbl_client statements in smtpd_client_restrictions.

Nope.  But thank you for mentioning that possibility.

Actually, having adjusted my smtpd_recipient_restrictions rather
dramatically today, and looking now at the day's maillog file,
I think that I am entirely less sure that the problem is what
I said it was earlier.  I am now getting at least _some_
rejects based on SURBL and URIBL, whereas earlier I had not
yet seen any at all in the maillog.

I think that I should ask everyone to ignore my earlier question
about this until I have time to gather more data and see if I can
see what's actually going on.  Perhaps there's nothing wrong at
all!


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2
In reply to this post by Stan Hoeppner

In message <[hidden email]>,
Stan Hoeppner <[hidden email]> wrote:

>> If not maybe a new restriction
>> verb would be useful to perform this exact check.
>
>Maybe you should explain why you're having a problem rejecting spamware
>that HELO's with an IP literal.

Did I say I was having a problem?

There's a difference between "Yea, I could probably spend half an afternoon
hacking up something external that will perform this parcticular check" and
"There's an already-built-in Postfix verb for that."

The latter appeals to my maximally lazy self, but the former doesn't
quite rise to the level of something that I would characterize as
"a problem".

>If rejecting based on a HELO string is
>your last line of defense you're in trouble Ron.

Last, first, does the order make any difference in the end?

>Surely a spamfighter
>of your experience isn't pinning his hopes on HELO. ;)

HELO is _very_ informative.

In the first hour after I re-jiggered my main.cf today, I could already
see spammers trying to HELO with [A.B.C.D].  In contrast to that, I
personally am not aware at the present time of any serious mail server
that I care to receive mail from that HELOs with the [A.B.C.D] style...
even if the RFC does allow it (which we both know it does).

(At some point, everyone running a mail server realizes that the old
admonition to "be liberal in what you accept" has already gone the
way of the dinoasaur some time ago.)

>If your IP literal HELO problem is indeed bot ware, then using
>Postscreen will stop these clients, before they have a chance to HELO.

I don't have any data to tell me what they are, exactly, just yet, and
actually, I don't even mind if they HELO.  I'd just like the simplest
and quickest thing to reject based on HELO with bracketed IP address,
and I'm not real eager to work on setting up postscreen today.  But
thanks for the suggestion.

If I can't reject on bracketed IP in HELO/EHLO then at least I would
have expected Postfix to provide some verb which would have the effect
of at least making sure the bracketed IP is correct.  Oh well. :-(

>> I am not using postscreen at the present time.
>>
>> Do I need to use that if I want to perform RHSBL checks?
>
>No, they are independent of one another.

OK, good.

>But if you want to easily stop
>bots Postscreen is the way to go.

Why?

The combination of 6 or so of the best RBLs, together with SURBL, URIBL,
and Spamhaus DBL seem to be taking care of pretty much everything as of
now, bot or otherwise.  So who am I to argue with success?

>With your current setup and described problem you could simply remark
>all of your reject_rbl_client statements temporarily and see if your
>reject_rhsbl_* statements catch anything.

Good point!  I think 'll try that.  Thanks!


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2
In reply to this post by Stan Hoeppner

In message <[hidden email]>,
Stan Hoeppner <[hidden email]> wrote:

>BTW, if you want to maximize potential hits on RHSBLs just short of
>doing body checks, you may want to give Sahil Tandon's TCP server based
>RHSBL header checker a spin.  It grabs domains from headers and checks
>them against the 3 most popular RHSBLs: DBL, SURBL, and URIBL.

Thank you.  I have just looked at that but I can't see what it does
that makes it in any way superior to the built-in restriction verbs
that I can (and have) already put in main.cf.


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Noel Jones-2
In reply to this post by Ronald F. Guilmette-2
On 8/4/2013 10:13 PM, Ronald F. Guilmette wrote:

> In message <[hidden email]>,
> Noel Jones <[hidden email]> wrote:
>
>> On 8/4/2013 8:06 PM, Ronald F. Guilmette wrote:
>>> Does reject_non_fqdn_helo_hostname, when placed in the
>>> smtpd_helo_restrictions, permit clients to HELO/EHLO
>>> with a square-bracket enclosed dotted quad IPv4 address?
>>
>> Yes.
>
> The documentatation should probably be adjusted to make that more clear.
> Right now it reads:
>
>      Reject the request when the HELO or EHLO hostname is not in fully-
>      qualified domain form, as required by the RFC.
>
>>> If so, is the dotted quad checked to see that it properly
>>> represents the actual IP address of the actual current client?
>>
>> No.
>
> Is there any restriction verb that would cause a HELO/EHLO which specifies
> a square-bracketed dotted quad IPv4 address to be rejected when & if the
> dotted quad does not match the actual current client IP address?

I use a pcre table to reject any HELO that starts with a bracket or
looks like an IP. Legit hosts that use this form are very rare here
-- maybe one every couple years.

>
> Would reject_unknown_helo_hostname do it?  If not maybe a new restriction
> verb would be useful to perform this exact check.

There is  no built-in postfix restriction to compare the HELO to the
client hostname, and I would question the value of such a feature.

Do you see lots of spam with incorrect IP in the HELO? Do you see
significant numbers of legit hosts using a bracketed IP HELO?


>
>>> Certainly, some spam
>>> that I believe should have been rejected on the basis of one or another
>>> of the above RHS filters I am instead seeing (in my maillog file) being
>>> rejected instead by one or another of the subsequent reject_rbl_client
>>> filters.   What could I be doing wrong?
>>

You'll need too show evidence for further help on this.


>>
>> Doing RBL client checks in postscreen?
>
> I am not using postscreen at the present time.
>
> Do I need to use that if I want to perform RHSBL checks?

RHSBL checks work without postscreen.  If you use postscreen, it
will reject clients before the smtpd_*_restrictions (and the smtpd
program itself) are ever run.

http://www.postfix.org/POSTSCREEN_README.html


  -- Noel Jones

>
>
> Regards,
> rfg
>

Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Noel Jones-2
In reply to this post by Ronald F. Guilmette-2
On 8/5/2013 3:16 AM, Ronald F. Guilmette wrote:

> In message <[hidden email]>,
> Stan Hoeppner <[hidden email]> wrote:
>
>> BTW, if you want to maximize potential hits on RHSBLs just short of
>> doing body checks, you may want to give Sahil Tandon's TCP server based
>> RHSBL header checker a spin.  It grabs domains from headers and checks
>> them against the 3 most popular RHSBLs: DBL, SURBL, and URIBL.
>
> Thank you.  I have just looked at that but I can't see what it does
> that makes it in any way superior to the built-in restriction verbs
> that I can (and have) already put in main.cf.
>
>
> Regards,
> rfg
>


The built-in restrictions can check envelope information for
RBL/RHSBL listings.

Sahil's lightweight TCP server can also check message headers such
as Message-ID: and From: header for RHSBL listed domains.  Other
than this clever TCP table, they only other way to check these are
with a milter or content_filter.

This used to catch some extra spam, but hasn't been very effective
for me lately due to changing spammer tactics.  YMMV.



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2
In reply to this post by Noel Jones-2

In message <[hidden email]>,
Noel Jones <[hidden email]> wrote:

>I use a pcre table to reject any HELO that starts with a bracket or
>looks like an IP. Legit hosts that use this form are very rare here
>-- maybe one every couple years.
>...
>There is  no built-in postfix restriction to compare the HELO to the
>client hostname, and I would question the value of such a feature.

Correct me if I'm wrong, but I think you just made the case for
the value of such a feature.

>Do you see lots of spam with incorrect IP in the HELO?

Some.

>Do you see
>significant numbers of legit hosts using a bracketed IP HELO?

None so far.


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Noel Jones-2
On 8/5/2013 12:54 PM, Ronald F. Guilmette wrote:

> In message <[hidden email]>,
> Noel Jones <[hidden email]> wrote:
>
>> I use a pcre table to reject any HELO that starts with a bracket or
>> looks like an IP. Legit hosts that use this form are very rare here
>> -- maybe one every couple years.
>> ...
>> There is  no built-in postfix restriction to compare the HELO to the
>> client hostname, and I would question the value of such a feature.
>
> Correct me if I'm wrong, but I think you just made the case for
> the value of such a feature.

No. Here, near-zero legit clients use bracketed HELO. Looks as if
I've whitelisted 2 clients in the last ~5 years (I see one of them
has fixed their HELO sometime since then).  That's close enough to
zero for me.

My solution is to reject everyone that has a bracketed IP in the
HELO, using a simple check_helo_access pcre map.  I don't care if a
spambot is RFC compliant, I still don't want their mail.

I see zero value in testing to see if the HELO IP is forged, since
using any IP seems to be a very strong spambot indicator.

I know my spam is not your spam, so maybe you see something
different. Provide some evidence if you think this is useful.

To make a case that any new feature is needed, it must be of
widespread benefit, and provide something that cannot (easily) be
done using existing tools. Including sample code and documentation
helps.


I will note that I'm referring to random internet clients and not
authorized SMTP AUTH or mynetworks clients. Desktop mail clients
send all manner of cruft as their HELO, and doing *any* kind of HELO
tests on authorized clients is foolish.


>> Do you see
>> significant numbers of legit hosts using a bracketed IP HELO?
>
> None so far.

The defense rests.


>
>
> Regards,
> rfg
>



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2

In message <[hidden email]>,
Noel Jones <[hidden email]> wrote:

>No. Here, near-zero legit clients use bracketed HELO. Looks as if
>I've whitelisted 2 clients in the last ~5 years (I see one of them
>has fixed their HELO sometime since then).  That's close enough to
>zero for me.

I agree.

>My solution is to reject everyone that has a bracketed IP in the
>HELO, using a simple check_helo_access pcre map.  I don't care if a
>spambot is RFC compliant, I still don't want their mail.

We appear to be in violent agreement.

>I see zero value in testing to see if the HELO IP is forged, since
>using any IP seems to be a very strong spambot indicator.

OK.  Works for me!  I just wish that it wasn't necessary to
have to run an external PCRE to catch it, and that the
reject_non_fqdn_helo_hostname verb actually did what it's name
intutively implies, and what the documentation says it does.

[A.B.C.D] is distinctly _not_ an FQDN.


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Noel Jones-2
On 8/5/2013 4:16 PM, Ronald F. Guilmette wrote:

>> I see zero value in testing to see if the HELO IP is forged, since
>> using any IP seems to be a very strong spambot indicator.
>
> OK.  Works for me!  I just wish that it wasn't necessary to
> have to run an external PCRE to catch it, and that the
> reject_non_fqdn_helo_hostname verb actually did what it's name
> intutively implies, and what the documentation says it does.
>
> [A.B.C.D] is distinctly _not_ an FQDN.

I can see where one might get confused.  I'll submit a one-line doc
patch rather than argue the point.


  -- Noel Jones


>
>
> Regards,
> rfg
>

Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Ronald F. Guilmette-2

In message <[hidden email]>,
Noel Jones <[hidden email]> wrote:

>On 8/5/2013 4:16 PM, Ronald F. Guilmette wrote:
>
>>> I see zero value in testing to see if the HELO IP is forged, since
>>> using any IP seems to be a very strong spambot indicator.
>>
>> OK.  Works for me!  I just wish that it wasn't necessary to
>> have to run an external PCRE to catch it, and that the
>> reject_non_fqdn_helo_hostname verb actually did what it's name
>> intutively implies, and what the documentation says it does.
>>
>> [A.B.C.D] is distinctly _not_ an FQDN.
>
>I can see where one might get confused.  I'll submit a one-line doc
>patch rather than argue the point.

I have no wish to belabor the point either, however for the record
I am not confused.  An FQDN is an FQDN, and [A.B.C.D] quite certainly
isn't one.

I don't know much but I do know that.


Regards,
rfg
Reply | Threaded
Open this post in threaded view
|

smtpd restriction order, rbl dnsbl rhsbl usage -- WAS: Re: Three trivial filtering questions

Stan Hoeppner
In reply to this post by Ronald F. Guilmette-2
On 8/5/2013 2:52 AM, Ronald F. Guilmette wrote:

> Actually, having adjusted my smtpd_recipient_restrictions rather
> dramatically today, and looking now at the day's maillog file,
> I think that I am entirely less sure that the problem is what
> I said it was earlier.  I am now getting at least _some_
> rejects based on SURBL and URIBL, whereas earlier I had not
> yet seen any at all in the maillog.
>
> I think that I should ask everyone to ignore my earlier question
> about this until I have time to gather more data and see if I can
> see what's actually going on.  Perhaps there's nothing wrong at
> all!

Fair enough.  In the mean time I'll share my experience with dnsbl and
rhsbl restrictions.  YMMV.  I use the following *BL restrictions after
all other main.cf restrictions, which is why the absolute numbers are so
low.  RBL checks are the most latency expensive built-in smtpd
restrictions so I evaluate them last.

        reject_rbl_client zen.spamhaus.org
        reject_rbl_client b.barracudacentral.org
        reject_rhsbl_reverse_client dbl.spamhaus.org
        reject_rhsbl_sender dbl.spamhaus.org
        reject_rhsbl_helo dbl.spamhaus.org

'gcml' is my simple keystroke saving script that greps the mail log  for
an input string.

~$ gcml reject|wc -l
9957
~$ gcml reject|grep zen|wc -l
149
~$ gcml reject|grep barracudacentral|wc -l
118
~$ gcml reject|grep dbl|grep Unverified|wc -l
15
~$ gcml reject|grep dbl|grep 'Sender address'|wc -l
30

Here we see the relative effectiveness of *BLs in general on this MX,
accounting for only 3% of rejections.  rhsbls account for 0.3% of
rejections.  45 of 9957 messages may seem not worth the effort, but
that's 45 fewer msgs going through Spamassassin spamd.  Body scanning
those 45 eats far more resources than rejecting the other 9912
connections combined, so for me it's worth the extra 3 lines in main.cf.


On 8/5/2013 3:11 AM, Ronald F. Guilmette wrote:

> Last, first, does the order make any difference in the end?


I'm responding a bit out of context to what you were asking with this in
your other post, but the question is a perfect lead-in to demonstrate
why main.cf restriction order matters.

Note above that reject_rhsbl_helo didn't reject a single connection.
This is most likely due to the fact that snowshoe spammers using $domain
in HELO also use $domain in the rDNS string and/or the envelop sender
address.  So if we rearrange the restriction order to something like this

        reject_rhsbl_helo dbl.spamhaus.org
        reject_rhsbl_reverse_client dbl.spamhaus.org
        reject_rhsbl_sender dbl.spamhaus.org
        reject_rbl_client zen.spamhaus.org
        reject_rbl_client b.barracudacentral.org

then we'll likely see the rhsbl checks rejecting many more connections.
 If I put this entire block near the top of my restrictions the numbers
would skyrocket with a much higher percent of that 9957 being rejected
by *BL Restrictions.

With all restrictions under smtpd_recipient_restrictions they are
processed in strict order.  The first match wins, so if the rhsbl_helo
check evaluates to true the msg is rejected and no other smtpd
restrictions below it are processed.  This strict ordering currently
allows 97% of the spam rejected in smtpd to be killed off with the
fewest resources consumed, first via inbuilt Postfix checks, then by my
various custom tables, and finally by *BL restrictions.  Any spam that
runs this smtpd restriction gauntlet is then rejected by header_checks
and if not enters the queue and is piped to spamd for tagging.

This strict evaluation ordering allows one to optimize resource
consumption per spam connection by using the least expensive
restrictions first and most expensive last.  This is not only critical
for very high volume MTAs, but also for low power machines or those that
serve multiple functions such as SOHO servers.  *BL checks won't
introduce sufficient latency to cause serious delays on the latter class
of machines, but they certainly can on high volume MTAs, even if using a
local resolver and/or rbldnsd.  Obviously it's best to reject spam
connections with the least number of DNS queries possible, from a
latency/throughput and/or resource consumption perspective.

I'm sure already know most of this Ron.  I'm using this opportunity to
get a recent post into the archives covering these topics, hence the
subject change with added search keywords.

--
Stan

Reply | Threaded
Open this post in threaded view
|

Re: Three trivial filtering questions

Stan Hoeppner
In reply to this post by Ronald F. Guilmette-2
On 8/5/2013 6:16 PM, Ronald F. Guilmette wrote:
> In message <[hidden email]>,
> Noel Jones <[hidden email]> wrote:

>>> OK.  Works for me!  I just wish that it wasn't necessary to
>>> have to run an external PCRE to catch it, and that the

PCRE tables don't "run externally".  They're simply tables.  Postfix
smtpd processes them by calling libpcre to which it is linked, just as
it is linked with other libraries including libc, libresolv, libssl, etc
that handle other functions:

~$ pmap 10296
10296:   smtpd -n smtp -t inet -u -c -o stress= -s 2 ...
...
b7104000    200K r-x--  /lib/libpcre.so.3.12.1
b7136000      4K rw---  /lib/libpcre.so.3.12.1
b7137000     12K r-x--  /usr/lib/postfix/dict_pcre.so
b713a000      4K r----  /usr/lib/postfix/dict_pcre.so
b713b000      4K rw---  /usr/lib/postfix/dict_pcre.so
...
b74f3000     88K r-x--  /usr/lib/libsasl2.so.2.0.23
b7509000      4K rw---  /usr/lib/libsasl2.so.2.0.23
...
b773e000     16K r-x--  /var/spool/postfix/lib/libnss_dns-2.11.3.so
b7742000      4K r----  /var/spool/postfix/lib/libnss_dns-2.11.3.so
b7743000      4K rw---  /var/spool/postfix/lib/libnss_dns-2.11.3.so
...
b7763000    184K r-x--  /usr/lib/postfix/smtpd
b7792000      8K r----  /usr/lib/postfix/smtpd
b7794000      4K rw---  /usr/lib/postfix/smtpd
b9178000    264K rw---    [ anon ]
bfd19000    132K rw---    [ stack ]
 total     7664K

>> I can see where one might get confused.  I'll submit a one-line doc
>> patch rather than argue the point.
>
> I have no wish to belabor the point either, however for the record
> I am not confused.  An FQDN is an FQDN, and [A.B.C.D] quite certainly
> isn't one.

Indeed.  You're not the first to be stumped by the discrepancy between
this parameter's behavior and that described in the docs.  I commend
Noel for submitting this long overdue patch.

--
Stan