RBLs in postscreen AND smtpd_*_restrictions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

RBLs in postscreen AND smtpd_*_restrictions

Michael Fox

I think I recall seeing something about this a while ago, but I can’t find it in the archives.

 

If I’m using several RBLs in postscreen_dnsbl_sites, each with its own weighting, then what is the best practice for using at least some of those RBLs in smtpd_*_restrictions, or not? 

 

Thanks,

Michael

 

 

Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox

Clarification:  I seem to recall someone asking whether they should leave RBLs in the smtpd_*_restrictions now that they’ve added them to postscreen.  And I seem to recall that the answer was something like “why not, it doesn’t hurt”.  But it seems to me that if would hurt because: a) it adds a redundant lookup (unless the postscreen cache is shared with postfix) and, b) unless they all have the same weight in postscreen, then postfix would reject on any one RBL, which would make the weighting in postscreen moot.  Hence, my question.

 

Michael

 

 

 

I think I recall seeing something about this a while ago, but I can’t find it in the archives.

 

If I’m using several RBLs in postscreen_dnsbl_sites, each with its own weighting, then what is the best practice for using at least some of those RBLs in smtpd_*_restrictions, or not? 

 

Thanks,

Michael

 

 

Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Wietse Venema
Michael Fox:
> Clarification:  I seem to recall someone asking whether they should leave
> RBLs in the smtpd_*_restrictions now that they've added them to postscreen.
> And I seem to recall that the answer was something like "why not, it doesn't
> hurt".  But it seems to me that if would hurt because: a) it adds a
> redundant lookup (unless the postscreen cache is shared with postfix) and,
> b) unless they all have the same weight in postscreen, then postfix would
> reject on any one RBL, which would make the weighting in postscreen moot.
> Hence, my question.

smtpd and postscreen use the same caching resolver, so the "extra"
queries don't travel far over the network. So the anser is "it
should not hurt".

That said, postscreen versions before 3.1 ignore the DNS reply TTL
(or its equivalent for NXDOMAIN replies) and use postscreen_dnsbl_ttl=1h
by default. That was fine when I wrote postscreen 5 years ago, but
it may be longer than the TTLs that some DNS reputations use these
days.

        Wietse
Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox


> Michael Fox:
> > Clarification:  I seem to recall someone asking whether they should
> leave
> > RBLs in the smtpd_*_restrictions now that they've added them to
> postscreen.
> > And I seem to recall that the answer was something like "why not, it
> doesn't
> > hurt".  But it seems to me that if would hurt because: a) it adds a
> > redundant lookup (unless the postscreen cache is shared with postfix)
> and,
> > b) unless they all have the same weight in postscreen, then postfix
> would
> > reject on any one RBL, which would make the weighting in postscreen
> moot.
> > Hence, my question.
>
> smtpd and postscreen use the same caching resolver, so the "extra"
> queries don't travel far over the network. So the anser is "it
> should not hurt".
>
> That said, postscreen versions before 3.1 ignore the DNS reply TTL
> (or its equivalent for NXDOMAIN replies) and use postscreen_dnsbl_ttl=1h
> by default. That was fine when I wrote postscreen 5 years ago, but
> it may be longer than the TTLs that some DNS reputations use these
> days.

Thanks Wietse.

So, as I understand it:  as long as the weight assigned to a DNSBL in
postscreen is >= postscreen_dnsbl_threshold, then there is no harm in also
adding the same DNSBL to smtpd_*_restrictions.  And, conversely, DNSBLs with
weights < postscreen_dnsbl_threshold should not be listed in
smtpd_*_restrictions because they could block an email on their own, even
though they are not trusted to do so by postscreen.

If a DNSBL in postscreen_dnsbl_sites has a weight >=
postscreen_dnsbl_threshold, then is there any advantage to also listing it
in smtpd_*_restrictions?  For example, is there some failure mode that
having the DNSBL listed in both places would protect against?

Michael


Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

allenc


On 02/06/16 17:45, Michael Fox wrote:
> If a DNSBL in postscreen_dnsbl_sites has a weight >=
> postscreen_dnsbl_threshold, then is there any advantage to also
> listing it in smtpd_*_restrictions? For example, is there some failure
> mode that having the DNSBL listed in both places would protect
> against? Michael

I frequently have remote hosts which pass the pregreet and DNSBL tests,
and then repeatedly access the server with a "PASS OLD" result from
postscreen.  Usually they try to send unauthorised relay messages.

The entry in smtpd_*_restrictions would pick these up as their DNSBL
status changes.

Allen C

Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox
> On 02/06/16 17:45, Michael Fox wrote:
> > If a DNSBL in postscreen_dnsbl_sites has a weight >=
> > postscreen_dnsbl_threshold, then is there any advantage to also
> > listing it in smtpd_*_restrictions? For example, is there some failure
> > mode that having the DNSBL listed in both places would protect
> > against? Michael
>
> I frequently have remote hosts which pass the pregreet and DNSBL tests,
> and then repeatedly access the server with a "PASS OLD" result from
> postscreen.  Usually they try to send unauthorised relay messages.
>
> The entry in smtpd_*_restrictions would pick these up as their DNSBL
> status changes.
>
> Allen C

Thanks Allen.

Ahhh.  
So, taking into account what Wietse just said about DNSBL lookups in
postscreen and smtpd sharing the same caching resolver, then, if I
understand you correctly, adding the same DNSBL to smtpd_*_restrictions
would catch the case where postscreen_dnsbl_ttl has expired for a given
client, but postscreen_cache_retention_time (default=7d) has not.  Is that
correct?

Michael


Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

allenc


On 02/06/16 19:21, Michael Fox wrote:

>> On 02/06/16 17:45, Michael Fox wrote:
>>> If a DNSBL in postscreen_dnsbl_sites has a weight >=
>>> postscreen_dnsbl_threshold, then is there any advantage to also
>>> listing it in smtpd_*_restrictions? For example, is there some failure
>>> mode that having the DNSBL listed in both places would protect
>>> against? Michael
>> I frequently have remote hosts which pass the pregreet and DNSBL tests,
>> and then repeatedly access the server with a "PASS OLD" result from
>> postscreen.  Usually they try to send unauthorised relay messages.
>>
>> The entry in smtpd_*_restrictions would pick these up as their DNSBL
>> status changes.
>>
>> Allen C
> Thanks Allen.
>
> Ahhh.  
> So, taking into account what Wietse just said about DNSBL lookups in
> postscreen and smtpd sharing the same caching resolver, then, if I
> understand you correctly, adding the same DNSBL to smtpd_*_restrictions
> would catch the case where postscreen_dnsbl_ttl has expired for a given
> client, but postscreen_cache_retention_time (default=7d) has not.  Is that
> correct?
>
> Michael
>
>
>
As I understand it, yes.

Once a new remote host has been accepted by postscreen, it becomes
difficult to "un-whitelist" it.    It has to be dealt with elsewhere...

Allen C

Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Wietse Venema
In reply to this post by Michael Fox
Michael Fox:

> > On 02/06/16 17:45, Michael Fox wrote:
> > > If a DNSBL in postscreen_dnsbl_sites has a weight >=
> > > postscreen_dnsbl_threshold, then is there any advantage to also
> > > listing it in smtpd_*_restrictions? For example, is there some failure
> > > mode that having the DNSBL listed in both places would protect
> > > against? Michael
> >
> > I frequently have remote hosts which pass the pregreet and DNSBL tests,
> > and then repeatedly access the server with a "PASS OLD" result from
> > postscreen.  Usually they try to send unauthorised relay messages.
> >
> > The entry in smtpd_*_restrictions would pick these up as their DNSBL
> > status changes.
> >
> > Allen C
>
> Thanks Allen.
>
> Ahhh.  
> So, taking into account what Wietse just said about DNSBL lookups in
> postscreen and smtpd sharing the same caching resolver, then, if I
> understand you correctly, adding the same DNSBL to smtpd_*_restrictions
> would catch the case where postscreen_dnsbl_ttl has expired for a given
> client, but postscreen_cache_retention_time (default=7d) has not.  Is that
> correct?

postscreen will query the DNS when the client connects after
postscreen_dnsbl_ttl has expired. With Postfix 3.1 and later,
that time is (also) determined by a TTL in the DNS response.

        Wietse
Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox
> postscreen will query the DNS when the client connects after
> postscreen_dnsbl_ttl has expired. With Postfix 3.1 and later,
> that time is (also) determined by a TTL in the DNS response.

Thanks for the clarification Wietse.  2 questions:

1)  Given that DNSBLs in postscreen_dnsbl_sites and smtpd_*_restrictions use
the same caching resolver and the same timeouts, they should produce the
same result.  Correct?  If so, then is there any reason at all for putting a
DNSBL in smtpd_*_restrictions if postscreen is already set up with a set of
weighted DNSBLs?  Or, put another way, is there any failure mode that
listing the DNSBL in both places might prevent?  Please explain.

2)  Please confirm my understanding of the postscreen_cache_retention_time:
2a)  A client that previously passed the pre-greet test will face the
pre-greet test again if postscreen_greet_ttl has expired, even if
postscreen_cache_retention_time has not expired.  Correct?
2b)  A client that previously passed the DNSBL test will be face the DNSBL
test again if postscreen_dnsbl_ttl has expired, even if
postscreen_cache_retention_time has not expired.  Correct?
2c)  A client that has previously passed the pre-greet and DNSBL tests, and
connects again before postscreen_cache_retention_time has expired, will be
logged as "PASS OLD" instead of "PASS NEW", regardless of whether or not the
pre-greet and/or DNSBL tests had to be rerun this time.  Correct?
2d)  The result of "tests after the 220 SMTP server greeting" are cached for
postscreen_cache_retention_time.  Correct?

Thanks,
Michael


Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Wietse Venema
Michael Fox:
> > postscreen will query the DNS when the client connects after
> > postscreen_dnsbl_ttl has expired. With Postfix 3.1 and later,
> > that time is (also) determined by a TTL in the DNS response.
>
> Thanks for the clarification Wietse.  2 questions:
>
> 1)  Given that DNSBLs in postscreen_dnsbl_sites and smtpd_*_restrictions use
> the same caching resolver and the same timeouts, they should produce the
> same result.  Correct?  

Each smtpd process has a short-lived cache (process lifetime).

Postscreen has postscreen_dnsbl_ttl (fixed time limit) or it uses
the DNS TTL, limited by postscreen_dnsbl_{min,max}_ttl.

Please see Postfix documentatiom, and report a bug if it is incomplete.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Bill Cole-3
In reply to this post by Michael Fox
On 2 Jun 2016, at 12:45, Michael Fox wrote:

> So, as I understand it:  as long as the weight assigned to a DNSBL in
> postscreen is >= postscreen_dnsbl_threshold, then there is no harm in
> also
> adding the same DNSBL to smtpd_*_restrictions.

True.

But this is not the whole story...

> And, conversely, DNSBLs with
> weights < postscreen_dnsbl_threshold should not be listed in
> smtpd_*_restrictions because they could block an email on their own,
> even
> though they are not trusted to do so by postscreen.

Not in all cases. Where postscreen by necessity offers limited ability
to whitelist sessions from DNSBL interference, the threshold mechanism
allows lists with different sorts of exceptional cases to block mail
only when multiple lists concur on a listing, effectively whitelisting
IPs based on how many lists of what scores they appear on. Yet you may
want to trust those lists as a basis to reject mail by default on their
own, with exceptions based on criteria that postscreen cannot use.

Consider this hypothetical example based on a substantial simplification
and minor obfuscation of my own personal system's config:

smtpd_recipient_restrictions = permit_mynetworks,
    check_sender_access regexp:/opt/local/etc/postfix/badsenders
    reject_unknown_sender_domain, reject_unauth_destination,
    check_sender_access regexp:/opt/local/etc/postfix/goodsenders,
    reject_unknown_reverse_client_hostname,
    check_sender_mx_access cidr:/opt/local/etc/postfix/bogus_mx.cidr
    reject_rbl_client zen.spamhaus.org
    check_recipient_access
regexp:/opt/local/etc/postfix/recipient_checks.regex,
    reject_rbl_client badrirblocks.local,
    reject_rbl_client deathstarblocks.local,
    reject_rbl_client overaggressive.example.com,
    reject_rbl_client staleentries.example.com,
    reject_rbl_client korea.services.net,
    reject_rbl_client psbl.surriel.com,
    reject_rbl_client ix.dnsbl.manitu.net,
    check_client_access
regexp:/opt/local/etc/postfix/client_checks.regex,
    permit

postscreen_access_list = permit_mynetworks,
    cidr:/opt/local/etc/postfix/ps-noblock.cidr

postscreen_dnsbl_sites = zen.spamhaus.org*4
    badrirblocks.local*1
    deathstarblocks.local*2
    overaggressive.example.com*2
    staleentries.example.com*1
    korea.services.net*3
    psbl.surriel.com*3
    ix.dnsbl.manitu.net*3

postscreen_dnsbl_threshold = 4

So, I have 2 distinct local DNSBLs. One (badrirblocks.local) includes
(hypothetically) every RIR-level allocation from which any of the few
users here have received one or more spams and no ham to untagged
addresses. In the other (deathstarblocks.local) I have tried imperfectly
to list all the ranges allocated to residential customers by a specific
very large (and hence inevitably a bit evil) ISP who shall remain
nameless. I also put almost absolute trust in SpamHaus and use various
other public DNSBLs that either have known false positive problems or
might occasionally correctly (by their standards) list an address from
which I'd like to be able to receive mail to SOME recipients. To be
rejected by postscreen, an address must either be in Zen OR at least 2
(maybe 3) of the other DNSBLs. (Note that I actually have a more
nuanced/messy config of different scores for various lists' specific
return values but I've reduced it to this level of simplicity for the
sake of clarity.)

Note that "goodsenders" is a kludge. It's a few magic sender patterns,
any of them entirely forgeable, but obscure enough that forgery is
hugely improbable: people whose mail I want NEVER to lose, even if
there's a momentary glitch listing at SpamHaus. So is ps-noblock.cidr: a
few networks which could earn 'escalated' SBL listings but sometimes
send mail here that is very much wanted as well as trusted networks from
whence I could accidentally trigger a PREGREET test in trying to
troubleshoot/verify mail functionality via telnet. (I have a lethal
tripwire on PREGREET...)

The key to this config pattern is recipient_checks.regex. It has a bunch
of special recipient addresses and patterns that are exempt from the
DNSBLs that follow it, as well as a number of addresses and patterns
that have been leaked and/or abused to the point where I've made them
traps. The overall result: I don't bother with DNSBL checks or content
filtering for addresses that have become spamtraps and I skip over some
DNSBL checks for recipient addresses that I deem special, including many
but not all tagged addresses. The overwhelming majority of mail hitting
this server which gets past postscreen either gets blocked ahead of the
DNSBL checks in smtpd or gets exempted from the latter set of DNSBLs,
which still manage to properly catch hundreds of messages per day that
would otherwise need to be content-filtered. I actually use PSBL and IX
with no whitelisting (and no known false positives) on some non-Postfix
systems where I don't have the luxury of nuance.

Of course, not everyone needs or wants this level of complexity. This is
for a server that has a small number of users with about half of us
trying to keep long-lived and heavily spam-targeted addresses usable. I
wouldn't do this for 500 users.

> If a DNSBL in postscreen_dnsbl_sites has a weight >=
> postscreen_dnsbl_threshold, then is there any advantage to also
> listing it
> in smtpd_*_restrictions?  For example, is there some failure mode that
> having the DNSBL listed in both places would protect against?

As noted by Allen Coates, if postscreen_dnsbl_ttl (v2.8-v3.0) or
postscreen_dnsbl_min_ttl (3.1) is higher than the negative cache TTL for
a DNSBL, postscreen can 'PASS OLD' an IP which has been listed in the
period since its prior connection during which the resolver cache of the
negative result has expired, so smtpd can see a listing that postscreen
does not.
Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox
> > And, conversely, DNSBLs with
> > weights < postscreen_dnsbl_threshold should not be listed in
> > smtpd_*_restrictions because they could block an email on their own,
> > even
> > though they are not trusted to do so by postscreen.
>
> Not in all cases. Where postscreen by necessity offers limited ability
> to whitelist sessions from DNSBL interference, the threshold mechanism
> allows lists with different sorts of exceptional cases to block mail
> only when multiple lists concur on a listing, effectively whitelisting
> IPs based on how many lists of what scores they appear on. Yet you may
> want to trust those lists as a basis to reject mail by default on their
> own, with exceptions based on criteria that postscreen cannot use.
 
[hypothetical example clipped]

Thank you very much Bill!  I had not thought of that type of situation and
that's exactly the type of example that I needed to help me understand the
nuance.

>
> > If a DNSBL in postscreen_dnsbl_sites has a weight >=
> > postscreen_dnsbl_threshold, then is there any advantage to also
> > listing it
> > in smtpd_*_restrictions?  For example, is there some failure mode that
> > having the DNSBL listed in both places would protect against?
>
> As noted by Allen Coates, if postscreen_dnsbl_ttl (v2.8-v3.0) or
> postscreen_dnsbl_min_ttl (3.1) is higher than the negative cache TTL for
> a DNSBL, postscreen can 'PASS OLD' an IP which has been listed in the
> period since its prior connection during which the resolver cache of the
> negative result has expired, so smtpd can see a listing that postscreen
> does not.

This is what I'm still a little fuzzy on.

In v2.8-3.0, and assuming the postscreen_dnsbl_ttl default of 1 hour, I can
understand Alan's example.  A new bad guy starts up and gets through the
first time, which postscreen caches for a default of 1 hour.  The bad guy
then starts his attacks within that hour of time and gets added to DNSBLs.
But postscreen doesn't see it for an hour, while smtpd does (assuming the
actual DNSBL's TTL is < 1 hour).  I think I understand that part.

But in v3.1, with the default postscreen_dnsbl_min_ttl of 60 seconds, and
postscreen using the actual TTL from the DNSBL, it seems like that issue no
longer exists.  Here's my understanding:  Using the same scenario,
postscreen caches the positive answer on the first connect.  And, according
to Wietse, smtpd uses the same caching resolver.  So when smtpd does the
lookup probably less than a second later, it should get the answer from the
resolver's cache, which should be the same answer that postscreen got.  Now
the bad guy starts his attacks and gets added to the DNSBLs.  If the TTL has
not yet expired, then postscreen uses it's positive cached value and smtpd
will still get a positive answer from the resolver's cache.  If the TTL has
expired, then both postscreen and smtpd would get a new, negative answer.
There may be miniscule differences in timing.  But I would think the results
would be virtually identical.

Is that right?  If not, what am I missing?

Also, for pre v3.1 users, is there a best practice recommended value for
postscreen_dnsbl_ttl that is better than the default of 1 hour?

Thanks,
Michael






Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Bill Cole-3
On 4 Jun 2016, at 1:52, Michael Fox wrote:

[Quoting me]

>> As noted by Allen Coates, if postscreen_dnsbl_ttl (v2.8-v3.0) or
>> postscreen_dnsbl_min_ttl (3.1) is higher than the negative cache TTL
>> for
>> a DNSBL, postscreen can 'PASS OLD' an IP which has been listed in the
>> period since its prior connection during which the resolver cache of
>> the
>> negative result has expired, so smtpd can see a listing that
>> postscreen
>> does not.
>
> This is what I'm still a little fuzzy on.
>
> In v2.8-3.0, and assuming the postscreen_dnsbl_ttl default of 1 hour,
> I can
> understand Alan's example.  A new bad guy starts up and gets through
> the
> first time, which postscreen caches for a default of 1 hour.  The bad
> guy
> then starts his attacks within that hour of time and gets added to
> DNSBLs.
> But postscreen doesn't see it for an hour, while smtpd does (assuming
> the
> actual DNSBL's TTL is < 1 hour).  I think I understand that part.
>
> But in v3.1, with the default postscreen_dnsbl_min_ttl of 60 seconds,
> and
> postscreen using the actual TTL from the DNSBL, it seems like that
> issue no
> longer exists.  Here's my understanding:  Using the same scenario,
> postscreen caches the positive answer on the first connect.  And,
> according
> to Wietse, smtpd uses the same caching resolver.

Maybe the confusion is here: postscreen and smtpd use the same caching
resolver because they run on the same machine and use whatever caching
resolver that system is configured to use. It is not part of Postfix.
Ideally this is a nameserver like Unbound or BIND on the same machine or
same LAN, but some people aren't concerned with performance or the
ability to use public DNSBLs and point their systems to Google or
OpenDNS or their ISP's resolvers.

Postscreen keeps its own internal DNSBL result cache for performance and
while it may seem counter-intuitive, doing so also is probably less
resource-intensive than depending on the system resolver and whatever
cache it ultimately uses. This cache is NOT usable by smtpd processes.

> So when smtpd does the
> lookup probably less than a second later, it should get the answer
> from the
> resolver's cache, which should be the same answer that postscreen got.
>  Now
> the bad guy starts his attacks and gets added to the DNSBLs.  If the
> TTL has
> not yet expired, then postscreen uses it's positive cached value and
> smtpd
> will still get a positive answer from the resolver's cache.  If the
> TTL has
> expired, then both postscreen and smtpd would get a new, negative
> answer.
> There may be miniscule differences in timing.  But I would think the
> results
> would be virtually identical.
>
> Is that right?  If not, what am I missing?

If postscreen_dnsbl_ttl or postscreen_dnsbl_min_ttl is 3600 (1 hour) but
the minimum TTL field of the DNSBL's SOA record is 10 (as it is for the
SBL) then postscreen will cache the lack of a DNSBL entry for as much as
59 minutes and 50 seconds longer than a proper caching resolver, which
will keep the negative response for 10 seconds at most (unless it's an
older Microsoft DNS server or a broken Unbound instance that has been
given a minimum TTL...)

> Also, for pre v3.1 users, is there a best practice recommended value
> for
> postscreen_dnsbl_ttl that is better than the default of 1 hour?

I cannot tell you who you are... :)

However, for almost everyone, an hour is too long. The 22 DNSBL's I find
interesting enough to have in my own multi-list tool (which are NOT all
fit for postscreen) have negative cache TTLs (the minimum TTL field of
the SOA record) that range from 0 to a whole day. There's absolutely no
reason for postscreen_dnsbl_ttl to be shorter than the negative cache
TTL of the DNSBLs you use. The postscreen_dnsbl_min_ttl default in 3.1
is 60s, which I think is reasonable for postscreen_dnsbl_ttl even though
it is longer than the TTL of some of the most useful DNSBLs (like the
Spamhaus lists.) However, if your resolver is distant, slow, or
overloaded you might not want to go that short. If you have smtpd
checking the same DNSBLs and have reasonable CPU and memory to spare,
you might even go as high as 600s: smtpd is heavier than postscreen but
it's not so bad unless you handle really high volumes of mail.
Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox

> If postscreen_dnsbl_ttl or postscreen_dnsbl_min_ttl is 3600 (1 hour) but
> the minimum TTL field of the DNSBL's SOA record is 10 (as it is for the
> SBL) then postscreen will cache the lack of a DNSBL entry for as much as
> 59 minutes and 50 seconds longer than a proper caching resolver, which
> will keep the negative response for 10 seconds at most (unless it's an
> older Microsoft DNS server or a broken Unbound instance that has been
> given a minimum TTL...)

Right.  As I mentioned, I understand that part.  My question was about v3.1+
where the default for postscreen_dnsbl_min_ttl is only 60s.  And, as I
understand it, the defaults for v3.1 would cause both the postscreen cache
ttl and the system resolver's cache ttl to be based on the same ttl from the
actual DNSBL record, thus rendering the same result for both, and the same
timeout for both.

Unless I've got that wrong, no need to respond.  


> > Also, for pre v3.1 users, is there a best practice recommended value
> > for
> > postscreen_dnsbl_ttl that is better than the default of 1 hour?
>
> I cannot tell you who you are... :)

Of course not.  But we can all learn from the experience and reasoning of
others.  What you're doing doesn't necessarily apply to what I need.  But
your willingness to explain both what you did and WHY you did it helps me to
understand the choices and consequences that I might also need to consider.
VERY helpful.


> However, for almost everyone, an hour is too long. [clipped]

O.K.  Your description helped verify that my own reasoning was on the right
track.

Thanks again for taking the time to explain.

Michael


Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Peter Ajamian
On 05/06/16 17:10, Michael Fox wrote:
> Right.  As I mentioned, I understand that part.  My question was about v3.1+
> where the default for postscreen_dnsbl_min_ttl is only 60s.  And, as I
> understand it, the defaults for v3.1 would cause both the postscreen cache
> ttl and the system resolver's cache ttl to be based on the same ttl from the
> actual DNSBL record, thus rendering the same result for both, and the same
> timeout for both.
>
> Unless I've got that wrong, no need to respond.  

I think you have it mostly right, but there are some cases where the
results could differ between postscreen and smtpd:

1.  There will be a very small window of time (we're talking
milliseconds) between when postscreen checks the expire time and when
smtpd attempts to lookup the record.  The DNS record could expire during
this very small window of time and if it has changed since the last time
that the resolver fetched the record the result could be different.

2.  The resolver might be broken and not caching the record, or caching
it for a shorter or longer period of time than the TTL states.

3.  You could (as is common) have two different resolvers listed in your
resolv.conf (or your OSes equivalent) file.  These resolvers could have
cached the record at different times, and if the record was updated in
between they could have different results.  It is possible that
postscreen could have randomly hit one resolver and smtpd hits the other
thereby giving different results.


Peter
Reply | Threaded
Open this post in threaded view
|

RE: RBLs in postscreen AND smtpd_*_restrictions

Michael Fox
Got it.  Thanks much Peter!


Michael


> -----Original Message-----
> From: [hidden email] [mailto:owner-postfix-
> [hidden email]] On Behalf Of Peter
> Sent: Saturday, June 4, 2016 11:30 PM
> To: [hidden email]
> Subject: Re: RBLs in postscreen AND smtpd_*_restrictions
>
> On 05/06/16 17:10, Michael Fox wrote:
> > Right.  As I mentioned, I understand that part.  My question was about
> v3.1+
> > where the default for postscreen_dnsbl_min_ttl is only 60s.  And, as I
> > understand it, the defaults for v3.1 would cause both the postscreen
> cache
> > ttl and the system resolver's cache ttl to be based on the same ttl from
> the
> > actual DNSBL record, thus rendering the same result for both, and the
> same
> > timeout for both.
> >
> > Unless I've got that wrong, no need to respond.
>
> I think you have it mostly right, but there are some cases where the
> results could differ between postscreen and smtpd:
>
> 1.  There will be a very small window of time (we're talking
> milliseconds) between when postscreen checks the expire time and when
> smtpd attempts to lookup the record.  The DNS record could expire during
> this very small window of time and if it has changed since the last time
> that the resolver fetched the record the result could be different.
>
> 2.  The resolver might be broken and not caching the record, or caching
> it for a shorter or longer period of time than the TTL states.
>
> 3.  You could (as is common) have two different resolvers listed in your
> resolv.conf (or your OSes equivalent) file.  These resolvers could have
> cached the record at different times, and if the record was updated in
> between they could have different results.  It is possible that
> postscreen could have randomly hit one resolver and smtpd hits the other
> thereby giving different results.
>
>
> Peter

Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Bill Cole-3
In reply to this post by Peter Ajamian
On 5 Jun 2016, at 2:30, Peter wrote:

> On 05/06/16 17:10, Michael Fox wrote:
>> Right.  As I mentioned, I understand that part.  My question was
>> about v3.1+
>> where the default for postscreen_dnsbl_min_ttl is only 60s.  And, as
>> I
>> understand it, the defaults for v3.1 would cause both the postscreen
>> cache
>> ttl and the system resolver's cache ttl to be based on the same ttl
>> from the
>> actual DNSBL record, thus rendering the same result for both, and the
>> same
>> timeout for both.
>>
>> Unless I've got that wrong, no need to respond.
>
> I think you have it mostly right, but there are some cases where the
> results could differ between postscreen and smtpd:
>
> 1.  There will be a very small window of time (we're talking
> milliseconds) between when postscreen checks the expire time and when
> smtpd attempts to lookup the record.  The DNS record could expire
> during
> this very small window of time and if it has changed since the last
> time
> that the resolver fetched the record the result could be different.
>
> 2.  The resolver might be broken and not caching the record, or
> caching
> it for a shorter or longer period of time than the TTL states.
>
> 3.  You could (as is common) have two different resolvers listed in
> your
> resolv.conf (or your OSes equivalent) file.  These resolvers could
> have
> cached the record at different times, and if the record was updated in
> between they could have different results.  It is possible that
> postscreen could have randomly hit one resolver and smtpd hits the
> other
> thereby giving different results.

4. The resolver cache honors (as most do) a DNSBL's negative cache TTL
which is less than 60 seconds, e.g. Spamcop (0 seconds) or the various
Spamhaus lists (10) and others.
Reply | Threaded
Open this post in threaded view
|

Documentation improvement request (was: RBLs in postscreen AND smtpd_*_restrictions)

Peter Ajamian
In reply to this post by Wietse Venema
On 03/06/16 22:20, Wietse Venema wrote:
> Postscreen has postscreen_dnsbl_ttl (fixed time limit) or it uses
> the DNS TTL, limited by postscreen_dnsbl_{min,max}_ttl.
>
> Please see Postfix documentatiom, and report a bug if it is incomplete.

dnsblog(8) states, "Otherwise it replies with the query arguments plus
an empty address list and the reply TTL (-1 if unavailable)."  It is
unclear that this references the negative cache TTL as returned by the
SOA record included in an NXDOMAIN response.

I had to look at the dnsblog.c source code for this to become clear.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: RBLs in postscreen AND smtpd_*_restrictions

Peter Ajamian
In reply to this post by Bill Cole-3
On 07/06/16 01:07, Bill Cole wrote:
> 4. The resolver cache honors (as most do) a DNSBL's negative cache TTL
> which is less than 60 seconds, e.g. Spamcop (0 seconds) or the various
> Spamhaus lists (10) and others.

postscreen (specifically dnsblog(8)) honors this as well, but it's not
made entirely clear in the docs.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Documentation improvement request

Peter Ajamian
In reply to this post by Peter Ajamian
On 07/06/16 08:49, Peter wrote:

> On 03/06/16 22:20, Wietse Venema wrote:
>> Postscreen has postscreen_dnsbl_ttl (fixed time limit) or it uses
>> the DNS TTL, limited by postscreen_dnsbl_{min,max}_ttl.
>>
>> Please see Postfix documentatiom, and report a bug if it is incomplete.
>
> dnsblog(8) states, "Otherwise it replies with the query arguments plus
> an empty address list and the reply TTL (-1 if unavailable)."  It is
> unclear that this references the negative cache TTL as returned by the
> SOA record included in an NXDOMAIN response.

...also it would help to reference dnsblog(8) from the
postscreen_dnsbl_*_ttl entries in postconf(5), just to tie that together.


Peter
12