smtpd_delay_reject with rspamd milter

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

smtpd_delay_reject with rspamd milter

Kai Schaetzl
I'm having trouble with access_maps kicking in after an upgrade from a
Postfix 2.something to Postfix 3.1. on Ubuntu 14.06 and using postscreen
and rspamd milter.

After some testing I'm not sure yet, but it looks like the recommended
smtpd_delay_reject = yes in connection with having the access_map checks
listed in the smtpd_recipient_restrictions is the cause of this. It allows
the milter to kick in, so that the access_check (almost?) never happens.

After changing to
smtpd_delay_reject = no
and moving the client access_maps to smtpd_client_restrictions they seem
to be working again.

rspamd is used as a milter (as specified in the rspamd docs) and not as a
policy service (which would allow putting it at the end of restrictions).

# rspamd
smtpd_milters = inet:localhost:11332
milter_protocol = 6
milter_mail_macros =  i {mail_addr} {client_addr} {client_name}
{auth_authen}
milter_default_action = accept


Is this a known problem? Other workarounds? Can I specify to use the
milter only after the restrictions? (Maybe doesn't make sense?)
Or am I misinterpreting something?

Thanks!

Kai


Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Carsten Rosenberg
Kai,

both are running simultaneously. So at smtpd_recipient_restriction stat
the milter will also get the recipients. As far as I have seen the
postfix restriction react faster.

So if you reject somebody with an access_map, you won't see any scan
result in rspamd. Only the milter connect, because rspamd starts to work
at the end_of_data_restrictions.

Do you have any problems with this situation?

--

Carsten

On 07.11.18 16:10, Kai Schaetzl wrote:

> I'm having trouble with access_maps kicking in after an upgrade from a
> Postfix 2.something to Postfix 3.1. on Ubuntu 14.06 and using postscreen
> and rspamd milter.
>
> After some testing I'm not sure yet, but it looks like the recommended
> smtpd_delay_reject = yes in connection with having the access_map checks
> listed in the smtpd_recipient_restrictions is the cause of this. It allows
> the milter to kick in, so that the access_check (almost?) never happens.
>
> After changing to
> smtpd_delay_reject = no
> and moving the client access_maps to smtpd_client_restrictions they seem
> to be working again.
>
> rspamd is used as a milter (as specified in the rspamd docs) and not as a
> policy service (which would allow putting it at the end of restrictions).
>
> # rspamd
> smtpd_milters = inet:localhost:11332
> milter_protocol = 6
> milter_mail_macros =  i {mail_addr} {client_addr} {client_name}
> {auth_authen}
> milter_default_action = accept
>
>
> Is this a known problem? Other workarounds? Can I specify to use the
> milter only after the restrictions? (Maybe doesn't make sense?)
> Or am I misinterpreting something?
>
> Thanks!
>
> Kai
>
>
Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Kai Schaetzl
Carsten Rosenberg wrote on Wed, 7 Nov 2018 16:23:54 +0100:

> So if you reject somebody with an access_map, you won't see any scan
> result in rspamd.

This would be fine ;-)

> Do you have any problems with this situation?

Yes, it's the other way around here. e.g. there is no rejection happening
by postfix, but the milter kicks in and greylists the mail (if it scores
enough the first time) and after greylisting scans it and scores
accordingly. But I would rather like it to get rejected by postfix because
of the access_map.

I have some generic TLDs listed that deliver only garbage, like .site,
host, .review etc. They were getting scored as spam by rspamd most of the
time but I wondered why they weren't getting rejected by postfix, anyway.
First I thought I might be using wrong syntax (site vs. .site), but I
scanned the postfix docs and found that the default compatibility setting
for access_maps should allow "site" to be used for subdomain matching as
well.
Now, after removing the delay it seems that postfix is now rejecting them.

I'm not 100% sure if that did it, because I have some sender rejects that
*may* have been before my changes. But never a client reject. I'm not sure
because I made several changes over the course of the day and am not sure
about exact times.

So, this seems to work now, but I've just realized I hit a new problem.
After smtpd_delay_reject = no the option permit_sasl_authenticated doesn't
work in permit_sasl_authenticated anymore. I had to revert to yes,
otherwise the checks *after* permit_sasl_authenticated hit the message and
reject it. After thinking about this, it's clear that if I check at helo
stage there hasn't been any authentication yet, permit_sasl_authenticated
is moot at this stage. If I want it and still use some rejections because
of helo I *have* to delay.

Is there a workaround for this which allows client and sender rejections
and have the milter kick in only after this?
Here's my current conf in this area:
(smtpd_client_restrictions was empty before today and most of the
restrictions had been in recipient_restrictions)

smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_relay_restrictions = permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination
smtpd_helo_restrictions =
    permit_sasl_authenticated,(obviously in vain)
    #permit_mynetworks,
    reject_invalid_helo_hostname,
    reject_non_fqdn_helo_hostname,
    reject_unknown_helo_hostname,
    check_helo_access hash:/etc/mail/access,
    check_helo_access hash:/etc/mail/disallow_my_domains,
    permit
#http://www.postfix.org/postconf.5.html#smtpd_relay_restrictions:
smtpd_client_restrictions =
    permit_sasl_authenticated,
    permit_mynetworks,
    sleep 1,
    reject_unauth_pipelining,
    check_client_access hash:/etc/mail/allow_clients,
    check_client_access hash:/etc/mail/access,
    reject_invalid_hostname,
    reject_unknown_client_hostname,
    permit
smtpd_sender_restrictions =
smtpd_recipient_restrictions =
    reject_non_fqdn_sender,
    reject_non_fqdn_recipient,
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination,
    reject_unknown_sender_domain,
    reject_unknown_recipient_domain,
    reject_unlisted_recipient,
    check_recipient_access hash:/etc/mail/allow_recipients,
    check_sender_access hash:/etc/mail/allow_senders,
    #check_client_access hash:/etc/mail/allow_clients,
    #check_client_access hash:/etc/mail/access,
    check_sender_access hash:/etc/mail/access,
    #reject_invalid_hostname,
    #reject_unknown_client_hostname,
    #reject_rbl_client ix.dnsbl.manitu.net,
    #check_policy_service inet:127.0.0.1:10023,
    check_policy_service inet:127.0.0.1:10024,
    permit
smtpd_data_restrictions = reject_unauth_pipelining,
reject_multi_recipient_bounce


Kai


Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Kai Schaetzl
Addendum.

Currently, I get client rejections with the setup shown in my last mail
(despite the delay). I don't know if it hits *always*, though. I can't
check if it didn't hit for some client where the name matches, there are
too many entries.

I expected it to carry out the helo checks before client checks.
e.g. in a "natural" order of helo, client, sender, rcpt.
Was this assumption wrong?

Example:

Nov  7 17:35:53 b04 postfix/smtpd[30957]: NOQUEUE: reject: RCPT from
com.check-prfofessional.online[185.52.2.216]: 554 5.7.1 <com.check-
prfofessional.online[185.52.2.216]>: Client host rejected: Access denied;
from=<[hidden email]> to=<[hidden email]>
proto=ESMTP helo=<com.check-prfofessional.online>

The helo contains the same name, so a helo check should have hit it. Which
means, helo checks are done after the client check?

Kai


Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Wietse Venema
Kai Schaetzl:
[ Charset ISO-8859-1 converted... ]

> Addendum.
>
> Currently, I get client rejections with the setup shown in my last mail
> (despite the delay). I don't know if it hits *always*, though. I can't
> check if it didn't hit for some client where the name matches, there are
> too many entries.
>
> I expected it to carry out the helo checks before client checks.
> e.g. in a "natural" order of helo, client, sender, rcpt.
> Was this assumption wrong?

The order of evaluation is in

        http://www.postfix.org/SMTPD_ACCESS_README.html

Postfix evaluates client restrictions before helo_restrictions
before sender restrictions before recipient restrictions.

HOWEVER, by default Postfix evaluates all of these at RCPT TO time.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Kai Schaetzl
Wietse Venema wrote on Wed, 7 Nov 2018 12:10:40 -0500 (EST):

> HOWEVER, by default Postfix evaluates all of these at RCPT TO time.

which means smtpd_delay_reject = yes is the default?

Am I correct in assuming that with "yes" it doesn't matter if I list the
client restrictions in smtpd_client_restrictions or in
smtpd_recipient_restrictions?
If so, does the order matter?
I mean it should matter in general, but if I mix different stages like
shown in my earlier mail like the following, is it still getting evaluated
in this order or getting reordered? See below for an exception I saw.

smtpd_recipient_restrictions =
    reject_non_fqdn_sender,
    reject_non_fqdn_recipient,
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination,
    reject_unknown_sender_domain,
    reject_unknown_recipient_domain,
    reject_unlisted_recipient,
    check_recipient_access hash:/etc/mail/allow_recipients,
    check_sender_access hash:/etc/mail/allow_senders,
    check_client_access hash:/etc/mail/allow_clients,
    check_client_access hash:/etc/mail/access,
    check_sender_access hash:/etc/mail/access,
    and some more.
   
I'm asking because I've seen rejections by sender earlier, although  
client_access should have hit first. An example:

Nov  7 14:15:24 b04 postfix/smtpd[6584]: NOQUEUE: reject: RCPT from
mx17.a.outbound.createsend.com[203.55.21.17]: 554 5.7.1
<[hidden email]>: Sender address rejected: Access denied;
from=<[hidden email]> to=<[hidden email]> proto=ESMTP
helo=<mx17.a.outbound.createsend.com>

with this in /etc/mail/access
createsend.com  REJECT
cmail20.com REJECT
and the exact same order of maps as above.

Shouldn't the client restriction have kicked in here instead of sender?

Thanks,

Kai


Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Noel Jones-2
On 11/7/2018 12:40 PM, Kai Schaetzl wrote:
> Wietse Venema wrote on Wed, 7 Nov 2018 12:10:40 -0500 (EST):
>
>> HOWEVER, by default Postfix evaluates all of these at RCPT TO time.
>
> which means smtpd_delay_reject = yes is the default?

Yes, that's the default, and generally should not be changed.

>
> Am I correct in assuming that with "yes" it doesn't matter if I list the
> client restrictions in smtpd_client_restrictions or in
> smtpd_recipient_restrictions?
> If so, does the order matter?
> I mean it should matter in general, but if I mix different stages like
> shown in my earlier mail like the following, is it still getting evaluated
> in this order or getting reordered? See below for an exception I saw.

Postfix always evaluates the smtpd_*_restrictions statements in the
documented order; they are never reordered.  Always
client-helo-sender-recipient.  This evaluation is by default delayed
until the client sends the first recipient, but the order stays the
same.

Within each smtpd_*_restrictions section, the restrictions are
checked in the order YOU specify.


>
> smtpd_recipient_restrictions =
>     reject_non_fqdn_sender,
>     reject_non_fqdn_recipient,
>     permit_sasl_authenticated,
>     permit_mynetworks,
>     reject_unauth_destination,
>     reject_unknown_sender_domain,
>     reject_unknown_recipient_domain,
>     reject_unlisted_recipient,
>     check_recipient_access hash:/etc/mail/allow_recipients,
>     check_sender_access hash:/etc/mail/allow_senders,
>     check_client_access hash:/etc/mail/allow_clients,
>     check_client_access hash:/etc/mail/access,
>     check_sender_access hash:/etc/mail/access,
>     and some more.

This will evaluate in exactly the order you have listed above.  They
are never reordered.


>    
> I'm asking because I've seen rejections by sender earlier, although  
> client_access should have hit first. An example:

With the above list, check_sender_access comes first.  Postfix does
not reorder the list you have specified.

>
> Shouldn't the client restriction have kicked in here instead of sender?

No, they are executed in the order you specify.

>
> Thanks,
>
> Kai
>
>



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Kai Schaetzl
Noel Jones wrote on Wed, 7 Nov 2018 13:30:08 -0600:

> With the above list, check_sender_access comes first.  Postfix does
> not reorder the list you have specified.

Thanks for the answer. But, please look again.

/etc/mail/access:
createsend.com  REJECT
cmail20.com REJECT

The order is:
> >     check_sender_access hash:/etc/mail/allow_senders,
-> this file does not contain matching data!
> >     check_client_access hash:/etc/mail/allow_clients,
> >     check_client_access hash:/etc/mail/access,
-> this is first in order and contains matching data!
> >     check_sender_access hash:/etc/mail/access,
-> but it matches only in the next step!


Kai

--
Get your web at Conactive Internet Services: http://www.conactive.com



Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Matus UHLAR - fantomas
>Noel Jones wrote on Wed, 7 Nov 2018 13:30:08 -0600:
>
>> With the above list, check_sender_access comes first.  Postfix does
>> not reorder the list you have specified.

On 08.11.18 01:31, Kai Schaetzl wrote:
>Thanks for the answer. But, please look again.
>
>/etc/mail/access:
>createsend.com  REJECT
>cmail20.com REJECT

you should specify .createsend.com, because the connecting domain is
mx17.a.outbound.createsend.com, not createsend.com.  You apparently did not
set append_dot_mydomain=yes (don't!).

Nov  7 14:15:24 b04 postfix/smtpd[6584]: NOQUEUE: reject: RCPT from
mx17.a.outbound.createsend.com[203.55.21.17]: 554 5.7.1
<[hidden email]>: Sender address rejected: Access denied;
from=<[hidden email]> to=<[hidden email]> proto=ESMTP
helo=<mx17.a.outbound.createsend.com>

>The order is:
>> >     check_sender_access hash:/etc/mail/allow_senders,
>-> this file does not contain matching data!
>> >     check_client_access hash:/etc/mail/allow_clients,
>> >     check_client_access hash:/etc/mail/access,
>-> this is first in order and contains matching data!
>> >     check_sender_access hash:/etc/mail/access,
>-> but it matches only in the next step!



--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Matus UHLAR - fantomas
>On 08.11.18 01:31, Kai Schaetzl wrote:
>>Thanks for the answer. But, please look again.
>>
>>/etc/mail/access:
>>createsend.com  REJECT
>>cmail20.com REJECT

On 08.11.18 12:06, Matus UHLAR - fantomas wrote:
>you should specify .createsend.com, because the connecting domain is
>mx17.a.outbound.createsend.com, not createsend.com.  You apparently did not
>set append_dot_mydomain=yes (don't!).

sorry, short brain outage.
the problem lies in "parent_domain_matches_subdomains" which is (and should
be) empty in postfix and apparently even is in new postfix version.


>Nov  7 14:15:24 b04 postfix/smtpd[6584]: NOQUEUE: reject: RCPT from
>mx17.a.outbound.createsend.com[203.55.21.17]: 554 5.7.1
><[hidden email]>: Sender address rejected: Access denied;
>from=<[hidden email]> to=<[hidden email]> proto=ESMTP
>helo=<mx17.a.outbound.createsend.com>
>
>>The order is:
>>>>     check_sender_access hash:/etc/mail/allow_senders,
>>-> this file does not contain matching data!
>>>>     check_client_access hash:/etc/mail/allow_clients,
>>>>     check_client_access hash:/etc/mail/access,
>>-> this is first in order and contains matching data!
>>>>     check_sender_access hash:/etc/mail/access,
>>-> but it matches only in the next step!

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Emacs is a complicated operating system without good text editor.
Reply | Threaded
Open this post in threaded view
|

Re: smtpd_delay_reject with rspamd milter

Kai Schaetzl
Matus UHLAR - fantomas wrote on Thu, 8 Nov 2018 12:12:49 +0100:

> the problem lies in "parent_domain_matches_subdomains" which is (and should
> be) empty in postfix and apparently even is in new postfix version.

As I wrote earlier, it *is* set implicitely by backwards-compatibilty.

> First I thought I might be using wrong syntax (site vs. .site), but I
> scanned the postfix docs and found that the default compatibility setting
> for access_maps should allow "site" to be used for subdomain matching as
> well.

postconf -d|grep parent_domain_matches_subdomains
parent_domain_matches_subdomains =
debug_peer_list,fast_flush_domains,mynetworks,permit_mx_backup_networks,qmqpd
_authorized_clients,relay_domains,smtpd_access_maps

At the moment it seems to be working as intended. But I can't see a config
change vs. yesterday morning that could have made a change with this. The
only change I made is stop the delay (temporarily) and move the
check_client_access from recipient_restrictions to client_restrictions (and
not move them back after delaying the checks again).
I'll see if I can come up with more clues/reproduction next week. At the
moment I'm happy that it *seems* to be working, although the order is now not
what I wanted.


Kai