Delay re-sending message following onward rejection

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Delay re-sending message following onward rejection

Dominic Raferd
Is there a way to delay re-sending a message following an onward
rejection? I am getting occasional messages back from an onward server
(gmail) about a bad email; within a second we remove the bad email
from the queue and block the originator's ip. But sometimes the first
of these actions is too late and the mail has already been sent to a
secondary mx, which then generates a 2nd rejection message (and so
on).

My idea is to force a delay (2 seconds say) between the initial
failure and the re-sending of the same email (same queue-id) to the
secondary mx (or fallback relay) - in the intervening time the message
may be pulled from the queue. Following earlier advice from Wietse
here I am already using 'smtp_transport_rate_delay = 5s', but although
helpful in other respects this does not delay the re-sending in this
case.

If you wondered, such a message will not have originated from my side;
it is crud from outside addressed to one of our users and my server is
relaying it to one of our own gmail mailboxes. We have a raft of
preventative measures including dnsbl lists and amavis, but bad stuff
still gets through occasionally. Gmail's testing is better and for us
definitive - if Gmail won't accept an email it is unacceptable by
definition because we use Gmail mailboxes, so I just want rid of it.

I've given example log messages below and added my comments. Thanks
for any help.

Dominic

# we've received an incoming mail, processed it and sent it on to
gmail - but it is rejected (the code and text message varies)
2017-01-04 16:20:07 mymx1 postfix/smtp[14166]: F064F3EC59: host
gmail-smtp-in.l.google.com[74.125.195.27] said: 421-4.7.0
[164.132.228.222      15] Our system has detected that this message is
421-4.7.0 suspicious due to the nature of the content and/or the links
within. 421-4.7.0 To best protect our users from spam, the message has
been blocked. 421-4.7.0 Please visit 421 4.7.0
https://support.google.com/mail/answer/188131 for more information.
p199si57699706wmd.1 - gsmtp (in reply to end of DATA command)

# relay-enforcer spots the log entry above, intervenes and uses
postsuper to remove the message from the queue...
2017-01-04 16:20:07 mymx1 postfix/postsuper[14358]: F064F3EC59: removed
2017-01-04 16:20:07 mymx1 postfix/postsuper[14358]: Deleted: 1 message
2017-01-04 16:20:07 mymx1 relay-enforcer-long: removed F064F3EC59 from
postfix queues (if any)

# ... and places a log message containing originator's ip - this will
be picked up by matching fail2ban jail
2017-01-04 16:20:07 mymx1 relay-enforcer-long: bannable F064F3EC59
6C6893E96A posta.skfuzlethaz.hu 178.48.9.21

# action by fail2ban jail (extract from syslog)
2017-01-04 16:20:08 mymx1 fail2ban.actions[1580]: NOTICE
[relay-enforcer-long] Ban 178.48.9.21

# but what's this? The same message had *already* (before the
deletion, presumably) been sent out to an alternate gmail mx - and is
rejected again
2017-01-04 16:20:08 mymx1 postfix/smtp[14166]: F064F3EC59: host
alt1.gmail-smtp-in.l.google.com[64.233.163.27] said: 421-4.7.0
[164.132.228.222      15] Our system has detected that this message is
421-4.7.0 suspicious due to the nature of the content and/or the links
within. 421-4.7.0 To best protect our users from spam, the message has
been blocked. 421-4.7.0 Please visit 421 4.7.0
https://support.google.com/mail/answer/188131 for more information.
a207si42877320lfd.65 - gsmtp (in reply to end of DATA command)
Reply | Threaded
Open this post in threaded view
|

Re: Delay re-sending message following onward rejection

Wietse Venema
Dominic Raferd:
> My idea is to force a delay (2 seconds say) between the initial
> failure and the re-sending of the same email (same queue-id) to the
> secondary mx (or fallback relay) - in the intervening time the message
> may be pulled from the queue. Following earlier advice from Wietse
> here I am already using 'smtp_transport_rate_delay = 5s', but although
> helpful in other respects this does not delay the re-sending in this
> case.

That won't work well because the 'next mx' is chosen in the SMTP
client (i.e. the message is not returned to the queue between
attempts to deliver the message to different MX hosts) and we don't
want to pause the SMTP client as that would prevent it from delivering
other email.

(the destination rate delay is per destination and must therefore
be implemented in the queue manager).

You can reduce the number of MX hosts to try to just 1, by setting
up an SMTP client for gmail etc. that has

    smtp -o smtp_mx_session_limit=1

With that, Postfix still tries multiple MX hosts until one responds,
and you will have $min_backoff-time or more to dig into the queue
and remove the offending message.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Delay re-sending message following onward rejection

Dominic Raferd
On 4 January 2017 at 16:52, Wietse Venema <[hidden email]> wrote:

> Dominic Raferd:
>> My idea is to force a delay (2 seconds say) between the initial
>> failure and the re-sending of the same email (same queue-id) to the
>> secondary mx (or fallback relay) - in the intervening time the message
>> may be pulled from the queue. Following earlier advice from Wietse
>> here I am already using 'smtp_transport_rate_delay = 5s', but although
>> helpful in other respects this does not delay the re-sending in this
>> case.
>
> That won't work well because the 'next mx' is chosen in the SMTP
> client (i.e. the message is not returned to the queue between
> attempts to deliver the message to different MX hosts) and we don't
> want to pause the SMTP client as that would prevent it from delivering
> other email.
>
> (the destination rate delay is per destination and must therefore
> be implemented in the queue manager).
>
> You can reduce the number of MX hosts to try to just 1, by setting
> up an SMTP client for gmail etc. that has
>
>     smtp -o smtp_mx_session_limit=1
>
> With that, Postfix still tries multiple MX hosts until one responds,
> and you will have $min_backoff-time or more to dig into the queue
> and remove the offending message.
>

Thanks Wietse that sounds like a good plan, how would I set up a
'special' smtp client for gmail servers with the reduced mx session
limit? I think I know how to create the special client in master.cf,
but how do I tell the delivery agent (queue?) to use this smtp client
for gmail destinations (and not for others)?
Reply | Threaded
Open this post in threaded view
|

Re: Delay re-sending message following onward rejection

Wietse Venema
Dominic Raferd:

> > You can reduce the number of MX hosts to try to just 1, by setting
> > up an SMTP client for gmail etc. that has
> >
> >     smtp -o smtp_mx_session_limit=1
> >
> > With that, Postfix still tries multiple MX hosts until one responds,
> > and you will have $min_backoff-time or more to dig into the queue
> > and remove the offending message.
>
> Thanks Wietse that sounds like a good plan, how would I set up a
> 'special' smtp client for gmail servers with the reduced mx session
> limit? I think I know how to create the special client in master.cf,
> but how do I tell the delivery agent (queue?) to use this smtp client
> for gmail destinations (and not for others)?

/etc/postfix/main.cf:
    transport_maps = hash:/etc/postfix/transport

/etc/postfix/transport:
    gmail.com:  one-mx-session
    ...ditto for other domains...

/etc/postfix/master.cf:
    one-mx-session    unix  -      -      -      -      -      smtp
        -o smtp_mx_session_limit=1

And leave smtp_mx_address_limit alone (don't set it to 1, otherwise
mail will back up in the queue for no good reason).

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Delay re-sending message following onward rejection

Dominic Raferd
On 4 January 2017 at 18:34, Wietse Venema <[hidden email]> wrote:

> Dominic Raferd:
>>
>> Thanks Wietse that sounds like a good plan, how would I set up a
>> 'special' smtp client for gmail servers with the reduced mx session
>> limit?
>
> /etc/postfix/main.cf:
>     transport_maps = hash:/etc/postfix/transport
>
> /etc/postfix/transport:
>     gmail.com:  one-mx-session
>     ...ditto for other domains...
>
> /etc/postfix/master.cf:
>     one-mx-session    unix  -      -      -      -      -      smtp
>         -o smtp_mx_session_limit=1
>
> And leave smtp_mx_address_limit alone (don't set it to 1, otherwise
> mail will back up in the queue for no good reason).
>

Brilliant, thank you Wietse. It didn't like the colon in
/etc/postfix/transport so I took it out, otherwise it seems to be
working. I will monitor over the coming days (I am only getting these
responses from gmail a couple of times per day on average) and check
that it is now pulling the offending mails before they get re-sent.

Have others previously encountered problems like this? Maybe I am
unusual in using gmail for our user mailboxes + postfix for relaying
(so we can use our own domain names, dmarc etc while using gmail front
ends)?

Dominic