SPF + outside backup MX relay = redelivery failures: Help requested

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino

I am running Postfix with opendkim, rspamd, pypolicyd-spf, and DMARC.
This is working fine for mail delivered directly to my domain.  However,
if my net connection goes down and mail gets queued by my backup MX at
another domain (which I do not control), then when my connection comes
back up and the MX relay attempts to redeliver all the queued mail,
delivery fails due to SPF failures like this one, because the sender's
domain has not authorized my mail relay to send mail on its behalf.  Of
course, I didn't *find out* about this problem until our uplink went
down for two and a half days.


Remote host said: 550 5.7.23 <[hidden email]>: Recipient address
rejected: Message rejected due to: SPF fail - not authorized. Please see
+http://www.openspf.net/Why?s=mfrom;id=ktk@...;ip=96.53.88.246;r=<UNKNOWN>


The failure is of course (in hindsight) because *our* backup MX relay is
not SPF-authorized by the *sender's* domain to send mail on behalf of
that domain, therefore it fails *my* SPF check and is rejected.  I've
added the MX relay to pypolicyd's HELO whitelist, but so far this does
not seem to be solving the problem.

Can anyone advise how I should best fix this?



--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Scott Kitterman-4
On Saturday, July 21, 2018 02:30:36 PM Phil Stracchino wrote:

> I am running Postfix with opendkim, rspamd, pypolicyd-spf, and DMARC.
> This is working fine for mail delivered directly to my domain.  However,
> if my net connection goes down and mail gets queued by my backup MX at
> another domain (which I do not control), then when my connection comes
> back up and the MX relay attempts to redeliver all the queued mail,
> delivery fails due to SPF failures like this one, because the sender's
> domain has not authorized my mail relay to send mail on its behalf.  Of
> course, I didn't *find out* about this problem until our uplink went
> down for two and a half days.
>
>
> Remote host said: 550 5.7.23 <[hidden email]>: Recipient address
> rejected: Message rejected due to: SPF fail - not authorized. Please see
> +http://www.openspf.net/Why?s=mfrom;id=ktk@...;ip=96.53.88.246;
> r=<UNKNOWN>
>
>
> The failure is of course (in hindsight) because *our* backup MX relay is
> not SPF-authorized by the *sender's* domain to send mail on behalf of
> that domain, therefore it fails *my* SPF check and is rejected.  I've
> added the MX relay to pypolicyd's HELO whitelist, but so far this does
> not seem to be solving the problem.
>
> Can anyone advise how I should best fix this?

Only check SPF at the external border of your email architecture.  The relay
from your backup MX is an internal relay.  SPF checks from that host should be
skipped.  If you look at the documentation provided with pypolicyd-spf,
particularly man (5) policyd-spf.conf, you'll see there are multiple options
available for doing this.  As an example (all see
https://git.launchpad.net/~kitterman/pypolicyd-spf/tree/policyd-spf.conf.commented ) this might work given the data you provided:

HELO_Whitelist = fritter.limelight.ca


Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino
On 07/21/18 21:25, Scott Kitterman wrote:
> Only check SPF at the external border of your email architecture.  The relay
> from your backup MX is an internal relay.  SPF checks from that host should be
> skipped.  If you look at the documentation provided with pypolicyd-spf,
> particularly man (5) policyd-spf.conf, you'll see there are multiple options
> available for doing this.  As an example (all see
> https://git.launchpad.net/~kitterman/pypolicyd-spf/tree/policyd-spf.conf.commented ) this might work given the data you provided:
>
> HELO_Whitelist = fritter.limelight.ca

Have now retested that method and it still does not work.  I was sure
that ought to.  There is clearly something here that I am failing to
understand.

What other means are there by which I can tell Postfix that I trust my
MX relay to relay mail to me?  I have to be missing something here.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Dominic Raferd


On Mon, 23 Jul 2018 at 15:04, Phil Stracchino <[hidden email]> wrote:
On 07/21/18 21:25, Scott Kitterman wrote:
> Only check SPF at the external border of your email architecture.  The relay
> from your backup MX is an internal relay.  SPF checks from that host should be
> skipped.  If you look at the documentation provided with pypolicyd-spf,
> particularly man (5) policyd-spf.conf, you'll see there are multiple options
> available for doing this.  As an example (all see
> https://git.launchpad.net/~kitterman/pypolicyd-spf/tree/policyd-spf.conf.commented ) this might work given the data you provided:
>
> HELO_Whitelist = fritter.limelight.ca

Have now retested that method and it still does not work.  I was sure
that ought to.  There is clearly something here that I am failing to
understand.

What other means are there by which I can tell Postfix that I trust my
MX relay to relay mail to me?  I have to be missing something here.

Off-topic but I am curious about your blocking policy based purely on SPF. I see a fair few SPF fails on incoming mails from genuine senders.​
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino
On 07/24/18 06:23, Dominic Raferd wrote:
> Off-topic but I am curious about your blocking policy based purely on
> SPF. I see a fair few SPF fails on incoming mails from genuine senders.​


I see very few EXCEPT when it's coming through my secondary MX.  As a
general rule, if a domain *has* an SPF record, it's good.  (Though there
are two sites that I have to whitelist because they have known SPF
problems,)  On the other hand, I get quite a lot of spam forged as
supposedly from known-good domains.  Since I don't know (yet) a way to
have the SPF check anything but on/off, I'm stuck with choosing on or
off, and there are FAR fewer breakages from having it on than false
positives from having it off (and by now I've pretty much found them all
and whitelisted them).


....EXCEPT when mail is being queued through my secondary MX because I'm
offline.  Then it's a problem, which I'm now trying to figure out how to
fix.



--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Viktor Dukhovni


> On Jul 24, 2018, at 8:36 AM, Phil Stracchino <[hidden email]> wrote:
>
> ....EXCEPT when mail is being queued through my secondary MX because I'm
> offline.  Then it's a problem, which I'm now trying to figure out how to
> fix.

You MUST NOT filter inbound traffic via your secondary MX.
If that MX host is too liberal in what it accepts, find another,
or live without a secondary MX.

This means that the secondary MX IP addresses must be included
in a whitelist that short-circuits everything but relay checks
(there's usually no reason to allow the secondary MX to send
outbound email through your system).

I am puzzled why this requires "figuring out"...

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino
On 07/24/18 08:42, Viktor Dukhovni wrote:
>> On Jul 24, 2018, at 8:36 AM, Phil Stracchino <[hidden email]> wrote:
>>
>> ....EXCEPT when mail is being queued through my secondary MX because I'm
>> offline.  Then it's a problem, which I'm now trying to figure out how to
>> fix.
>
> You MUST NOT filter inbound traffic via your secondary MX.
> If that MX host is too liberal in what it accepts, find another,
> or live without a secondary MX.

I'm not TRYING to filter traffic from my secondary MX.  I'm just not
sure of the best way to set things up such that it does NOT get filtered.



--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino
On 07/24/18 09:59, Phil Stracchino wrote:

> On 07/24/18 08:42, Viktor Dukhovni wrote:
>>> On Jul 24, 2018, at 8:36 AM, Phil Stracchino <[hidden email]> wrote:
>>>
>>> ....EXCEPT when mail is being queued through my secondary MX because I'm
>>> offline.  Then it's a problem, which I'm now trying to figure out how to
>>> fix.
>>
>> You MUST NOT filter inbound traffic via your secondary MX.
>> If that MX host is too liberal in what it accepts, find another,
>> or live without a secondary MX.
>
> I'm not TRYING to filter traffic from my secondary MX.  I'm just not
> sure of the best way to set things up such that it does NOT get filtered.

Also, for clarification, all of this was working — i.e. mail queued
through my backup MX was correctly delivered — *UNTIL* I added SPF
filtering a year or two ago.  I only recently discovered that this
created a problem when the MX backup flushed queued mail.  The single
thing that I am specifically trying to figure out is how to set things
up to *NOT* SPF-filter mail coming in through my MX relay, without
turning off SPF checking altogether.  Theory says that adding the MX
relay to HELO_whitelist should have accomplished this, but it appears it
didn't.

Do I need a smarter SPF filter than pypolicyd-spf?
*IS* there a different SPF filter?
Or am I just missing something in my configuration that *should* be obvious?


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Viktor Dukhovni
In reply to this post by Phil Stracchino
On Tue, Jul 24, 2018 at 09:59:18AM -0400, Phil Stracchino wrote:

> I'm just not
> sure of the best way to set things up such that it does NOT get filtered.

Add the backup MX to an IP-based whitelist.  Check that whitelist
before all restrictions other than relay control.

        cidr = cidr:${config_directory}/

        smtpd_relay_restrictions =
            # Something along the lines of
            permit_mynetworks,
            permit_sasl_authenticated,
            reject_unauth_destination
        smtp_recipient_restrictions =
            check_client_access ${cidr}whitelist.cidr,
            # ...
        smtp_client_restrictions =
            # If non-empty, start with whitelist check
            # check_client_access ${cidr}whitelist.cidr,
            # ...
        smtp_helo_restrictions =
            # If non-empty, start with whitelist check
            # check_client_access ${cidr}whitelist.cidr,
            # ...
        smtp_sender_restrictions =
            # If non-empty, start with whitelist check
            # check_client_access ${cidr}whitelist.cidr,
            # ...

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

James B. Byrne
In reply to this post by Phil Stracchino

On Tue, July 24, 2018 10:27, Phil Stracchino wrote:

> On 07/24/18 09:59, Phil Stracchino wrote:
>> On 07/24/18 08:42, Viktor Dukhovni wrote:
>>>> On Jul 24, 2018, at 8:36 AM, Phil Stracchino
>>>> <[hidden email]> wrote:
>>>>
>>>> ....EXCEPT when mail is being queued through my secondary MX
>>>> because I'm
>>>> offline.  Then it's a problem, which I'm now trying to figure out
>>>> how to
>>>> fix.
>>>
>>> You MUST NOT filter inbound traffic via your secondary MX.
>>> If that MX host is too liberal in what it accepts, find another,
>>> or live without a secondary MX.
>>
>> I'm not TRYING to filter traffic from my secondary MX.  I'm just not
>> sure of the best way to set things up such that it does NOT get
>> filtered.
>
> Also, for clarification, all of this was working — i.e. mail queued
> through my backup MX was correctly delivered — *UNTIL* I added SPF
> filtering a year or two ago.  I only recently discovered that this
> created a problem when the MX backup flushed queued mail.  The single
> thing that I am specifically trying to figure out is how to set things
> up to *NOT* SPF-filter mail coming in through my MX relay, without
> turning off SPF checking altogether.  Theory says that adding the MX
> relay to HELO_whitelist should have accomplished this, but it appears
> it
> didn't.
>
> Do I need a smarter SPF filter than pypolicyd-spf?
> *IS* there a different SPF filter?
> Or am I just missing something in my configuration that *should* be
> obvious?
>
>
From: policyd-spf.conf

#  Whitelist: CIDR Notation list of IP addresses not to check SPF for.
#  Example (default is no whitelist):
#  Whitelist = 192.168.0.0/31,192.168.1.12

#  Domain_Whitelist: List of domains whose sending IPs
#  (defined by passing
#  their SPF check should be whitelisted from SPF.
#  Example (default is no domain whitelist):
#  Domain_Whitelist = pobox.com,trustedforwarder.org
Domain_Whitelist = bellnexxia.net,lcbo.com

# Domain_Whitelist_PTR: List of domains to whitelist against SPF
checks base
# on PTR match.
# Example (default is no PTR whitelist)
# Domain_Whitelist_PTR = yahoo.com

. . .

--
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino
In reply to this post by Viktor Dukhovni
On 07/24/18 10:36, Viktor Dukhovni wrote:

> On Tue, Jul 24, 2018 at 09:59:18AM -0400, Phil Stracchino wrote:
>
>> I'm just not
>> sure of the best way to set things up such that it does NOT get filtered.
>
> Add the backup MX to an IP-based whitelist.  Check that whitelist
> before all restrictions other than relay control.
>
> cidr = cidr:${config_directory}/
>
> smtpd_relay_restrictions =
>    # Something along the lines of
>    permit_mynetworks,
>    permit_sasl_authenticated,
>    reject_unauth_destination
> smtp_recipient_restrictions =
>    check_client_access ${cidr}whitelist.cidr,
>    # ...
> smtp_client_restrictions =
>    # If non-empty, start with whitelist check
>    # check_client_access ${cidr}whitelist.cidr,
>    # ...
> smtp_helo_restrictions =
>    # If non-empty, start with whitelist check
>    # check_client_access ${cidr}whitelist.cidr,
>    # ...
> smtp_sender_restrictions =
>    # If non-empty, start with whitelist check
>    # check_client_access ${cidr}whitelist.cidr,
>    # ...


OK, it took me a couple of readings to parse what exactly you were doing
with the variable there, but once I cleared that up, it seems to be
working correctly for normal direct delivery.  Now waiting for my
outside co-conspirator to do another forced-backup-MX test.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: SPF + outside backup MX relay = redelivery failures: Help requested

Phil Stracchino
On 07/24/18 14:00, Phil Stracchino wrote:

> OK, it took me a couple of readings to parse what exactly you were doing
> with the variable there, but once I cleared that up, it seems to be
> working correctly for normal direct delivery.  Now waiting for my
> outside co-conspirator to do another forced-backup-MX test.

And that did the trick, where whitelisting in pypolicyd-spf did not.
Thanks, Viktor.


--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958