Why does SPF fail sometimes?

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Why does SPF fail sometimes?

Christian Rößner
Hi,

sorry, if this question might be a little off-topic, but I really do not understand some DMARC reports that I receive in conjunction to this mailing list and maybe someone can help me in digging down the problem:

<?xml version="1.0" encoding="UTF-8" ?>
<feedback>
 <report_metadata>
  <org_name>*****.com</org_name>
  <email>noreply-dmarc@*****.com</email>
  <report_id>roessner-network-solutions.com:1418590509</report_id>
  <date_range>
   <begin>1418552542</begin>
   <end>1418552542</end>
  </date_range>
 </report_metadata>
 <policy_published>
  <domain>roessner-network-solutions.com</domain>
  <adkim>s</adkim>
  <aspf>r</aspf>
  <p>none</p>
  <sp>none</sp>
  <pct>100</pct>
 </policy_published>
 <record>
  <row>
   <source_ip>168.100.1.3</source_ip>
   <count>1</count>
   <policy_evaluated>
    <disposition>none</disposition>
    <dkim>pass</dkim>
    <spf>fail</spf>
   </policy_evaluated>
  </row>
  <identifiers>
   <header_from>roessner-network-solutions.com</header_from>
  </identifiers>
  <auth_results>
   <spf>
    <domain>postfix.org</domain>
    <result>unknown</result>
   </spf>
   <dkim>
    <domain>roessner-network-solutions.com</domain>
    <result>pass</result>
   </dkim>
  </auth_results>
 </record>
</feedback>

If I do understand this report right, DKIM passes, but SPF failed. If I look to my last mail, I sent this day, I see this in the headers:

DMARC-Filter: OpenDMARC Filter v1.3.0 mx.roessner-net.de 3k0hcj6S5RzGpN5
Authentication-Results: mx.roessner-net.de; dmarc=pass header.from=roessner-network-solutions.com
Authentication-Results: mx.roessner-net.de; spf=pass smtp.mailfrom=[hidden email]
DKIM-Filter: OpenDKIM Filter v2.9.2 mx.roessner-net.de 3k0hcj6S5RzGpN5
Authentication-Results: mx.roessner-net.de;
        dkim=pass (2048-bit key; secure) header.d=roessner-network-solutions.com header.i=@roessner-network-solutions.com header.b=G+zYCJvI;
        dkim-adsp=pass; dkim-atps=neutral

So why does my own MX say, everything is fine, with incoming mails from the list, but I get reports from other people that SPF failed?

Is this a misconfiguration somewhere remote or am I missing something?

Thanks a lot for helping me

Kind regards

Christian
--
Bachelor of Science Informatik
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com


signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Wietse Venema
Christian R??ner:
> sorry, if this question might be a little off-topic, but I really
> do not understand some DMARC reports that I receive in conjunction
> to this mailing list and maybe someone can help me in digging down
> the problem:

Perhaps a stupid question: can you exclude DNS lookup problems?
Packet loss happens, and for practical reasons verifiers cannot
retry indefinitely.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

lists@rhsoft.net
In reply to this post by Christian Rößner

Am 14.12.2014 um 23:48 schrieb Christian Rößner:

> If I do understand this report right, DKIM passes, but SPF failed. If I look to my last mail, I sent this day, I see this in the headers:
>
> DMARC-Filter: OpenDMARC Filter v1.3.0 mx.roessner-net.de 3k0hcj6S5RzGpN5
> Authentication-Results: mx.roessner-net.de; dmarc=pass header.from=roessner-network-solutions.com
> Authentication-Results: mx.roessner-net.de; spf=pass smtp.mailfrom=[hidden email]
> DKIM-Filter: OpenDKIM Filter v2.9.2 mx.roessner-net.de 3k0hcj6S5RzGpN5
> Authentication-Results: mx.roessner-net.de;
> dkim=pass (2048-bit key; secure) header.d=roessner-network-solutions.com header.i=@roessner-network-solutions.com header.b=G+zYCJvI;
> dkim-adsp=pass; dkim-atps=neutral
>
> So why does my own MX say, everything is fine, with incoming mails from the list, but I get reports from other people that SPF failed?
>
> Is this a misconfiguration somewhere remote or am I missing something?

i guess that fools apply the SPF test to the From-Header instead to the
envelope, frankly Barracuda Networks does the same for
Spoofing-Protection because "customers complained"

without knowing details i would suggest the problem is on the receivers
side, but however: try to make your SPF as tiny as possible and avoid
names and references

after switch to plain "ipv4:ip/network-range" i never faced any SPF
problem while before it happened again and again by listing servers with
names and things like a/mx/include
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Christian Rößner
In reply to this post by Wietse Venema

> Am 14.12.2014 um 23:53 schrieb Wietse Venema <[hidden email]>:
>
> Christian R??ner:
>> sorry, if this question might be a little off-topic, but I really
>> do not understand some DMARC reports that I receive in conjunction
>> to this mailing list and maybe someone can help me in digging down
>> the problem:
>
> Perhaps a stupid question: can you exclude DNS lookup problems?
> Packet loss happens, and for practical reasons verifiers cannot
> retry indefinitely.

There is a local unbound resolver on the MX and there are to BIND9 resolvers.

We do have packet loss very seldom. It’s a fibre line.

So I would not expect these problems.

And I saw that I really only get very, very few reports, so it seems most MXes out there (who do DMARC) seem to be happy.

Maybe really not a problem on my side…

Thanks

Christian
--
Bachelor of Science Informatik
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com

Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Christian Rößner
In reply to this post by lists@rhsoft.net

> Am 14.12.2014 um 23:56 schrieb [hidden email]:
>
> i guess that fools apply the SPF test to the From-Header instead to the envelope, frankly Barracuda Networks does the same for Spoofing-Protection because "customers complained"
>
> without knowing details i would suggest the problem is on the receivers side, but however: try to make your SPF as tiny as possible and avoid names and references

Thanks. That was what I thought. People using the header-from field. But I couldn’t believe that. But now that you gave me this feedback, I think this might be the reason.

And concerning the SPF record. I have set the mx and two a: fields. Exactly those three servers that are really permitted to send mail.

Thanks you very much

Christian
--
Bachelor of Science Informatik
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Wietse Venema
In reply to this post by Christian Rößner
Christian R??ner:

>
> > Am 14.12.2014 um 23:53 schrieb Wietse Venema <[hidden email]>:
> >
> > Christian R??ner:
> >> sorry, if this question might be a little off-topic, but I really
> >> do not understand some DMARC reports that I receive in conjunction
> >> to this mailing list and maybe someone can help me in digging down
> >> the problem:
> >
> > Perhaps a stupid question: can you exclude DNS lookup problems?
> > Packet loss happens, and for practical reasons verifiers cannot
> > retry indefinitely.
>
> There is a local unbound resolver on the MX and there are to BIND9 resolvers.
>
> We do have packet loss very seldom. It's a fibre line.
>
> So I would not expect these problems.

Not between your own systems.

What about the rest of the Internet?  I doubt that your local SLA
covers communication with remote destinations.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Christian Rößner

> Am 15.12.2014 um 00:36 schrieb Wietse Venema <[hidden email]>:
>
> Christian R??ner:
>>
>>> Am 14.12.2014 um 23:53 schrieb Wietse Venema <[hidden email]>:
>>>
>>> Christian R??ner:
>>>> sorry, if this question might be a little off-topic, but I really
>>>> do not understand some DMARC reports that I receive in conjunction
>>>> to this mailing list and maybe someone can help me in digging down
>>>> the problem:
>>>
>>> Perhaps a stupid question: can you exclude DNS lookup problems?
>>> Packet loss happens, and for practical reasons verifiers cannot
>>> retry indefinitely.
>>
>> There is a local unbound resolver on the MX and there are to BIND9 resolvers.
>>
>> We do have packet loss very seldom. It's a fibre line.
>>
>> So I would not expect these problems.
>
> Not between your own systems.
>
> What about the rest of the Internet?  I doubt that your local SLA
> covers communication with remote destinations.
I found the answer and I fear there is no chance to solve this:

https://datatracker.ietf.org/doc/draft-kucherawy-dmarc-base/?include_text=1

It’s the problem with DMARC. Nearly the same problem that I posted some days ago. It’s all about the RFC5322 from address. DMARC uses this information and checks SPF and DKIM against this header field. So it is not enough to have SPF passed; it also MUST have been sent from a SPF legitimated system.

As Postfix mailing list does not send with its mailing list address (which would solve this problem) as RFC5322 from address, the SPF test will always fail.

So most lists are not DMARC-ready.

See under 1. Introduction

2.  Receivers compare the RFC5322 From: address in the mail to the
       SPF and DKIM results, if present, and the DMARC policy in DNS.

3.1.1.  Authentication Mechanisms


   The following mechanisms for determining Authenticated Identifiers
   are supported in this version of DMARC:

   o  [DKIM], which provides a domain-level identifier in the content of
      the "d=" tag of a validated DKIM-Signature header field.

   o  [SPF], which authenticates the domain found in an [SMTP] MAIL
      command when it is the authorized domain.

An authorized domain is defined like:

3.  Terminology and Definitions

Author Domain:  The domain name of the apparent author, as extracted
      from the RFC5322.From field.

So I only can ask mailing list administrators to fix there lists, but I fear only a few will do this.

If I look at dmarc.org at the bottom, there are many organizations right now that use DMARC, so this topic will probably come up more and more.

Thanks for helping

Christian
--
Bachelor of Science Informatik
Erlenwiese 14, 36304 Alsfeld
T: +49 6631 78823400, F: +49 6631 78823409, M: +49 171 9905345
USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com


signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Wietse Venema
Christian R??ner:
> I found the answer and I fear there is no chance to solve this:
>
> https://datatracker.ietf.org/doc/draft-kucherawy-dmarc-base/?include_text=1
>
> It's the problem with DMARC. Nearly the same problem that I posted
> some days ago. It's all about the RFC5322 from address. DMARC uses
> this information and checks SPF and DKIM against this header field.
> So it is not enough to have SPF passed; it also MUST have been
> sent from a SPF legitimated system.

DMARC "verifies" the From: header against SPF, DKIM or both, but
only a poorly-informed person would require that the From: address
*always* verifies with SPF.

It would be unreasonable to expect that mailing list managers replace
the From: address of mailing list postings to match the list server's
IP addresses.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Richard Damon
On 12/14/14, 7:36 PM, Wietse Venema wrote:

> Christian R??ner:
>> I found the answer and I fear there is no chance to solve this:
>>
>> https://datatracker.ietf.org/doc/draft-kucherawy-dmarc-base/?include_text=1
>>
>> It's the problem with DMARC. Nearly the same problem that I posted
>> some days ago. It's all about the RFC5322 from address. DMARC uses
>> this information and checks SPF and DKIM against this header field.
>> So it is not enough to have SPF passed; it also MUST have been
>> sent from a SPF legitimated system.
> DMARC "verifies" the From: header against SPF, DKIM or both, but
> only a poorly-informed person would require that the From: address
> *always* verifies with SPF.
>
> It would be unreasonable to expect that mailing list managers replace
> the From: address of mailing list postings to match the list server's
> IP addresses.
>
> Wietse
DMARC says that if a domain requests DMARC protection then any message
that has a RFC5322 domain pointing to it, must be verifiable as coming
from that domain, thus such an address can NOT use a 3rd party (like a
mailing list manager) to deliver a message for it without adding it to
SPF or giving it the DKIM signing keys.

Since DMARC was intended to protect "high value" emails, like from
something like a bank, this wouldn't normally be a problem. Effectively
emails from a DMARC protected domain shouldn't be used for non-official
communication, and any 3rd party service is presumably trusted so you
can make the needed arrangements. The problem is that YAHOO and AOL
have, via their DMARC settings, declared emails from their domain to be
this type of high value, and in effect that their users are not to use
3rd party distribution methods (but haven't told their users this).

Other mailing list systems have adopted some work arounds for this
problem, a common one is to "munge" the From: line to be the list
address (and setting Reply-To: to the poster), or wrapping the message
in a wrapper that is from the list, and the message to be distributed is
included as an attachment. (And some will just reject any message from a
domain that uses DMARC protection)

The problem isn't really with DMARC, it is doing what it was intended to
do, the problem is the services misusing DMARC. It sounds like if
pushed, they will even admit that they are abusing it, but feel they
need to due to a lot of messages being forged as from them.

Yes, it is arguably a violation of the RFC's to rewrite the From:
address of a message going through a mailing list manager, but it is one
of the ways to handle the misuse of DMARC that has happened. It comes
down to a question of what are you willing to do to make things "work"
and who are you willing to make bear the brunt of problems.

--
Richard Damon

Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

James B. Byrne

On Sun, December 14, 2014 20:05, Richard Damon wrote:

> DMARC says that if a domain requests DMARC protection then any message
> that has a RFC5322 domain pointing to it, must be verifiable as coming
> from that domain, thus such an address can NOT use a 3rd party (like a
> mailing list manager) to deliver a message for it without adding it to
> SPF or giving it the DKIM signing keys.
>
> Since DMARC was intended to protect "high value" emails, like from
> something like a bank, this wouldn't normally be a problem. Effectively
> emails from a DMARC protected domain shouldn't be used for non-official
> communication, and any 3rd party service is presumably trusted so you
> can make the needed arrangements. The problem is that YAHOO and AOL
> have, via their DMARC settings, declared emails from their domain to be
> this type of high value, and in effect that their users are not to use
> 3rd party distribution methods (but haven't told their users this).
>
> Other mailing list systems have adopted some work arounds for this
> problem, a common one is to "munge" the From: line to be the list
> address (and setting Reply-To: to the poster), or wrapping the message
> in a wrapper that is from the list, and the message to be distributed is
> included as an attachment. (And some will just reject any message from a
> domain that uses DMARC protection)
>
> The problem isn't really with DMARC, it is doing what it was intended to
> do, the problem is the services misusing DMARC. It sounds like if
> pushed, they will even admit that they are abusing it, but feel they
> need to due to a lot of messages being forged as from them.
>
> Yes, it is arguably a violation of the RFC's to rewrite the From:
> address of a message going through a mailing list manager, but it is one
> of the ways to handle the misuse of DMARC that has happened. It comes
> down to a question of what are you willing to do to make things "work"
> and who are you willing to make bear the brunt of problems.
>

DMARC was forced upon the IETF by the big mail hosting companies.  The reason
that the FROM header is checked instead of the SENDER is because the FROM is
what virtually all MUA's display to the end user; and that is what the mail
hosting companies want verified.  Banks and other 'high value' email sources
are red-herrings.  They could care less.  Nothing of any import is ever sent
by email from a bank; Or by anyone else that has any sense (PGP/GPG/SMIME
users excepted, maybe).

DMARC is doing exactly what was expected of it by the people pushing-for /
forcing its adoption.  It is also breaking every mailing list manager exactly
as was predicted.  Mailman MLM has since had a mod made to rewrite the from
and set a few other switches to handle SPF.

As for it being a violation of RFCs to rewrite the FROM header one has to
consider what the source really is for any message coming through a mailing
list forwarder.  If all the messages sent through a MLM over some period are
digested and sent as one message then what should the from id be?  If the from
id for all the messages sent through a mailing list as a single digest is the
MLM itself then why should the same messages sent through the same list
individually be treated differently?

The role of an MLM is really no different than if you or I forwarded a message
we received on to a third party.  Who is the FROM id in that case?  Arguably,
most MLMs have been doing it wrong since the beginning and DMARC is just
highlighting the logical inconsistencies and contradictions in prevalent MLM
practice.

--
***          E-Mail is NOT a SECURE channel          ***
James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Richard Damon
On 12/14/14, 10:10 PM, James B. Byrne wrote:
>
 > On Sun, December 14, 2014 20:05, Richard Damon wrote:
 >> DMARC says that if a domain requests DMARC protection then any
 >> message that has a RFC5322 domain pointing to it, must be
 >> verifiable as coming from that domain, thus such an address can
 >> NOT use a 3rd party (like a mailing list manager) to deliver a
 >> message for it without adding it to SPF or giving it the DKIM
 >> signing keys.
 >>
 >> Since DMARC was intended to protect "high value" emails, like from
 >>  something like a bank, this wouldn't normally be a problem.
 >> Effectively emails from a DMARC protected domain shouldn't be used
 >> for non-official communication, and any 3rd party service is
 >> presumably trusted so you can make the needed arrangements. The
 >> problem is that YAHOO and AOL have, via their DMARC settings,
 >> declared emails from their domain to be this type of high value,
 >> and in effect that their users are not to use 3rd party
 >> distribution methods (but haven't told their users this).
 >>
 >> Other mailing list systems have adopted some work arounds for this
 >>  problem, a common one is to "munge" the From: line to be the list
 >>  address (and setting Reply-To: to the poster), or wrapping the
 >> message in a wrapper that is from the list, and the message to be
 >> distributed is included as an attachment. (And some will just
 >> reject any message from a domain that uses DMARC protection)
 >>
 >> The problem isn't really with DMARC, it is doing what it was
 >> intended to do, the problem is the services misusing DMARC. It
 >> sounds like if pushed, they will even admit that they are abusing
 >> it, but feel they need to due to a lot of messages being forged as
 >> from them.
 >>
 >> Yes, it is arguably a violation of the RFC's to rewrite the From:
 >> address of a message going through a mailing list manager, but it
 >> is one of the ways to handle the misuse of DMARC that has
 >> happened. It comes down to a question of what are you willing to do
 >> to make things "work" and who are you willing to make bear the
 >> brunt of problems.
 >>
 >
 > DMARC was forced upon the IETF by the big mail hosting companies. The
 > reason that the FROM header is checked instead of the SENDER is
 > because the FROM is what virtually all MUA's display to the end
 > user; and that is what the mail hosting companies want verified.
 > Banks and other 'high value' email sources are red-herrings. They
 > could care less.  Nothing of any import is ever sent by email from a
 > bank; Or by anyone else that has any sense (PGP/GPG/SMIME users
 > excepted, maybe).

I regularly get important messages from Financial Institutions.
Yes, they will typically ask me to log into their web site for confirmation
of the message or to send "sensitive" information, but they do
send notices by email that they hope I will see.

In fact, it is only because they DO send me email that a scammer has
much chance of succeeding by sending a fake message, hoping that I
will click on a link taking me to the wrong place.

>
 > DMARC is doing exactly what was expected of it by the people
 > pushing-for / forcing its adoption.  It is also breaking every
 > mailing list manager exactly as was predicted.  Mailman MLM has
 > since had a mod made to rewrite the from and set a few other switches
 > to handle SPF.
 >
 > As for it being a violation of RFCs to rewrite the FROM header one
 > has to consider what the source really is for any message coming
 > through a mailing list forwarder.  If all the messages sent through
 > a MLM over some period are digested and sent as one message then
 > what should the from id be?  If the from id for all the messages
 > sent through a mailing list as a single digest is the MLM itself then
 > why should the same messages sent through the same list individually
 > be treated differently?

If you read the standard, is says:

   The originator fields indicate the mailbox(es) of the source of the
    message.  The "From:" field specifies the author(s) of the message,
    that is, the mailbox(es) of the person(s) or system(s) responsible
    for the writing of the message.  The "Sender:" field specifies the
    mailbox of the agent responsible for the actual transmission of the
    message.  For example, if a secretary were to send a message for
    another person, the mailbox of the secretary would appear in the
    "Sender:" field and the mailbox of the actual author would appear in
    the "From:" field.

The *AUTHOR* of the message is the person who originally wrote it, not
the mailing list.
I digest is something different, the digest, as a whole, WAS created by
the list,
just like if a person collects a number of pieces written by other
people, that
person IS the author of the collection, and the individuals who wrote
the pieces are the
authors of the pieces, but not the whole.

The mailing list software is NOT the author of the individual messages, but
much more like the secretary mentioned in the RFC.

(I did say arguably because some people differ in this intent)

>
 >
 > The role of an MLM is really no different than if you or I forwarded
 > a message we received on to a third party.  Who is the FROM id in
 > that case?  Arguably, most MLMs have been doing it wrong since the
 > beginning and DMARC is just highlighting the logical inconsistencies
 > and contradictions in prevalent MLM practice.
 >

The big difference is that the MLM is an AUTOMATED PROCESS (not
significantly
different from postfix). If I manually forward a message, that is not
(by definition)
an automated process, and generally the MUA will build a new message
containing
the original message (and possibly my notes about the message), so this
is reasonable
to change authorship of the forwarding to be the forwarder. As a point
of reference,
when you setup and automated forward rule for a mailbox to some other
mailbox,
THAT forwarding does NOT normally change the From: line.


--
Richard Damon

Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Benny Pedersen-2
In reply to this post by Christian Rößner
On 15. dec. 2014 00.21.30 Christian Rößner
<[hidden email]> wrote:

> Thanks. That was what I thought. People using the header-from field. But I
> couldn’t believe that. But now that you gave me this feedback, I think this
> might be the reason.

opendmarc links to libspf2, with in some versions checks sender-id, if
sender-id was supported in opendmarc it would properly be stable

I see same problem here with bind9 9.10.x gentoo, so imho not just a
unbound issue

> And concerning the SPF record. I have set the mx and two a: fields. Exactly
> those three servers that are really permitted to send mail.

Should be fine, but its more safe with strict ip

> Thanks you very much

Thanks for the milter, is there a gentoo ebuild for it ?, note to you that
/usr/local prefix is for make install builds, while its /usr prefix for ebuilds
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Benny Pedersen-2
In reply to this post by Christian Rößner
On 15. dec. 2014 01.19.02 Christian Rößner
<[hidden email]> wrote:

> https://datatracker.ietf.org/doc/draft-kucherawy-dmarc-base/?include_text=1

> 2.  Receivers compare the RFC5322 From: address in the mail to the
>        SPF and DKIM results, if present, and the DMARC policy in DNS.

Hopefully opendmarc will have this resolved in 1.3.1, dkim use from header,
but spf must not doit

And we both see dmarc=pass here on this list

Tip: setup pypolicyd-spf to report AR header and disable libspf2 in
opendmarc, then it should work, but only add the header on spf pass, so
this header is not there if spf failed someway
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

A. Schulze
In reply to this post by Wietse Venema

wietse:

> DMARC "verifies" the From: header against SPF, DKIM or both, but
> only a poorly-informed person would require that the From: address
> *always* verifies with SPF.

for that reason it's more important the existing DKIM signature
is still valid when the mlm redistribute the message to all subscribers.

Unfortunately there are some rare situations a message must be modified.
Well known example: Mailman sometimes change the message encoding.
Not so often known: The mlm MTA submit to a subcriber MTA not capable  
8BITMIME.

In the last case a message is also modified and will not pass DMARC test
even if the MLM host is known to "usually" not modifying messages.


And exactly to debug such situations I with postfix could log
"here is a message with 8 bit content and the remote MTA does not
   announce 8BITMIME. I have to recode the message and that will invalidate
   potential existing DKIM signatures"

The other option is to convert every message to 7bit before signing.
Doesn't sound as a strategy for the future ...

> It would be unreasonable to expect that mailing list managers replace
> the From: address of mailing list postings to match the list server's
> IP addresses.

Ehm,
ironically that's exactly the solution preferred on the dmarc-discuss ml :-/

Andreas


Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

lst_hoe02
In reply to this post by James B. Byrne

Zitat von "James B. Byrne" <[hidden email]>:

> On Sun, December 14, 2014 20:05, Richard Damon wrote:
>
>> DMARC says that if a domain requests DMARC protection then any message
>> that has a RFC5322 domain pointing to it, must be verifiable as coming
>> from that domain, thus such an address can NOT use a 3rd party (like a
>> mailing list manager) to deliver a message for it without adding it to
>> SPF or giving it the DKIM signing keys.
>>
>> Since DMARC was intended to protect "high value" emails, like from
>> something like a bank, this wouldn't normally be a problem. Effectively
>> emails from a DMARC protected domain shouldn't be used for non-official
>> communication, and any 3rd party service is presumably trusted so you
>> can make the needed arrangements. The problem is that YAHOO and AOL
>> have, via their DMARC settings, declared emails from their domain to be
>> this type of high value, and in effect that their users are not to use
>> 3rd party distribution methods (but haven't told their users this).
>>
>> Other mailing list systems have adopted some work arounds for this
>> problem, a common one is to "munge" the From: line to be the list
>> address (and setting Reply-To: to the poster), or wrapping the message
>> in a wrapper that is from the list, and the message to be distributed is
>> included as an attachment. (And some will just reject any message from a
>> domain that uses DMARC protection)
>>
>> The problem isn't really with DMARC, it is doing what it was intended to
>> do, the problem is the services misusing DMARC. It sounds like if
>> pushed, they will even admit that they are abusing it, but feel they
>> need to due to a lot of messages being forged as from them.
>>
>> Yes, it is arguably a violation of the RFC's to rewrite the From:
>> address of a message going through a mailing list manager, but it is one
>> of the ways to handle the misuse of DMARC that has happened. It comes
>> down to a question of what are you willing to do to make things "work"
>> and who are you willing to make bear the brunt of problems.
>>
>
> DMARC was forced upon the IETF by the big mail hosting companies.  The reason
> that the FROM header is checked instead of the SENDER is because the FROM is
> what virtually all MUA's display to the end user; and that is what the mail
> hosting companies want verified.  Banks and other 'high value' email sources
> are red-herrings.  They could care less.  Nothing of any import is ever sent
> by email from a bank; Or by anyone else that has any sense (PGP/GPG/SMIME
> users excepted, maybe).
>
> DMARC is doing exactly what was expected of it by the people pushing-for /
> forcing its adoption.  It is also breaking every mailing list manager exactly
> as was predicted.  Mailman MLM has since had a mod made to rewrite the from
> and set a few other switches to handle SPF.
>
So DMARC is just another piece of junk invented to break e-mail or the  
rest of it...

I don't get it why people still resist in using working solutions for  
"high value" e-mail like S/MIME and PGP and instead try to use  
broken-by-design "solutions" like SPF/DMARC whatever.

No average user understand any of this DMARC/SPF whatever cruft so  
they one day simply will use Google or Hotmail because most thers will  
also do in the hope the big ones will do something right about it.

Regards

Andreas




smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Darren Pilgrim
In reply to this post by Richard Damon
On 12/14/2014 5:05 PM, Richard Damon wrote:
> Other mailing list systems have adopted some work arounds for this
> problem, a common one is to "munge" the From: line to be the list
> address (and setting Reply-To: to the poster), or wrapping the message
> in a wrapper that is from the list, and the message to be distributed is
> included as an attachment. (And some will just reject any message from a
> domain that uses DMARC protection)

It's extra fun when they do so to an email with a DKIN signature
covering the From: header.

Yes, I've seen it in the wild.  No, I couldn't find a clue hammer large
enough to convince the ML admin of their idiocy.
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Wietse Venema
In reply to this post by A. Schulze
A. Schulze:
> > It would be unreasonable to expect that mailing list managers replace
> > the From: address of mailing list postings to match the list server's
> > IP addresses.
>
> Ehm,
> ironically that's exactly the solution preferred on the dmarc-discuss ml :-/

Folks who set up such policies should not join mailing lists.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

lists@rhsoft.net
In reply to this post by Benny Pedersen-2

Am 15.12.2014 um 06:15 schrieb Benny Pedersen:

> On 15. dec. 2014 00.21.30 Christian Rößner
> <[hidden email]> wrote:
>
>> Thanks. That was what I thought. People using the header-from field.
>> But I couldn’t believe that. But now that you gave me this feedback, I
>> think this might be the reason.
>
> opendmarc links to libspf2, with in some versions checks sender-id, if
> sender-id was supported in opendmarc it would properly be stable
>
> I see same problem here with bind9 9.10.x gentoo, so imho not just a
> unbound issue

why should this be a unbound issue?

SPF is nothing more than an ordinary TXT record and hence agnostic to a
specific dns server software
Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

Richard Damon
In reply to this post by A. Schulze
On 12/15/14, 4:24 AM, A. Schulze wrote:

>
> wietse:
>
>> DMARC "verifies" the From: header against SPF, DKIM or both, but
>> only a poorly-informed person would require that the From: address
>> *always* verifies with SPF.
>
> for that reason it's more important the existing DKIM signature
> is still valid when the mlm redistribute the message to all subscribers.
>
> Unfortunately there are some rare situations a message must be modified.
> Well known example: Mailman sometimes change the message encoding.
> Not so often known: The mlm MTA submit to a subcriber MTA not capable
> 8BITMIME.
>
> In the last case a message is also modified and will not pass DMARC test
> even if the MLM host is known to "usually" not modifying messages.
>
Actually, I find that it is "normal" for the mailing list manager to
make changes that will
cause the message to fail a DKIM signature, and if fact it is required
by law in some
jurisdiction (some jurisdiction REQUIRE an unsubscribe link in every
message from a
mailing list). It is also very common to add a list code to the subject
for filtering.

>
> And exactly to debug such situations I with postfix could log
> "here is a message with 8 bit content and the remote MTA does not
>   announce 8BITMIME. I have to recode the message and that will
> invalidate
>   potential existing DKIM signatures"
>
> The other option is to convert every message to 7bit before signing.
> Doesn't sound as a strategy for the future ...
>
>> It would be unreasonable to expect that mailing list managers replace
>> the From: address of mailing list postings to match the list server's
>> IP addresses.
>
> Ehm,
> ironically that's exactly the solution preferred on the dmarc-discuss
> ml :-/
>
> Andreas
Yes, From replacement is currently, at least one of, the preferred
method to handle DMARC
issues with mailing list, as proposed by the DMARC group and the major
mail systems
that are causing the DMARC problem.

--
Richard Damon

Reply | Threaded
Open this post in threaded view
|

Re: Why does SPF fail sometimes?

James B. Byrne
In reply to this post by Richard Damon

On Sun, December 14, 2014 23:09, Richard Damon wrote:
>
> I regularly get important messages from Financial Institutions.
> Yes, they will typically ask me to log into their web site for confirmation
> of the message or to send "sensitive" information, but they do
> send notices by email that they hope I will see.

As you write, I should have used either the word confidential or sensitive
instead of important.  On the other hand nothing sent by email is guaranteed
to arrive at its intended destination.

>
> In fact, it is only because they DO send me email that a scammer has
> much chance of succeeding by sending a fake message, hoping that I
> will click on a link taking me to the wrong place.
>

Yes, which is why one never takes such links but copies and pastes them into
the address bar instead.


>
> If you read the standard, is says:
>
>    The originator fields indicate the mailbox(es) of the source of the
>     message.  The "From:" field specifies the author(s) of the message,
>     that is, the mailbox(es) of the person(s) or system(s) responsible
>     for the writing of the message.  The "Sender:" field specifies the
>     mailbox of the agent responsible for the actual transmission of the
>     message.  For example, if a secretary were to send a message for
>     another person, the mailbox of the secretary would appear in the
>     "Sender:" field and the mailbox of the actual author would appear in
>     the "From:" field.
>
> The *AUTHOR* of the message is the person who originally wrote it, not
> the mailing list. I digest is something different, the digest, as a whole,
> WAS created by the list, just like if a person collects a number of
> pieces written by other people, that person IS the author of the
> collection, and the individuals who wrote the pieces are the
> authors of the pieces, but not the whole.
>
> The mailing list software is NOT the author of the individual messages, but
> much more like the secretary mentioned in the RFC.
>
> (I did say arguably because some people differ in this intent)
>

Many MLMs modify the contents of every message that they process.  If the
message that is retransmitted is not in every respect identical to the one
originally received then how is that different from your postulated situation?
 Is it not a new message if the content changes, even if the change consists
of boilerplate?  I suppose one could make the same argument respecting those
noxious MTAs that automatically add twelve lines of vacuous legal babble
threatening how misdirected mail should be handled or else.

In the original RFC much provision was given over to identifying multiple
authors and multiple resenders.  Perhaps this situation could be avoided if
DMARC and MLM could agree on using the resent-from headers instead of fixating
on the From.

>  >
>  > The role of an MLM is really no different than if you or I forwarded
>  > a message we received on to a third party.  Who is the FROM id in
>  > that case?  Arguably, most MLMs have been doing it wrong since the
>  > beginning and DMARC is just highlighting the logical inconsistencies
>  > and contradictions in prevalent MLM practice.
>  >
>
> The big difference is that the MLM is an AUTOMATED PROCESS (not
> significantly different from postfix). If I manually forward a message,
> that is not (by definition) an automated process, and generally the MUA
> will build a new message containing the original message (and possibly
> my notes about the message), so this is reasonable to change authorship
> of the forwarding to be the forwarder. As a point of reference,
> when you setup and automated forward rule for a mailbox to some other
> mailbox, THAT forwarding does NOT normally change the From: line.
>

Not so long ago I thought similarly but now have reservations on the matter.
For one, when one sends a message to a MLM one has sent it to the MLM and not
to the subscribers.  It has been delivered to its intended recipient.  That
transaction is complete. What the MLM chooses to do with your message is now
beyond your capacity to influence.  And whether the process is automated or
subject to manual intervention by a list owner really does not change the
situation irrespective of any limitations to either approach.

As for the secretaries in the RFC example that betrays a mindset of which the
pointy-haired boss would be proud.  Given the time at which the RFC was
written (1982) I much suspect that what the RFC authors had in mind was that
the secretary would compose the message and then send it out under her boss's
name in much the same manner as secretaries once typed the signatory's
initials in caps and their own in lower-case (AKL/jbb) at the bottom of
letters that they, the secretaries, generally wrote and had their boss sign.
I doubt that the authors had in mind that the boss would compose an email and
send it to his secretary just to have her retransmit it.  I do not recall
offices working like that.

Indeed, I do not recall managers even having access to electronic messaging
apparatus until well into the 1990s or early 2000s.  The big thing in 1982 was
the IBM PC and Word Perfect for DOS. And no manager with ambition would be
caught dead sitting at a word-processing machine.  That was secretarial work.



--
***          E-Mail is NOT a SECURE channel          ***
James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

12