Big problem with this mailing list and Majordomo regarding DMARC

classic Classic list List threaded Threaded
48 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Scott Kitterman-4
On Friday, April 19, 2019 05:38:08 PM B. Reino wrote:

> On Fri, 19 Apr 2019, TG Servers wrote:
> > according to RFC this would be the full list for rspamd
> >
> >  sign_headers = 'from:reply-to:subject:date:\
> >  to:cc:resent-to:resent-cc:resent-from:resent-date\
> >  in-reply-to:references:<all the list commands I could not use here
> > because the mailing list interprets them as commands :) and blocks the
> > message>';
> >
> > although they leave it open as "subjective" regarding message-id,
> > in-reply-to and references
>
> Thanks for the clarification!
>
> Yet, "subjective" (or trade-off, etc.) does not mean "will be changed
> remotely", so I fail to see the issue here (and man 5 opendkim.conf does
> not mention it AFAICT..)

I'd suggest reading RFC 6376, Section 5.4 (Determine the Header Fields to
Sign) [1] directly.  I think it's advice holds up reasonably well.

Scott K

[1] https://tools.ietf.org/html/rfc6376#section-5
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Ralph Seichter-2
In reply to this post by B. Reino
* B. Reino:

> On Fri, 19 Apr 2019, Benny Pedersen wrote:
>
>> B. Reino skrev den 2019-04-19 15:48:
>>
>>> sign_headers = 'from:to:subject:date:message-id:in-reply-to:references';
>>
>> man 5 opendkim.conf
>>
>> dont sign headers that are added or changed remotely
>
> I'm not sure I follow here. AFAIK all of the headers I mentioned above
> are user/MUA generated (.. I know Message-ID can be generated by MTA
> if the MUA sucks and doesn't do it itself).

Your header selection is quite alright, if a bit shorter than my own
preferred list:

  Autocrypt, From, To, Subject, Date, Content-Language, Content-Type,
  In-Reply-To, Message-ID, References, User-Agent, X-Mailer

The message ID should definitely be signed. Even if it is generated
by the MTA, that should happen before the DKIM signature is crated.
Otherwise I'd consider the MTA/DKIM pair misconfigured by the admin.

-Ralph
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Benny Pedersen-2
In reply to this post by Scott Kitterman-4
Scott Kitterman skrev den 2019-04-19 17:48:

> I'd suggest reading RFC 6376, Section 5.4 (Determine the Header Fields
> to
> Sign) [1] directly.  I think it's advice holds up reasonably well.

thanks for link, its just that mailman breaks that rfc then, if changing
reply-to
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

TG Servers
In reply to this post by Benny Pedersen-2
yes it seems mailman changes reply-to, I just took the fields out of
RFC6376 now...

On 19/04/2019 17:47, Benny Pedersen wrote:

> TG Servers skrev den 2019-04-19 16:48:
>> according to RFC this would be the full list for rspamd
>>
>>  sign_headers = 'from:reply-to:subject:date:\
>>  to:cc:resent-to:resent-cc:resent-from:resent-date\
>>  in-reply-to:references:<all the list commands I could not use here
>> because the mailing list interprets them as commands :) and blocks the
>> message>';
>
> mailman changes reply-to, no ?
>
> is it time to let rspamd solve its own problems ?

Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Richard Damon
In reply to this post by Benny Pedersen-2
On 4/19/19 12:00 PM, Benny Pedersen wrote:
> Scott Kitterman skrev den 2019-04-19 17:48:
>
>> I'd suggest reading RFC 6376, Section 5.4 (Determine the Header
>> Fields to
>> Sign) [1] directly.  I think it's advice holds up reasonably well.
>
> thanks for link, its just that mailman breaks that rfc then, if
> changing reply-to
>
Or that RFC break Mailman, and many other long established methods of
operation for mailing list.

From what I have seen from DKIM is that domains that implement it really
should be required to tell their users that they should not be using any
remailing service (like most mailing list) that don't adhere to some new
DKIM rules,  needed to avoid breaking the DKIM signature, even though
following these rules will break some traditionally provided features,
and may even force the list to break some laws (laws on mass mailings
that REQUIRE instructions on how to unsubscribe be included in the message)

--
Richard Damon

Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Peter Ajamian
In reply to this post by Nick-5
On 19/04/19 11:16 PM, Nick wrote:
> You might want to consider reducing the list of headers in your DKIM
> signatures.  E.g. your signed-headers list includes 'sender' but the
> mailing list adds its own 'sender', which is enough to invalidate your
> signature.

This is going to be an ongoing problem because RFC6376 actually
recommends including the Sender header:

 From 5.4 INFORMATIVE OPERATIONS NOTE:

"For this reason, signing fields present in the message such as Date,
Subject, Reply-To, *Sender*, and all MIME header fields are highly
advised." (emphasis mine)

Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Richard Damon
On 4/19/19 10:04 PM, Peter wrote:

> On 19/04/19 11:16 PM, Nick wrote:
>> You might want to consider reducing the list of headers in your DKIM
>> signatures.  E.g. your signed-headers list includes 'sender' but the
>> mailing list adds its own 'sender', which is enough to invalidate your
>> signature.
>
> This is going to be an ongoing problem because RFC6376 actually
> recommends including the Sender header:
>
> From 5.4 INFORMATIVE OPERATIONS NOTE:
>
> "For this reason, signing fields present in the message such as Date,
> Subject, Reply-To, *Sender*, and all MIME header fields are highly
> advised." (emphasis mine)
>
If you look at the background behind DKIM, one of the major impetuses
was protecting transactional emails, and protection from attacks like
phishing. For these sorts of emails, that stricter protection makes
sense. These sorts of emails also aren't sent through mailing lists.

Effectively, if you decide to use DKIM to protect your domain's outgoing
email, then you really need to tell your users about the issue with
mailing lists, as the choice to use DKIM basically says that most
mailing list should be off limits to your users, as it is very common
for mailing lists to break the DKIM signature, so it really is YOUR
problem to adjust your DKIM settings and Authorized Usage Policy to make
your system work for your users. I have to regularly tell users of a
mailing list that I run that the reason the list removes their email
address out of the From: field is that they are using a broken email
system that isn't compatible with the use of mailing list.

Note also, these RFCs are just Standards Track, which says that they are
not yet 'full standards' but still evolving, and I believe that one of
the issues that needs to be worked out is to figure out how to improve
their interoperability for general emails with traditional mailing lists.

--
Richard Damon

Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Peter Ajamian
On 20/04/19 2:50 PM, Richard Damon wrote:

> If you look at the background behind DKIM, one of the major impetuses
> was protecting transactional emails, and protection from attacks like
> phishing. For these sorts of emails, that stricter protection makes
> sense. These sorts of emails also aren't sent through mailing lists.
>
> Effectively, if you decide to use DKIM to protect your domain's outgoing
> email, then you really need to tell your users about the issue with
> mailing lists, as the choice to use DKIM basically says that most
> mailing list should be off limits to your users, as it is very common
> for mailing lists to break the DKIM signature, so it really is YOUR
> problem to adjust your DKIM settings and Authorized Usage Policy to make
> your system work for your users. I have to regularly tell users of a
> mailing list that I run that the reason the list removes their email
> address out of the From: field is that they are using a broken email
> system that isn't compatible with the use of mailing list.
>
> Note also, these RFCs are just Standards Track, which says that they are
> not yet 'full standards' but still evolving, and I believe that one of
> the issues that needs to be worked out is to figure out how to improve
> their interoperability for general emails with traditional mailing lists.

I'm not disagreeing with any of this.  It simply boils down to that when
a current RFC recommends a certain practice you shouldn't be surprised
that people will follow that recommendation.  What then follows is that
people who use google, microsoft or other major ESPs that enforce DMARC
will end up either not getting a large portion of messages sent to the
list, or have to hunt through Spam to find them.  At the end of the day
this means that the practical implications of this are problematic at best.

It means that I also take issue when Wietse ways that the mailing list
is DKIM compliant, because clearly that statement is based on the DKIM
signature not including certain headers that the mailing list alters.
What might be more accurate is to say that the mailing list is DKIM
compliant just as long as the DKIM signature doesn't include certain
headers, some of which are actually recommended to be included by the
relevant RFCs.  When looked at in that light it becomes more clear that
the DKIM compliance of the mailing list is spotty at best.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Bill Cole-3
In reply to this post by Richard Damon
On 19 Apr 2019, at 22:50, Richard Damon wrote:

> Note also, these RFCs are just Standards Track, which says that they are
> not yet 'full standards' but still evolving, and I believe that one of
> the issues that needs to be worked out is to figure out how to improve
> their interoperability for general emails with traditional mailing lists.

Sadly, no. See https://www.rfc-editor.org/info/std76



--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Peter Ajamian
In reply to this post by Peter Ajamian
On 20/04/19 3:15 PM, Peter wrote:

> I'm not disagreeing with any of this.  It simply boils down to that when
> a current RFC recommends a certain practice you shouldn't be surprised
> that people will follow that recommendation.  What then follows is that
> people who use google, microsoft or other major ESPs that enforce DMARC
> will end up either not getting a large portion of messages sent to the
> list, or have to hunt through Spam to find them.  At the end of the day
> this means that the practical implications of this are problematic at best.
>
> It means that I also take issue when Wietse ways that the mailing list
> is DKIM compliant, because clearly that statement is based on the DKIM
> signature not including certain headers that the mailing list alters.
> What might be more accurate is to say that the mailing list is DKIM
> compliant just as long as the DKIM signature doesn't include certain
> headers, some of which are actually recommended to be included by the
> relevant RFCs.  When looked at in that light it becomes more clear that
> the DKIM compliance of the mailing list is spotty at best.

Just as a follow on, I've been finding that taking the approach of
trying to pass on messages in a mailing list unaltered without them
being marked as SPAM is in this day and age becoming increasingly
difficult and perhaps this approach should be abandoned as I believe the
situation will only get worse in the future.

Instead of taking the approach that we can pass on these messages
unaltered and keep the original authenticity intact, perhaps we should
intead take the approach that we are not just passing these messages on,
but rather re-authoring the messages so that they originate from the
mailing list itself rather than from the original sender.  This
essentially requires taking ownership of the messages so that it becomes
the mailing lists own reputation that defines deliver-ability rather
than that of the original sender.  This is the approach being taken by
an increasing number of MLMs (such as GNU MailMan).

Fortunately we don't actually have to switch to a modern MLM in order to
take this approach as it can be achieved largely through the postfix
backend without the help of the MLM:

* Use header_checks to strip out any existing DKIM signatures and
rewrite the From: header.

* Use canonical_maps or sender_canonical_maps to rewrite the envelope
sender (probably not needed here or already implemented as the envelope
sender is indeed already rewritten for this list).

* DKIM: sign the resulting messages with our own DKIM signature, after
making any changes to the message.

* Anti-SPAM: I have yet to see much of any SPAM sent through this list,
so I assume that the Anti-SPAM solution on it is already quite good, but
as a general rule, since we would be taking complete ownership of any
posts sent to the list it pays to not be doing so for a bunch of SPAM.

While I do understand the ideal of keeping messages pristine and
unaltered, I think the current and future email landscapes with major
ESPs is simply going to make that approach increasingly impractical.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Scott Kitterman-4
In reply to this post by Peter Ajamian


On April 20, 2019 3:15:18 AM UTC, Peter <[hidden email]> wrote:

>On 20/04/19 2:50 PM, Richard Damon wrote:
>> If you look at the background behind DKIM, one of the major impetuses
>> was protecting transactional emails, and protection from attacks like
>> phishing. For these sorts of emails, that stricter protection makes
>> sense. These sorts of emails also aren't sent through mailing lists.
>>
>> Effectively, if you decide to use DKIM to protect your domain's
>outgoing
>> email, then you really need to tell your users about the issue with
>> mailing lists, as the choice to use DKIM basically says that most
>> mailing list should be off limits to your users, as it is very common
>> for mailing lists to break the DKIM signature, so it really is YOUR
>> problem to adjust your DKIM settings and Authorized Usage Policy to
>make
>> your system work for your users. I have to regularly tell users of a
>> mailing list that I run that the reason the list removes their email
>> address out of the From: field is that they are using a broken email
>> system that isn't compatible with the use of mailing list.
>>
>> Note also, these RFCs are just Standards Track, which says that they
>are
>> not yet 'full standards' but still evolving, and I believe that one
>of
>> the issues that needs to be worked out is to figure out how to
>improve
>> their interoperability for general emails with traditional mailing
>lists.
>
>I'm not disagreeing with any of this.  It simply boils down to that
>when
>a current RFC recommends a certain practice you shouldn't be surprised
>that people will follow that recommendation.  What then follows is that
>
>people who use google, microsoft or other major ESPs that enforce DMARC
>
>will end up either not getting a large portion of messages sent to the
>list, or have to hunt through Spam to find them.  At the end of the day
>
>this means that the practical implications of this are problematic at
>best.
>
>It means that I also take issue when Wietse ways that the mailing list
>is DKIM compliant, because clearly that statement is based on the DKIM
>signature not including certain headers that the mailing list alters.
>What might be more accurate is to say that the mailing list is DKIM
>compliant just as long as the DKIM signature doesn't include certain
>headers, some of which are actually recommended to be included by the
>relevant RFCs.  When looked at in that light it becomes more clear that
>
>the DKIM compliance of the mailing list is spotty at best.

Not at all.  The unusual aspect of this case is the originator including Sender in the mail sent to the list.  Don't do that and it's all fine.

Scott K
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Peter Ajamian
On 20/04/19 3:57 PM, Scott Kitterman wrote:
> Not at all.  The unusual aspect of this case is the originator including Sender in the mail sent to the list.  Don't do that and it's all fine.

So we should expect people to do what we think is best practice instead
of what is recommended in the appropriate standards documents, and when
they don't it's their fault for actually following the standards?  Ummm, ok.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Scott Kitterman-4


On April 20, 2019 4:23:09 AM UTC, Peter <[hidden email]> wrote:

>On 20/04/19 3:57 PM, Scott Kitterman wrote:
>> Not at all.  The unusual aspect of this case is the originator
>including Sender in the mail sent to the list.  Don't do that and it's
>all fine.
>
>So we should expect people to do what we think is best practice instead
>
>of what is recommended in the appropriate standards documents, and when
>
>they don't it's their fault for actually following the standards?
>Ummm, ok.

Sigh.  No.  That's not what I wrote at all.

RFC 6376 suggests signing the sender field if present.  It says nothing about adding it.  For that you want RFC 5322, Section 3.6.2.  (Originator Fields).

Unless a third party is transmitting mails to a mailing list on your behalf, it shouldn't be there.

Scott K
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Peter Ajamian
On 20/04/19 4:46 PM, Scott Kitterman wrote:
> RFC 6376 suggests signing the sender field if present.  It says nothing about adding it.  For that you want RFC 5322, Section 3.6.2.  (Originator Fields).
>
> Unless a third party is transmitting mails to a mailing list on your behalf, it shouldn't be there.

...and I never said anything about adding it either.  We're talking
about what fields to sign, not whether they should be added or exist in
the message prior to signing.  RFC 6376 does go on to say that headers
can be signed that are not present to enforce that they not be added later.

Granted in this particular case, and given what Sender is for, it
probably shouldn't be signed if it's not present, but the RFC does not
make that explicitly clear, and I would not hold someone at fault for
signing the Sender header based on what that RFC says.

But this is just one example of where a header might be signed and then
is later added or altered by the mailing list, thereby invalidating the
signature.  I'm sure there have been others as I regularly see mail from
this list end up in Spam and my point remains that this seems to be an
issue that will only get worse in the future, not better.

At the end of the day, messages from this list are ending up in people's
Spam folder, or are not being delivered at all.  We can go on all day
pointing the finger at the sender but that really won't solve the issue
in general, because there will always be another sender who breaks the
rules or doesn't get it, or can't actually control this stuff for their
email.


Peter
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Scott Kitterman-4
On Saturday, April 20, 2019 05:04:24 PM Peter wrote:

> On 20/04/19 4:46 PM, Scott Kitterman wrote:
> > RFC 6376 suggests signing the sender field if present.  It says nothing
> > about adding it.  For that you want RFC 5322, Section 3.6.2.  (Originator
> > Fields).
> >
> > Unless a third party is transmitting mails to a mailing list on your
> > behalf, it shouldn't be there.
> ...and I never said anything about adding it either.  We're talking
> about what fields to sign, not whether they should be added or exist in
> the message prior to signing.  RFC 6376 does go on to say that headers
> can be signed that are not present to enforce that they not be added later.
>
> Granted in this particular case, and given what Sender is for, it
> probably shouldn't be signed if it's not present, but the RFC does not
> make that explicitly clear, and I would not hold someone at fault for
> signing the Sender header based on what that RFC says.
>
> But this is just one example of where a header might be signed and then
> is later added or altered by the mailing list, thereby invalidating the
> signature.  I'm sure there have been others as I regularly see mail from
> this list end up in Spam and my point remains that this seems to be an
> issue that will only get worse in the future, not better.
>
> At the end of the day, messages from this list are ending up in people's
> Spam folder, or are not being delivered at all.  We can go on all day
> pointing the finger at the sender but that really won't solve the issue
> in general, because there will always be another sender who breaks the
> rules or doesn't get it, or can't actually control this stuff for their
> email.

If you signed a non-existent sender field and send mail to a mailing list that
(not atypically adds it) and your signature is broken, look in the mirror for
the source of the problem.  It's neither the mailing list nor the RFCs.

I'm done.

Scott K
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

@lbutlr
In reply to this post by Peter Ajamian
On 19 Apr 2019, at 23:04, Peter <[hidden email]> wrote:
> But this is just one example of where a header might be signed and then is later added or altered by the mailing list,

The only headers that mailing lists often alter are subject (though i think that is dying off, hopefully) and the only non-list-mumble header they add generally is sender.

Ig you are signing a non-existent sender in a message to a list, that is a mistake.

If you are adding a sender header to a message to a list and signing with that header, that is a mistake.

Or, you can insist you are right and break list messages. Your call.


--
Major Strasser has been shot. Round up the usual suspects.


Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Dominic Raferd
In reply to this post by Benny Pedersen-2
On Fri, 19 Apr 2019 at 14:11, Benny Pedersen <[hidden email]> wrote:
i have now disabled milters from trusted maillists ips

How did you do this? It might help me. I have missed some of this thread because OP's mails are blocked (correctly of course) by my opendmarc.
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

TG Servers

Dominic, you should get the mails now, don't you? 

On 20 April 2019 12:04:30 Dominic Raferd <[hidden email]> wrote:

On Fri, 19 Apr 2019 at 14:11, Benny Pedersen <[hidden email]> wrote:
i have now disabled milters from trusted maillists ips

How did you do this? It might help me. I have missed some of this thread because OP's mails are blocked (correctly of course) by my opendmarc.

Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Dominic Raferd
In reply to this post by Dominic Raferd


On Sat, 20 Apr 2019 at 11:18, TG Servers <[hidden email]> wrote:

Dominic, you should get the mails now, don't you? 

On 20 April 2019 12:04:30 Dominic Raferd <[hidden email]> wrote:

On Fri, 19 Apr 2019 at 14:11, Benny Pedersen <[hidden email]> wrote:
i have now disabled milters from trusted maillists ips

How did you do this? It might help me. I have missed some of this thread because OP's mails are blocked (correctly of course) by my opendmarc.

yes I receive them now.
Reply | Threaded
Open this post in threaded view
|

Re: Big problem with this mailing list and Majordomo regarding DMARC

Ralph Seichter-2
In reply to this post by Peter Ajamian
* Peter:

> Granted in this particular case, and given what Sender is for, it
> probably shouldn't be signed if it's not present, but the RFC does not
> make that explicitly clear, and I would not hold someone at fault for
> signing the Sender header based on what that RFC says.

Signing a non-existing (!) header. Right. Mind if I watch? :-)

The Postfix mailing list is one of the few that does not rewrite message
subjects or bodies, giving existing DKIM signatures at least a fighting
chance. If somebody insists on adding a Sender header and including it
in the signature before posting to this mailing list, that's his/her
problem. Speaking of which:

> At the end of the day, messages from this list are ending up in
> people's Spam folder, or are not being delivered at all.

DKIM signature mismatches can add just a little or a lot to a spam
score. The scores can be offset by taking other headers like List-Id
into account. The recipient therefore has a choice about how a given
incoming message is processed.

As I mentioned before, I use (sub)domains without DMARC policies for
mailing lists, and for some of them even forgo DKIM signatures. DKIM and
MLs just don't play well together, so I try to mitigate the issues.

-Ralph
123