Cannot sign with DKIM on same-server web and mail

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Cannot sign with DKIM on same-server web and mail

linkcheck
I've looked online for solutions to this problem (including postfix and
sendmail documentation) but with no luck so far.

I've  been running a Postfix mail server for several years (currently Linux
Mint 18.1 (Ubuntu 16.4) with postfix 3.1.0) and implemented SPF, DKIM and
DMARC a few years ago. All works well for about two dozen domains.

I also have a Windows web server which sends out mail from web forms via the
mail server (using a local mail sender client) to the domains hosted on the
mail server through port 25. This has also worked well for many years and
achieves passes for SPF, DKIM and DMARC.

I have a second, recentlly set up web server - Apache 2.4.18 - on the same
VPS as the mail server. It's form mail is sent using php's mail() which
sends via "/usr/sbin/sendmail -t -i". This does not, by default, DKIM sign,
although remote recipients pass SPF and DMARC. In order to get DKIM
authentication I have removed the no_milters option from master.cf's
receive_override_options. This, of course, adds a second DKIM signature to
each email from Windows forms and general mail.

What changes to main/master do I need to make in order to DKIM sign all mail
once only, preferably before spamassassin?

master.cf extract...
pickup    fifo  n       -       n       60      1       pickup
  -o content_filter=
  -o
receive_override_options=no_header_body_checks,no_unknown_recipient_checks
#  ,no_milters

smtp      inet  n       -       n       -       -       smtpd
  -o content_filter=spamfilter

main.cf extract...
inet_interfaces = all
milter_default_action = accept
milter_protocol = 6
# list of: clamav, dkim, dmarc
smtpd_milters =
unix:/var/run/clamav/clamav-milter.ctl,local:/var/run/opendkim/opendkim.sock,inet:localhost:8893
non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock,inet:localhost:8893





--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

Jaroslaw Rafa
Dnia 31.10.2019 o godz. 12:16:56 linkcheck pisze:
>
> What changes to main/master do I need to make in order to DKIM sign all mail
> once only, preferably before spamassassin?

Not long ago I asked a similar question here.

The best answer is to use spamassassin as a milter, not as a post-queue
content filter as you have (and as I had).
After I changed configuration to run spamassassin as milter, everything is
signed only once.

This is the relevant part of my config:

main.cf:

smtpd_milters = inet:localhost:10025, unix:spamass/spamass.sock
non_smtpd_milters = inet:localhost:10025

(DKIM is running on localhost:10025)

master.cf:

submission inet n       -       -       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
  -o smtpd_milters=inet:localhost:10025

As you can see, I don't run spamassassin on outgoing mail (either submitted
via /usr/sbin/sendmail or via submission port - in both cases there is only
DKIM specified in milters, and spamassassin is run only for smtpd. If you
want to run spamassassin for outgoing mail too, you should add
unix:spamass/spamass.sock to non_smtpd_milters as well and remove the "-o
smtpd_milters=..." line from aobove master.cf entry.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
Jaroslaw Rafa wrote
> Dnia 31.10.2019 o godz. 12:16:56 linkcheck pisze:
> The best answer is to use spamassassin as a milter, not as a post-queue
> content filter as you have (and as I had).
> After I changed configuration to run spamassassin as milter, everything is
> signed only once.

Thanks, but not sure that's the answer.

I tried running spamassassin as a milter when I first set up postfix and
eventually had to resort to the setup I gave.

Also, note that the problem I'm trying to solve is apache posting through
pickup, which does not DKIM sign due to DKIM being applied before pickup, as
I understand it.




--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

Jaroslaw Rafa
In reply to this post by linkcheck
> Also, note that the problem I'm trying to solve is apache posting through
> pickup, which does not DKIM sign due to DKIM being applied before pickup, as
> I understand it.

To have DKIM applied to messages posted via pickup, you have to include DKIM milter in non_smtpd_milters= . This parameter applies to messages posted via pickup, while smtpd_milters= applies to messages posted via SMTP client.

But if you do that and run spamassassin as content filter, every message will be signed twice. That's why you have to run spamassassin as milter as well.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

@lbutlr
In reply to this post by linkcheck
On 01 Nov 2019, at 10:03, linkcheck <[hidden email]> wrote:
> Jaroslaw Rafa wrote
>> Dnia 31.10.2019 o godz. 12:16:56 linkcheck pisze:
>> The best answer is to use spamassassin as a milter, not as a post-queue
>> content filter as you have (and as I had).
>> After I changed configuration to run spamassassin as milter, everything is
>> signed only once.
>
> Thanks, but not sure that's the answer.

It definitely is.

> I tried running spamassassin as a milter when I first set up postfix and
> eventually had to resort to the setup I gave.

Had to why?

> Also, note that the problem I'm trying to solve is apache posting through
> pickup, which does not DKIM sign due to DKIM being applied before pickup, as
> I understand it.

Apache should not be posting mail via pickup. Use an SMTP plugin that authenticates just like anyone else.



--
Oh, he's just like any other man, only more so.

Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
In reply to this post by Jaroslaw Rafa
Sorry for the delay in replying. I've been looking at this and trying to make
it work in my head, but keep coming up with DKIM running twice. Please bear
with me. Your setup of...

smtpd_milters = inet:localhost:10025, unix:spamass/spamass.sock
non_smtpd_milters = inet:localhost:10025

...suggests to me that the sequence of operation is DKIM followed by SPAMASS
(both from smtpd_milters, assuming they run in sequence) followed by DKIM
via pickup... And I see the flaw in that now!

Pickup only gets run from sendmail which is called by content_filter OR by
apache. So that now makes sense. Taken me hours to see that. :(

So what I need is, as you said, to remove content_filter in master.cf and in
main.cf to put in a new sequence...

smtpd_milters = unix:/var/run/opendkim/opendkim.sock,
unix:spamass/spamass.sock, unix:/var/run/clamav/clamav-milter.ctl,
unix:/var/run/opendmarc/opendmarc.sock

non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock

I assume I do not need to include dmarc in the non_smtpd_milters since it's
outgoing only. Should I move dmarc between dkim and spamass in
smtpd_milters?

If the above is correct my remaining problem would be to determine which of
the various spamassassin / spamass-milter / spamd file groups I have to set
up and where to put the sock.

Thanks for your input. Sorry I doubted you first posting. :(




--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

Jaroslaw Rafa
Dnia  4.11.2019 o godz. 04:31:51 linkcheck pisze:
>
> Pickup only gets run from sendmail which is called by content_filter OR by
> apache. So that now makes sense. Taken me hours to see that. :(
>
> So what I need is, as you said, to remove content_filter in master.cf and in
> main.cf to put in a new sequence...

Exactly :) That's the whole point of this setup - to avoid running pickup a
second time.

> I assume I do not need to include dmarc in the non_smtpd_milters since it's
> outgoing only. Should I move dmarc between dkim and spamass in
> smtpd_milters?

I don't know as I don't use DMARC. I only DKIM sign outgoing mail, I don't
verify DKIM nor DMARC on incoming mail. Just try what order works best.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
In reply to this post by @lbutlr
@lbutlr wrote
> On 01 Nov 2019, at 10:03, linkcheck &lt;

> postfix@.co

> &gt; wrote:
>> Jaroslaw Rafa wrote
> Apache should not be posting mail via pickup. Use an SMTP plugin that
> authenticates just like anyone else.

If the mail and web servers were separate I would agree but there is a lot
of overhead in adding (eg) phpmail when all that is required is a simple
non-authenticated posting into postfix's sendmail




--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
In reply to this post by Jaroslaw Rafa
Jaroslaw Rafa wrote
> Dnia  4.11.2019 o godz. 04:31:51 linkcheck pisze:
> I don't know as I don't use DMARC. I only DKIM sign outgoing mail, I don't
> verify DKIM nor DMARC on incoming mail. Just try what order works best.

Ok. Thanks for all the help. :)




--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
In reply to this post by linkcheck
I applied the recommendations from this thread (for which, many thanks!)
with some help from the spamassassin forum. Almost all of it works now
with the following exception. On postfix restart the following message
is logged.

"Could not retrieve sendmail macro "i"!. Please add it to
confMILTER_MACROS_ENVFROM"

I added a few lines found elsewhere, specifically
mua_recipient_restrictions, milter_connect_macros, mua_milters in
main.cf and modified master.cf accordingly but still get that error in
the logs.

The postfix document MILTER_README.html, under the workarounds section,
states...

"Some Milter applications use the "{if_addr}" macro to recognize local
mail; this macro does not exist in Postfix. Workaround: use the
"{daemon_addr}" (Postfix ≥ 3.2) or "{client_addr}" macro instead."

My setup for the relevant parameters is:

==========
main.cf:

mua_recipient_restrictions =
   reject_non_fqdn_recipient,
   reject_unknown_recipient_domain,
   permit_sasl_authenticated,
   reject

milter_default_action = accept
milter_protocol = 6
milter_connect_macros="i j {daemon_name} v {if_name} _"

smtpd_milters = unix:/var/run/opendkim/opendkim.sock,
unix:/var/run/opendmarc/opendmarc.sock,
unix:/var/run/spamass/spamass.sock, unix:/var/run/clamav/clamav-milter.ctl

non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock

mua_milters = unix:/var/run/opendkim/opendkim.sock,
unix:/var/run/clamav/clamav-milter.ctl

master.cf:

smtp inet n       -       n       -       -       smtpd

submission inet n       -       n       -       -       smtpd
   -o syslog_name=postfix/submission
   -o smtpd_tls_wrappermode=no
   -o smtpd_tls_security_level=encrypt
   -o smtpd_recipient_restrictions=$mua_recipient_restrictions
   -o smtpd_sasl_auth_enable=yes
   -o receive_override_options=no_header_body_checks
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_sasl_type=dovecot
   -o smtpd_sasl_path=private/auth
   -o smtpd_milters=$mua_milters

pickup    fifo  n       -       n       60      1       pickup
   -o content_filter=
   -o
receive_override_options=no_header_body_checks,no_unknown_recipient_checks
==========

Spamassasin now adds the token UNPARSEABLE_RELAY to every email.
Research online suggests this is due to an incorrect setting in
milter_connect_macros.

I have removed the "i" from milter-connect-macros (most online texts
omit it anyway) with the result that dkim authentication in emails shows
"unknown-host", although it correctly passes the authentication...

   DKIM-Filter: OpenDKIM Filter v2.10.3 unknown-host 5A44B320316
   Authentication-Results: unknown-host; dkim=pass (etc)

I tried removing {if_name} but with no positive result. I've also tried
changing if_name to daemon_addr and client_addr, still with no success.

What should milter_connect_macros actually be? Is there something else
I'm missing?

I know I can disable UNPARSEABLE_RELAY in spamassassin but I would
rather get this right.

Software Versions:

Postfix:
   postconf -d | grep mail_version
   mail_version = 3.1.0

Spamassassin:
   spamassassin -V
   SpamAssassin version 3.4.2
     running on Perl version 5.22.1

Opendkim
   opendkim -V
opendkim: OpenDKIM Filter v2.10.3
        Compiled with OpenSSL 1.0.2g  1 Mar 2016

Opendmarc
   opendmarc -V
opendmarc: OpenDMARC Filter v1.3.1

Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

David Bürgin
On 01/12/2019 12:01, Linkcheck wrote:
> I applied the recommendations from this thread (for which, many thanks!) with some help from the spamassassin forum. Almost all of it works now with the following exception. On postfix restart the following message is logged.
>
> "Could not retrieve sendmail macro "i"!. Please add it to confMILTER_MACROS_ENVFROM"

The ‘Could not retrieve sendmail macro "i"’ message is due to a bug in
spamass-milter. It cannot be worked around, but it is also completely
harmless, ie it does not actually impact operation in any way.

This bug has been open for many years, more info at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=696856.


--
David
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
Ok, thanks. I can live with that.

But what about UNPARSEABLE_RELAY? How can I preoperly fix that? Do I
really have to nullify the rule or is there something in postfix that
I've got wrong?
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

David Bürgin
On 01/12/2019 15:18, Linkcheck wrote:
> But what about UNPARSEABLE_RELAY? How can I preoperly fix that? Do I really have to nullify the rule or is there something in postfix that I've got wrong?

Try appending _ to the default connect macros. That does it for me.

milter_connect_macros = j {daemon_name} {daemon_addr} v _


--
David
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

Jaroslaw Rafa
In reply to this post by linkcheck
Dnia  1.12.2019 o godz. 14:18:32 Linkcheck pisze:
> Ok, thanks. I can live with that.
>
> But what about UNPARSEABLE_RELAY? How can I preoperly fix that? Do I
> really have to nullify the rule or is there something in postfix
> that I've got wrong?

I fixed this in one of SpamAssassin's Perl modules,
/usr/share/perl5/Mail/SpamAssassin/Message/Metadata/Received.pm.

Here is my diff:

*** Received.orig.pm 2014-02-07 09:36:23.000000000 +0100
--- Received.pm 2019-09-27 12:38:23.000000000 +0200
***************
*** 146,153 ****
 
      my $relay = $self->parse_received_line ($line);
      if (!defined $relay) {
!       dbg("received-header: unparseable: $line");
!       $self->{num_relays_unparseable}++;
      }
 
      # undefined or 0 means there's no result, so goto the next header
--- 146,161 ----
 
      my $relay = $self->parse_received_line ($line);
      if (!defined $relay) {
!       # workaround: if SpamAssassin is running as a milter, first header is
!       # artificially generated and doesn't contain IP address nor Message ID
!       # like this:
!       # from brama.eko.wroc.pl (unknown)_ by rafa.eu.org(Postfix 2.9.6/8.13.0) with SMTP id unknown_ Fri, 27 Sep 2019 12:31:50 +0200_ (envelope-from <[hidden email]>
!       if ($line =~ m/^from +[-.a-zA-Z0-9]+ +\(unknown\).*with +SMTP +id +unknown.*\(envelope-from/) {
!         dbg("received-header: added by milter-ignored: $line");
!       } else {
!         dbg("received-header: unparseable: $line");
!         $self->{num_relays_unparseable}++;
!       }
      }
 
      # undefined or 0 means there's no result, so goto the next header

--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
In reply to this post by David Bürgin
Thanks, but I already had that. Although I had the "v" before the
daemon_addr when I first tried it...
   milter_connect_macros="i j {daemon_name} v {daemon_addr} _"

I have now tried it with the v where you suggest it but still gives
UNPARSEABLE_RELAY. Also, I understand the quotes are essential because
the line includes spaces?


Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

Jaroslaw Rafa
Dnia  2.12.2019 o godz. 11:12:15 Linkcheck pisze:
> Thanks, but I already had that. Although I had the "v" before the
> daemon_addr when I first tried it...
>   milter_connect_macros="i j {daemon_name} v {daemon_addr} _"
>
> I have now tried it with the v where you suggest it but still gives
> UNPARSEABLE_RELAY. Also, I understand the quotes are essential
> because the line includes spaces?

Try fixing it in SpamAssassin's code, as I wrote. It works perfectly for me.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

Wietse Venema
In reply to this post by linkcheck
Linkcheck:
[ Charset windows-1252 converted... ]
> Thanks, but I already had that. Although I had the "v" before the
> daemon_addr when I first tried it...
>    milter_connect_macros="i j {daemon_name} v {daemon_addr} _"

Drop the quotes! Where does Postfix documentation say that
you need to use quoted strings?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

David Bürgin
In reply to this post by linkcheck
On 02/12/2019 12:12, Linkcheck wrote:
> Thanks, but I already had that. Although I had the "v" before the daemon_addr when I first tried it...
>   milter_connect_macros="i j {daemon_name} v {daemon_addr} _"
>
> I have now tried it with the v where you suggest it but still gives UNPARSEABLE_RELAY. Also, I understand the quotes are essential because the line includes spaces?

spamass-milter needs the _ macro to construct a valid ‘Received’ header
line for SpamAssassin. Try ‘postconf -d milter_connect_macros’ to see
the default setting.

$ postconf -d | grep milter_connect_macros
milter_connect_macros = j {daemon_name} {daemon_addr} v

Then add the _ macro to that setting (no quotes, order doesn’t matter).
I gave my setting in my earlier message. You’re using an old version of
Postfix, so yours will be different. In any case no hack should be
necessary, it is a config issue.


Cheers,
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
Thanks for your help. I checked as you suggested and got
milter_connect_macros=j {daemon_name} v (no quotes, no underscore). I
commented out my own version and ran with the default, which correctly
includes the mail server name in the dkim check but still has
UNPARSEABLE_RELAY.

I suppose it may have been attended to in a later version of either
postfix or spamassassin and that you have that version. Mine are

Postfix: mail_version = 3.1.0
Spamassassin: SpamAssassin version 3.4.2 running on Perl version 5.22.1

I'm currently studying the solution proposed by Jaroslaw Rafa. Once I
understand what is going on I will probably apply it and see what happens.
Reply | Threaded
Open this post in threaded view
|

Re: Cannot sign with DKIM on same-server web and mail

linkcheck
In reply to this post by Wietse Venema
 > Drop the quotes!

I just have whilst following advice from David Burgin - in fact I
commented out the macro entirely but still the same - dkim ok but
UNPARSEABLE_RELAY still present.

 > Where does Postfix documentation say that you need to use quoted strings?

It doesn't, I agree. And forgot it doesn't. Sorry.

I copied several lines from a posting in, I think, the spamassassin
forum but have seen quotes elsewhere plus the injunction to quote it if
it has spaces in the line. Most of what I copied seems to work but not
that part. :(

I have just run tests with the quotes removed on the line...
   milter_connect_macros=j {daemon_name} {daemon_addr} v _

... and it now seems to work. In fact, I initially forgot to remove the
opening quote and it still worked; odd.

I still get a single warning concerning macro "i" after a restart and a
handle_user warning "unable to find user" every email but I can live
with those.

Thanks for the help, Wietse, and to all others in this forum who helped
sort this problem. Initial tests show DKIM working correctly and
spamassassin running as a milter.
12