Milter header position semantincs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Milter header position semantincs

Wietse Venema
(This message is mostly of interest to Claus Assmann)

This week I fixed a problem or PREPEND actions in access maps or
policy server reponses that was giving problems with DMARC setups
that combine of SPF policy server with a DKIM Milter.

Meanwhile, Christian R??ner has pointed out that there still is a
different problem with header insert requests, this time in Milter
responses.

Here is a quote from the Milter API documentation, on-line at
https://www.milter.org/developers/api/smfi_insheader

int smfi_insheader(
    SMFICTX *ctx,
    int hdridx, /* header position */
    char *headerf, /* header label */
    char *headerv /* header value */
    );

  * A filter will receive only headers that have been sent by the
    SMTP client and those header modifications by earlier filters.
    It will not receive the headers that are inserted by sendmail
    itself.  This makes the header insertion position highly dependent
    on the headers that exist in the incoming message and those
    that are configured to be added by sendmail. For example,
    sendmail will always add a Received: header to the beginning
    of the headers.  Setting hdridx to 0 will actually insert the
    header before this Received: header. However, later filters can
    be easily confused as they receive the added header, but not
    the Received: header, thus making it hard to insert a header
    at a fixed position.

Thus, the first Milter in a list of Milters sees a message header
like this:

    Headers from SMTP client

When the first Milter issues header insert requests with index 0,
the resulting message looks like this with Postfix and Sendmail:

    Header2 inserted with index 0   (internal header index 0)
    Header1 inserted with index 0   (internal header index 1)
    Received: header from receiving MTA (internal header index 2)
    Headers from SMTP client        (internal header index 3...)

With Sendmail, the second etc. Milters see the following header:

    Header2 inserted with index 0   (internal header index 0)
    Header1 inserted with index 0   (internal header index 1)
    Headers from SMTP client        (internal header index 3...)

While with Postfix they see something different:

    Header1 inserted with index 0
    Received: header from receiving MTA
    Headers from SMTP client

A naive workaround is to put "index 0" headers under Postfix's own
Received header. Then, Milters see the same headers as they would
see with Sendmail. But the problem is that the internal header
indices would differ from those with Sendmail:

    Header2 inserted with index 0 (internal header index 1)
    Header1 inserted with index 0 (internal header index 2)
    Headers from SMTP client      (internal header index 3...)

These header index differences complicate life for Milter users.

To avoid this difference with Sendmail, Postfix would have to
implement the same behavior as Sendmail: ignore the MTA's own
received header when reporting headers to Milters, but don't ignore
the MTA's own received header when receiving Milter requests with
a header index. That is, ignore the header based on its name, not
on its position.

If we do that, then we can also roll back this week's patch that
placed access/policy PREPENDed headers after the MTA's own received
header.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Milter header position semantincs

Robert Schetterer-2
Am 17.10.2014 um 16:48 schrieb Wietse Venema:

> (This message is mostly of interest to Claus Assmann)
>
> This week I fixed a problem or PREPEND actions in access maps or
> policy server reponses that was giving problems with DMARC setups
> that combine of SPF policy server with a DKIM Milter.
>
> Meanwhile, Christian R??ner has pointed out that there still is a
> different problem with header insert requests, this time in Milter
> responses.
>
> Here is a quote from the Milter API documentation, on-line at
> https://www.milter.org/developers/api/smfi_insheader
>
> int smfi_insheader(
>     SMFICTX *ctx,
>     int hdridx, /* header position */
>     char *headerf, /* header label */
>     char *headerv /* header value */
>     );
>
>   * A filter will receive only headers that have been sent by the
>     SMTP client and those header modifications by earlier filters.
>     It will not receive the headers that are inserted by sendmail
>     itself.  This makes the header insertion position highly dependent
>     on the headers that exist in the incoming message and those
>     that are configured to be added by sendmail. For example,
>     sendmail will always add a Received: header to the beginning
>     of the headers.  Setting hdridx to 0 will actually insert the
>     header before this Received: header. However, later filters can
>     be easily confused as they receive the added header, but not
>     the Received: header, thus making it hard to insert a header
>     at a fixed position.
>
> Thus, the first Milter in a list of Milters sees a message header
> like this:
>
>     Headers from SMTP client
>
> When the first Milter issues header insert requests with index 0,
> the resulting message looks like this with Postfix and Sendmail:
>
>     Header2 inserted with index 0   (internal header index 0)
>     Header1 inserted with index 0   (internal header index 1)
>     Received: header from receiving MTA (internal header index 2)
>     Headers from SMTP client        (internal header index 3...)
>
> With Sendmail, the second etc. Milters see the following header:
>
>     Header2 inserted with index 0   (internal header index 0)
>     Header1 inserted with index 0   (internal header index 1)
>     Headers from SMTP client        (internal header index 3...)
>
> While with Postfix they see something different:
>
>     Header1 inserted with index 0
>     Received: header from receiving MTA
>     Headers from SMTP client
>
> A naive workaround is to put "index 0" headers under Postfix's own
> Received header. Then, Milters see the same headers as they would
> see with Sendmail. But the problem is that the internal header
> indices would differ from those with Sendmail:
>
>     Header2 inserted with index 0 (internal header index 1)
>     Header1 inserted with index 0 (internal header index 2)
>     Headers from SMTP client      (internal header index 3...)
>
> These header index differences complicate life for Milter users.
>
> To avoid this difference with Sendmail, Postfix would have to
> implement the same behavior as Sendmail: ignore the MTA's own
> received header when reporting headers to Milters, but don't ignore
> the MTA's own received header when receiving Milter requests with
> a header index. That is, ignore the header based on its name, not
> on its position.
>
> If we do that, then we can also roll back this week's patch that
> placed access/policy PREPENDed headers after the MTA's own received
> header.
>
>         Wietse
>

Thank you Wietse  Christian and Claus for loooking to a code solution at
that milter stuff !!!


Best Regards
MfG Robert Schetterer

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
Reply | Threaded
Open this post in threaded view
|

PATCH: Milter header position semantics

Wietse Venema
In reply to this post by Wietse Venema
Wietse Venema:

> This week I fixed a problem or PREPEND actions in access maps or
> policy server reponses that was giving problems with DMARC setups
> that combine of SPF policy server with a DKIM Milter.
>
> Meanwhile, Christian R??ner has pointed out that there still is a
> different problem with [visibility of headers inserted with Milter
> requests]
>
> To avoid [incompatibility] with Sendmail, Postfix would have to
> implement the same behavior as Sendmail: ignore the MTA's own
> received header when reporting headers to Milters, but don't ignore
> the MTA's own received header when receiving Milter requests [...].

I have patches for evaluation:

Postfix 2.12 released 20140918 or later:

ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.patch
ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.patch.asc
ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.patch.sig

Postfix 2.8, 2.9. 2.10, 2.11, and Postfix 2.12 released before 20140918:

ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.11.patch
ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.11.patch.asc
ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.11.patch.sig

This just shows how little the Postfix Milter implementation has
changed over time.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: PATCH: Milter header position semantics

A. Schulze

wietse:

> I have patches for evaluation:
> Postfix 2.12 released 20140918 or later:
just compiling ...

> Postfix 2.8, 2.9. 2.10, 2.11, and Postfix 2.12 released before 20140918:
> ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.11.patch
this link is broken

BTW:
I played with SMTPUTF8. Debian Wheeze is the minimum Debian version required.
Maybe that should/could be noted on  
http://www.postfix.org/SMTPUTF8_README.html#building

Andreas

Reply | Threaded
Open this post in threaded view
|

Re: PATCH: Milter header position semantics

Wietse Venema
A. Schulze:

>
> wietse:
>
> > I have patches for evaluation:
> > Postfix 2.12 released 20140918 or later:
> just compiling ...
>
> > Postfix 2.8, 2.9. 2.10, 2.11, and Postfix 2.12 released before 20140918:
> > ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/feature-patches/20141017-milter-auto-header-hide-2.12.11.patch
> this link is broken

Should be 2.11.2-patch.

        Wietse
> BTW:
> I played with SMTPUTF8. Debian Wheeze is the minimum Debian version required.
> Maybe that should/could be noted on  
> http://www.postfix.org/SMTPUTF8_README.html#building
>
> Andreas
>
>
Reply | Threaded
Open this post in threaded view
|

Re: PATCH: Milter header position semantics

A. Schulze
In reply to this post by Wietse Venema
Wietse Venema:
> > To avoid [incompatibility] with Sendmail, Postfix would have to
> > implement the same behavior as Sendmail: ignore the MTA's own
> > received header when reporting headers to Milters, but don't ignore
> > the MTA's own received header when receiving Milter requests [...].
>
> I have patches for evaluation:
>
> Postfix 2.12 released 20140918 or later:

no difference to 2.12-20141011

the milterchain "spf-milter, amavisd-milter, opendkim + opendmarc" produce
the same header chain:

...
Authentication-Results: from opendmarc-milter
Authentication-Results: from opendkim-milter
X-Spam-foo: added by amavisd-milter from amavisd + spamassassin
Received: from MTA
Authentication-Results: from spf-milter
...

in short: no problem here (as before ...)

Andreas
Reply | Threaded
Open this post in threaded view
|

Re: PATCH: Milter header position semantics

Wietse Venema
Andreas Schulze:

> Wietse Venema:
> > > To avoid [incompatibility] with Sendmail, Postfix would have to
> > > implement the same behavior as Sendmail: ignore the MTA's own
> > > received header when reporting headers to Milters, but don't ignore
> > > the MTA's own received header when receiving Milter requests [...].
> >
> > I have patches for evaluation:
> >
> > Postfix 2.12 released 20140918 or later:
>
> no difference to 2.12-20141011
>
> the milterchain "spf-milter, amavisd-milter, opendkim + opendmarc" produce
> the same header chain:
>
> ...
> Authentication-Results: from opendmarc-milter
> Authentication-Results: from opendkim-milter
> X-Spam-foo: added by amavisd-milter from amavisd + spamassassin
> Received: from MTA
> Authentication-Results: from spf-milter
> ...
>
> in short: no problem here (as before ...)

I changed the algorithm that decides what headers are VISIBLE to a
Milter.  I did not change the header ORDER.

        Wietse