On DKIM and Content-Transfer-Encoding

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

On DKIM and Content-Transfer-Encoding

Mauricio Tavares
      This email is mostly about understanding what is going on, how
the process works than whether my install of postfix is working
properly or the MUA is lying. In fact, I am using ssmtp to send the
email to postfix because I want to make my test as simple as possible
(avoid "helpful" MUAs adding stuff to the email). So, let's say I
create simple email:

raub@desktop:~$ cat 8bit_test
Subject: 8bit test
Content-Transfer-Encoding: 8bit

Text with umlauts here

raub@desktop:~$

It only has two headers (the From: header is added when I actually
send email) and a single line in the body, which has a few 8bit
characters. I am using ssmtp to send the above email to my postfix
server,

ssmtp -v [hidden email] < 8bit_test

which is setup to do DKIM and announces that supports the following

250-PIPELINING
250-SIZE
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

What I have noticed is that it will fail DKIM, and then will change
the Content-Transfer-Encoding: header to quoted-printable. Now, if I
submit the same email setting

Content-Transfer-Encoding: quoted-printable

it passes DKIM test. So, what is the 8bit encoding here anyway and how
it compared to quoted-printable? From
http://tools.ietf.org/html/rfc2045#page-14, it seems that
quoted-printable converts the body to something 7bit can deal with (or
at least the MTA will see that header and do the convertion). What
about 8bit? Is it not translated? Or, is it expected to be fed in some
kind of (mime) encoded format (hence the 8bitmime) so the MTA does not
touch it (hence the mime processing controls mentioned in
http://www.postfix.org/cleanup.8.html)?
Reply | Threaded
Open this post in threaded view
|

Re: On DKIM and Content-Transfer-Encoding

Wietse Venema
Mauricio Tavares:

>       This email is mostly about understanding what is going on, how
> the process works than whether my install of postfix is working
> properly or the MUA is lying. In fact, I am using ssmtp to send the
> email to postfix because I want to make my test as simple as possible
> (avoid "helpful" MUAs adding stuff to the email). So, let's say I
> create simple email:
>
> raub@desktop:~$ cat 8bit_test
> Subject: 8bit test
> Content-Transfer-Encoding: 8bit
>
> Text with umlauts here

This email requires a MIME-Version header. Otherwise it is
not valid MIME.

> raub@desktop:~$
>
> It only has two headers (the From: header is added when I actually
> send email) and a single line in the body, which has a few 8bit
> characters. I am using ssmtp to send the above email to my postfix
> server,
>
> ssmtp -v [hidden email] < 8bit_test
>
> which is setup to do DKIM and announces that supports the following
>
> 250-PIPELINING
> 250-SIZE
> 250-ETRN
> 250-STARTTLS
> 250-ENHANCEDSTATUSCODES
> 250-8BITMIME
> 250 DSN

If the client sends 8BITMIME, then it MUST issue the 8BITMIME keyword
on the MAIL FROM command line. Otherwise, behavior is undefined
(i.e. different systems may handle this in different ways).

> What I have noticed is that it will fail DKIM, and then will change
> the Content-Transfer-Encoding: header to quoted-printable. Now, if I

As required by the MIME RFCs, an MTA must either bounce mail or
convert it to quoted-printable when it needs to deliver 8BITMIME
mail to an SMTP server that does not announce 8BITMIME support.

DKIM signatures of 8BITMIME mail may break unless all SMTP servers
in the path implement and announce 8BITMIME support. Otherwise, it
is better to down-convert to quoted-printable before DKIM signing.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: On DKIM and Content-Transfer-Encoding

Mauricio Tavares
On Tue, Jun 24, 2014 at 10:58 AM, Wietse Venema <[hidden email]> wrote:

> Mauricio Tavares:
>>       This email is mostly about understanding what is going on, how
>> the process works than whether my install of postfix is working
>> properly or the MUA is lying. In fact, I am using ssmtp to send the
>> email to postfix because I want to make my test as simple as possible
>> (avoid "helpful" MUAs adding stuff to the email). So, let's say I
>> create simple email:
>>
>> raub@desktop:~$ cat 8bit_test
>> Subject: 8bit test
>> Content-Transfer-Encoding: 8bit
>>
>> Text with umlauts here
>
> This email requires a MIME-Version header. Otherwise it is
> not valid MIME.
>
      Thanks for the info. I then added MIME-Version:

raub@desktop:~$ cat 8bit_test
Subject: 8bit test
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Italienisches Olivenöl

raub@desktop:~$

and tried again using ssmtp. It still fails.

>>
>> It only has two headers (the From: header is added when I actually
>> send email) and a single line in the body, which has a few 8bit
>> characters. I am using ssmtp to send the above email to my postfix
>> server,
>>
>> ssmtp -v [hidden email] < 8bit_test
>>
>> which is setup to do DKIM and announces that supports the following
>>
>> 250-PIPELINING
>> 250-SIZE
>> 250-ETRN
>> 250-STARTTLS
>> 250-ENHANCEDSTATUSCODES
>> 250-8BITMIME
>> 250 DSN
>
> If the client sends 8BITMIME, then it MUST issue the 8BITMIME keyword
> on the MAIL FROM command line. Otherwise, behavior is undefined
> (i.e. different systems may handle this in different ways).

       Let me use an example to see if I understand what you are
saying: here is the transaction when I sent that email using ssmtp
(which is talking to mail.domain.com, which is using postfix):

 ssmtp -v [hidden email] < 8bit_test.bare
[<-] 220 mail.domain.com Test Mail Server
[->] HELO raub.internal.domain.com
[<-] 250 mail.domain.com
[->] MAIL FROM:<[hidden email]>
[<-] 250 2.1.0 Ok
[->] RCPT TO:<[hidden email]>
[<-] 250 2.1.5 Ok
[->] DATA
[<-] 354 End data with <CR><LF>.<CR><LF>
[->] Received: by desktop.internal.domain.com (sSMTP sendmail
emulation); Tue, 24 Jun 2014 17:05:06 -0400
[->] From: "Mauricio Tavares" <[hidden email]>
[->] Date: Tue, 24 Jun 2014 17:05:06 -0400
[->] Subject: 8bit test - bare
[->] MIME-Version: 1.0
[->] Content-Transfer-Encoding: 8bit
[->] Content-Type: text/plain; charset=ISO-8859-1; format=flowed
[->]
[->] Italienisches Oliven�l
[->]
[->] Same text
[->]
[->] .
[<-] 250 2.0.0 Ok: queued as CC9FE80042
[->] QUIT
[<-] 221 2.0.0 Bye

the proper MAIL FROM command line should be
(http://tools.ietf.org/html/rfc6152):

MAIL FROM:<[hidden email]> BODY=8BITMIME

If that is the case, I think that explains issues I observed with a
few email clients, notably thunderbird. Instead of posting the problem
I saw, I thought it would make sense for me to understand how this is
supposed to work first. Now I can rerun thunderbird in debug mode and
see if it is sending a proper MAIL FROM command line stating to use
8BITMIME.

>
>> What I have noticed is that it will fail DKIM, and then will change
>> the Content-Transfer-Encoding: header to quoted-printable. Now, if I
>
> As required by the MIME RFCs, an MTA must either bounce mail or
> convert it to quoted-printable when it needs to deliver 8BITMIME
> mail to an SMTP server that does not announce 8BITMIME support.
>
> DKIM signatures of 8BITMIME mail may break unless all SMTP servers
> in the path implement and announce 8BITMIME support. Otherwise, it
> is better to down-convert to quoted-printable before DKIM signing.
>
      Can I down-convert the email in postfix before DKIM signing? I
believe some email clients using this server are sending 8BITMIME
emails without a properly created MAIL FROM command line. Since I
cannot do the right thing and correct that at the MUA side, I would
like to do the next best thing.

>         Wietse
Reply | Threaded
Open this post in threaded view
|

Re: On DKIM and Content-Transfer-Encoding

A. Schulze
Mauricio Tavares:
> Content-Transfer-Encoding: 8bit
>
> Italienisches Olivenöl
depending on your shell it's possible the 'ö' is encoded as 2 byte in UTF-8.
so you may need a charset declaration, too.

does your test pass if you simply replace ö by oe ?
that way you may check if you test the right way at all.

Andreas

Reply | Threaded
Open this post in threaded view
|

Re: On DKIM and Content-Transfer-Encoding

Wietse Venema
In reply to this post by Mauricio Tavares
Mauricio Tavares:
> [<-] 220 mail.domain.com Test Mail Server
> [->] HELO raub.internal.domain.com
> [<-] 250 mail.domain.com
> [->] MAIL FROM:<[hidden email]>

You send HELO. That means you can only send 7-bit ASCII email.
Please read RFC 5321 for the 7-bit requirement of SMTP.

In order to send 8BIT mail over SMTP, the client must announce that
it supports ESMTP, the server must announce that it supports 8BITMIME,
and the client must issue the 8BITMIME parameter in the MAIL FROM
command.

Please read RFC 1869 for how to negotiate ESMTP.
Please read RFC 1652 for how to negotiate 8BITMIME.

> > As required by the MIME RFCs, an MTA must either bounce mail or
> > convert it to quoted-printable when it needs to deliver 8BITMIME
> > mail to an SMTP server that does not announce 8BITMIME support.
> >
> > DKIM signatures of 8BITMIME mail may break unless all SMTP servers
> > in the path implement and announce 8BITMIME support. Otherwise, it
> > is better to down-convert to quoted-printable before DKIM signing.
> >
>       Can I down-convert the email in postfix before DKIM signing? I
> believe some email clients using this server are sending 8BITMIME
> emails without a properly created MAIL FROM command line. Since I
> cannot do the right thing and correct that at the MUA side, I would
> like to do the next best thing.

You can force downconversion with a null content filter and by
suppressing or ignoring the 8BITMIME server announcement.

Untested example:

/etc/postfix/master.cf:
    smtp .. .. .. .. .. .. .. smtpd
        -o content_filter=dummy:127.0.0.1:12345

    dummy .. .. .. .. .. .. .. smtp
        -o smtp_discard_ehlo_keywords=8bitmime,silent-discard

    127.0.0.1:12345 .. .. .. .. .. .. .. smtpd
        -o smtpd_authorized_xforward_hosts=127.0.0.1
        -o smtpd_client_restrictions=
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_relay_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject

Once mail is received with the 127.0.0.1:12345 SMTP server, it will
have been down-converted, provided that it was formatted properly,
and that the proper ESMTP commands were used.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: On DKIM and Content-Transfer-Encoding

Mauricio Tavares
On Sun, Jun 29, 2014 at 8:20 AM, Wietse Venema <[hidden email]> wrote:

> Mauricio Tavares:
>> [<-] 220 mail.domain.com Test Mail Server
>> [->] HELO raub.internal.domain.com
>> [<-] 250 mail.domain.com
>> [->] MAIL FROM:<[hidden email]>
>
> You send HELO. That means you can only send 7-bit ASCII email.
> Please read RFC 5321 for the 7-bit requirement of SMTP.
>
> In order to send 8BIT mail over SMTP, the client must announce that
> it supports ESMTP, the server must announce that it supports 8BITMIME,
> and the client must issue the 8BITMIME parameter in the MAIL FROM
> command.
>
> Please read RFC 1869 for how to negotiate ESMTP.
> Please read RFC 1652 for how to negotiate 8BITMIME.
>
      So, based on those two RFCs, the transaction should look something like:

raub@desktop:/tmp$ nc -t mail.domain.com 25
220 mail.domain.com Test Mail Server
EHLO desktop.domain.com
250-mail.domain.com
250-PIPELINING
250-SIZE
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM:<[hidden email]> BODY=8BITMIME
250 2.1.0 Ok
RCPT TO:<[hidden email]>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
From: "Mauricio Tavares" <[hidden email]>
Subject: 8bit test - manual6
Date: Mon, 30 Jun 2014 11:10:05 -0400
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Olivenöl

.
250 2.0.0 Ok: queued as 2BA1F80041
QUIT
221 2.0.0 Bye
raub@desktop:/tmp$

or am I missing something?

>> > As required by the MIME RFCs, an MTA must either bounce mail or
>> > convert it to quoted-printable when it needs to deliver 8BITMIME
>> > mail to an SMTP server that does not announce 8BITMIME support.
>> >
>> > DKIM signatures of 8BITMIME mail may break unless all SMTP servers
>> > in the path implement and announce 8BITMIME support. Otherwise, it
>> > is better to down-convert to quoted-printable before DKIM signing.
>> >
>>       Can I down-convert the email in postfix before DKIM signing? I
>> believe some email clients using this server are sending 8BITMIME
>> emails without a properly created MAIL FROM command line. Since I
>> cannot do the right thing and correct that at the MUA side, I would
>> like to do the next best thing.
>
> You can force downconversion with a null content filter and by
> suppressing or ignoring the 8BITMIME server announcement.
>
> Untested example:
>
> /etc/postfix/master.cf:
>     smtp .. .. .. .. .. .. .. smtpd
>         -o content_filter=dummy:127.0.0.1:12345
>
>     dummy .. .. .. .. .. .. .. smtp
>         -o smtp_discard_ehlo_keywords=8bitmime,silent-discard
>
>     127.0.0.1:12345 .. .. .. .. .. .. .. smtpd
>         -o smtpd_authorized_xforward_hosts=127.0.0.1
>         -o smtpd_client_restrictions=
>         -o smtpd_helo_restrictions=
>         -o smtpd_sender_restrictions=
>         -o smtpd_relay_restrictions=
>         -o smtpd_recipient_restrictions=permit_mynetworks,reject
>
> Once mail is received with the 127.0.0.1:12345 SMTP server, it will
> have been down-converted, provided that it was formatted properly,
> and that the proper ESMTP commands were used.
>
>         Wietse
Reply | Threaded
Open this post in threaded view
|

Re: On DKIM and Content-Transfer-Encoding

Wietse Venema
Mauricio Tavares:

> raub@desktop:/tmp$ nc -t mail.domain.com 25
> 220 mail.domain.com Test Mail Server
> EHLO desktop.domain.com
> 250-mail.domain.com
> 250-PIPELINING
> 250-SIZE
> 250-ETRN
> 250-STARTTLS
> 250-ENHANCEDSTATUSCODES
> 250-8BITMIME
> 250 DSN
> MAIL FROM:<[hidden email]> BODY=8BITMIME
> 250 2.1.0 Ok
> RCPT TO:<[hidden email]>
> 250 2.1.5 Ok
> DATA
> 354 End data with <CR><LF>.<CR><LF>
> From: "Mauricio Tavares" <[hidden email]>
> Subject: 8bit test - manual6
> Date: Mon, 30 Jun 2014 11:10:05 -0400
> MIME-Version: 1.0
> Content-Transfer-Encoding: 8bit
>
> Oliven?l
>
> .
> 250 2.0.0 Ok: queued as 2BA1F80041
> QUIT
> 221 2.0.0 Bye
> raub@desktop:/tmp$
>
> or am I missing something?

That's correct (except I can't tell if nc sends lines ending with
bare <LF>, or if it sends lines ending with <CR><LF> as required
by SMTP; Postfix will accept either so it does not matter here).

Now, you can try to use the null-filter trick and see what it
looks like after 8-to-7 downconversion.

BTW domain.com is a real site; use example.com, example.net etc.
for examples. There is an RFC on that topic.

        Wietse