Feature Request: Allow Rejecting UTF BOM in MAIL FROM

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Zach Callear
My server has been receiving a lot of spam lately where the username
portion of the email address in the MAIL FROM command contains a UTF
byte-order mark (BOM).

With my configuration until this point:

1. Postfix does recipient verification with Dovecot LMTP, which doesn't
care about the BOM as the envelope sender isn't being checked at that point.
2. Postfix sends the SMTP session with the client, indicating success.
3. Postfix delivers the message via Dovecot LMTP, at which point Dovecot
rejects it due to the invalid envelope sender, resulting in Postfix
generating a bounce message.
4. The bounce message, being sent with a RCPT TO header which now
contains the BOM, generally fails to send, but can result in backscatter
if the receiving host strips the BOM or uses a catch-all forwarder.

I was able to solve this problem locally by using "check_sender_access"
in "smtpd_sender_restrictions", using a "pcre" lookup table, with a rule
of "/\xEF\xBB\xBF/ REJECT" (if this gets mangled, it's a pattern simply
consisting of the three hexadecimal escapes for the bytes composing the
BOM).

However, it would be nice if there was an easy way to detect and handle
this situation, maybe with something else which could be used in
"smtpd_sender_restrictions".

Example SMTP session script (I piped to telnet) to simulate this UTF BOM
MAIL FROM behavior:
---
#!/bin/bash
printf 'EHLO my.actual.host.name\n'
sleep 1
utf_bom=$(printf '%b' '\xEF\xBB\xBF')
printf 'MAIL FROM: <za'"$utf_bom"'[hidden email]>\n'
#printf 'MAIL FROM: <[hidden email]>\n'
sleep 1
printf 'RCPT TO: <[hidden email]>\n'
sleep 1
printf 'DATA\n'
cat <<BODY
Subject: Test Message
From: Zach Callear <[hidden email]>
To: Zach Callear <[hidden email]>

This is a test message.
.

BODY
---
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Wietse Venema
Zach Callear:
> My server has been receiving a lot of spam lately where the username
> portion of the email address in the MAIL FROM command contains a UTF
> byte-order mark (BOM).
>
> With my configuration until this point:
>
> 1. Postfix does recipient verification with Dovecot LMTP, which doesn't
> care about the BOM as the envelope sender isn't being checked at that point.

What is the difference with final delivery to Dovecot?

If Dovecot is not consistent in the way it accepts mail, then that
is the problem that needs to be addressed.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Zach Callear
Wietse Venema:
> What is the difference with final delivery to Dovecot?

There is nothing special about the fact that I'm doing final delivery to
Dovecot, but it is what made me notice the problem.  I just personally
consider it invalid for an envelope sender address to contain a UTF BOM,
and I wish Postfix would allow me to address that without resulting to
quite custom configuration with PCRE matching.  If it is actually
technically valid to have a UTF BOM right in the middle of the envelope
sender, then I agree that Postfix should continue behaving as it does,
but it still would be nice to have an easy configuration option to
affect that behavior, as I find it highly unlikely that there's a
legitimate use case for it.

It seems to me that Dovecot's behavior of not considering the envelope
sender during the recipient check is valid, as the envelope sender is
outside the scope of checking whether a recipient is valid, but it would
of course be very nice if Postfix could work together with Dovecot's
LMTP to do a more thorough check not only limited to the recipient;
maybe it can, as I'm no expert.

Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Viktor Dukhovni
> On Feb 11, 2019, at 2:16 PM, Zach Callear <[hidden email]> wrote:
>
> it still would be nice to have an easy configuration option to affect that behavior,
> as I find it highly unlikely that there's a legitimate use case for it.

Have you tried:

        strict_rfc821_envelopes = yes

?  There's nothing special about a "UTF-8 BOM" (there's no such thing in
UTF-8 actually UTF-8 has no little-endian form).  It is rather likely
that your sender was not using SMTPUTF8, and so the encoding of non-ASCII
envelopes is unspecified and use of such envelopes is invalid.  Indeed
your Dovecot LMTP service would likely equally object to any non-ASCII
envelope.

--
--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Zach Callear
Viktor Dukhovni:
> Have you tried:
> strict_rfc821_envelopes = yes
>
> ?  There's nothing special about a "UTF-8 BOM" (there's no such thing in
> UTF-8 actually UTF-8 has no little-endian form).  It is rather likely
> that your sender was not using SMTPUTF8, and so the encoding of non-ASCII
> envelopes is unspecified and use of such envelopes is invalid.  Indeed
> your Dovecot LMTP service would likely equally object to any non-ASCII
> envelope.

I just tested it.  With "strict_rfc821_envelopes = yes", and with a
blank "smtpd_helo_restrictions" setting, email sent with the example
SMTP script from my first message (wherein a UTF BOM is used in the
email address in the MAIL FROM command) is accepted by Postfix.

Regarding, "UTF-8 BOM", I intentionally have only said "UTF BOM" thus
far, because I understand what you're saying about UTF-8.

Indeed this behavior I'm talking about is without the client specifying
SMTPUTF8.

I don't know if this behavior is specific only to a BOM.  I just know
that in my case, a BOM is what has caused my problem (bouncebacks for
spam piling up in my queue, and backscatter).
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Viktor Dukhovni
On Mon, Feb 11, 2019 at 01:56:56PM -0700, Zach Callear wrote:

> Viktor Dukhovni:
> > Have you tried:
> > strict_rfc821_envelopes = yes
>
> I just tested it.  With "strict_rfc821_envelopes = yes", and with a
> blank "smtpd_helo_restrictions" setting, email sent with the example
> SMTP script from my first message (wherein a UTF BOM is used in the
> email address in the MAIL FROM command) is accepted by Postfix.

Indeed I forgot that this does not enforce an ASCII character-set:

    http://www.postfix.org/postconf.5.html#strict_rfc821_envelopes

However, right below that is:

    http://www.postfix.org/postconf.5.html#strict_smtputf8

which will do the job.

> Indeed this behavior I'm talking about is without the client specifying
> SMTPUTF8.

In which case it is not specifically the bytes you're reporting
that are problematic.  Likely any non-ascii envelope will fail
in exactly the same way.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Zach Callear
Viktor Dukhovni:
> However, right below that is:
>      http://www.postfix.org/postconf.5.html#strict_smtputf8
>
> which will do the job.

Thank you, Victor.  That setting indeed allows me to reject Unicode
characters (and not just BOMs) in a MAIL FROM command, when SMTPUTF8
isn't specified, as I initially requested.

However, unrelated then to Postfix behavior, it highlights the real root
of my problem, which isn't even solved by my PCRE check (unless I ban
all Unicode), and that's that Dovecot doesn't have proper SMTPUTF8
support.  Postfix verifies the RCPT TO address with Dovecot LMTP,
Postfix ends the session with the client, indicating success, Postfix
sends the message onto Dovecot LMTP, and Dovecot LMTP generates a
bounce.  The solution would then be to make Dovecot not bounce Unicode
MAIL FROM addresses, so I'll pursue that angle. Thanks again.
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Viktor Dukhovni
> On Feb 11, 2019, at 6:08 PM, Zach Callear <[hidden email]> wrote:
>
>> However, right below that is:
>>     http://www.postfix.org/postconf.5.html#strict_smtputf8
>>
>> which will do the job.
>
> Thank you, Victor.  That setting indeed allows me to reject Unicode characters (and not just BOMs) in a MAIL FROM command, when SMTPUTF8 isn't specified, as I initially requested.

The parameter should probably be mentioned in:

  http://www.postfix.org/SMTPUTF8_README.html#compatibility

under "Pre-existing non-ASCII email flows".  Anyone care to
to contribute a patch for proto/SMTPUTF8_README.html?

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Andrey Repin-2
Greetings, Viktor Dukhovni!

>>
>>> However, right below that is:
>>>     http://www.postfix.org/postconf.5.html#strict_smtputf8
>>>
>>> which will do the job.
>>
>> Thank you, Victor.  That setting indeed allows me to reject Unicode
>> characters (and not just BOMs) in a MAIL FROM command, when SMTPUTF8 isn't
>> specified, as I initially requested.

> The parameter should probably be mentioned in:

>   http://www.postfix.org/SMTPUTF8_README.html#compatibility

> under "Pre-existing non-ASCII email flows".  Anyone care to
> to contribute a patch for proto/SMTPUTF8_README.html?

If you point in the direction of a repository and hint on what you want to see
in it, I can try my hand.


--
With best regards,
Andrey Repin
Tuesday, February 12, 2019 17:37:28

Sorry for my terrible english...

Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Viktor Dukhovni
> On Feb 12, 2019, at 9:38 AM, Andrey Repin <[hidden email]> wrote:
>
> If you point in the direction of a repository and hint on what you want to see
> in it, I can try my hand.

The postfix source code is available from any of the various mirrors
listed at: http://www.postfix.org/download.html

I have a github repository with the Postfix source code, but it is
not the canonical source of truth, and is not actively monitored
for pull requests, feature requests, ... but you can check out
the code from there if you prefer:

        https://github.com/vdukhovni/postfix

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Andrey Repin-2
In reply to this post by Viktor Dukhovni
Greetings, Viktor Dukhovni!

> On Mon, Feb 11, 2019 at 01:56:56PM -0700, Zach Callear wrote:

>> Viktor Dukhovni:
>> > Have you tried:
>> >     strict_rfc821_envelopes = yes
>>
>> I just tested it.  With "strict_rfc821_envelopes = yes", and with a
>> blank "smtpd_helo_restrictions" setting, email sent with the example
>> SMTP script from my first message (wherein a UTF BOM is used in the
>> email address in the MAIL FROM command) is accepted by Postfix.

> Indeed I forgot that this does not enforce an ASCII character-set:

>     http://www.postfix.org/postconf.5.html#strict_rfc821_envelopes

> However, right below that is:

>     http://www.postfix.org/postconf.5.html#strict_smtputf8

> which will do the job.

Does it disable the mechanics outlined in "SMTPUTF8 autodetection"[1] ?
Or I'm grossly misunderstanding the implications?

>> Indeed this behavior I'm talking about is without the client specifying
>> SMTPUTF8.

> In which case it is not specifically the bytes you're reporting
> that are problematic.  Likely any non-ascii envelope will fail
> in exactly the same way.

[1] http://www.postfix.org/SMTPUTF8_README.html#detecting


--
With best regards,
Andrey Repin
Thursday, February 14, 2019 0:42:26

Sorry for my terrible english...
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Viktor Dukhovni
On Thu, Feb 14, 2019 at 12:45:37AM +0300, Andrey Repin wrote:

> > Indeed I forgot that this does not enforce an ASCII character-set:
> >
> >     http://www.postfix.org/postconf.5.html#strict_rfc821_envelopes
> >
> > However, right below that is:
> >
> >     http://www.postfix.org/postconf.5.html#strict_smtputf8
> >
> > which will do the job.
>
> Does it disable the mechanics outlined in "SMTPUTF8 autodetection"[1] ?
> Or I'm grossly misunderstanding the implications?

As documented, it only affects the processing of the message envelope,
(in the "MAIL FROM", "RCPT TO" and "VRFY" commands), which will reject
non-ascii input when SMTPUTF8 is not signalled by the client.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Wietse Venema
Viktor Dukhovni:

> On Thu, Feb 14, 2019 at 12:45:37AM +0300, Andrey Repin wrote:
>
> > > Indeed I forgot that this does not enforce an ASCII character-set:
> > >
> > >     http://www.postfix.org/postconf.5.html#strict_rfc821_envelopes
> > >
> > > However, right below that is:
> > >
> > >     http://www.postfix.org/postconf.5.html#strict_smtputf8
> > >
> > > which will do the job.
> >
> > Does it disable the mechanics outlined in "SMTPUTF8 autodetection"[1] ?
> > Or I'm grossly misunderstanding the implications?
>
> As documented, it only affects the processing of the message envelope,
> (in the "MAIL FROM", "RCPT TO" and "VRFY" commands), which will reject
> non-ascii input when SMTPUTF8 is not signalled by the client.

And perhaps surprisingly, that is what the code does. This setting
is used only while parsing SMTP commands, in the SMTP daemon.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Andrey Repin-2
Greetings, Wietse Venema!

> Viktor Dukhovni:
>> On Thu, Feb 14, 2019 at 12:45:37AM +0300, Andrey Repin wrote:
>>
>> > > Indeed I forgot that this does not enforce an ASCII character-set:
>> > >
>> > >     http://www.postfix.org/postconf.5.html#strict_rfc821_envelopes
>> > >
>> > > However, right below that is:
>> > >
>> > >     http://www.postfix.org/postconf.5.html#strict_smtputf8
>> > >
>> > > which will do the job.
>> >
>> > Does it disable the mechanics outlined in "SMTPUTF8 autodetection"[1] ?
>> > Or I'm grossly misunderstanding the implications?
>>
>> As documented, it only affects the processing of the message envelope,
>> (in the "MAIL FROM", "RCPT TO" and "VRFY" commands), which will reject
>> non-ascii input when SMTPUTF8 is not signalled by the client.

> And perhaps surprisingly, that is what the code does. This setting
> is used only while parsing SMTP commands, in the SMTP daemon.

Makes sense, thank you.

So, next question is, do you want it to be mentioned in "Enabling Postfix
SMTPUTF8 support" [2] or separately?

[2] http://www.postfix.org/SMTPUTF8_README.html#enabling


--
With best regards,
Andrey Repin
Thursday, February 14, 2019 3:46:45

Sorry for my terrible english...

Reply | Threaded
Open this post in threaded view
|

Re: Feature Request: Allow Rejecting UTF BOM in MAIL FROM

Viktor Dukhovni
On Thu, Feb 14, 2019 at 03:48:45AM +0300, Andrey Repin wrote:

> Makes sense, thank you.
>
> So, next question is, do you want it to be mentioned in "Enabling Postfix
> SMTPUTF8 support" [2] or separately?
>
> [2] http://www.postfix.org/SMTPUTF8_README.html#enabling

My guess would be under:

   http://www.postfix.org/SMTPUTF8_README.html#compatibility

in the un-anchored "Pre-existing non-ASCII email flows" section.

--
        Viktor.