SMTPUTF8 problem with Exchange servers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

SMTPUTF8 problem with Exchange servers

Patrick Proniewski
Hello,

I have at work a Postfix infrastructure that sits between Internet and our Exchange servers. Postfix is used for MX and SMTP roles, ensure filtering with Amavisd/Clamav/etc.
For some time now I notice that some messages, either originating from Internet or from internal servers are bounced when they arrive on the last hop: Exchange.

Typical Inbound email will go through:

MX (postfix 3.4.x + Amavisd-new) -> MAILGW (postfix 3.3.x) -> Exchange farm

Typical Internal email will go through:

SMTP (postfix 3.3.x + Amavisd-new) -> MAILGW (postfix 3.3.x) -> Exchange farm

Sample log:

Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256: to=<[hidden email]>, orig_to=<userpart@domain>, relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0, dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by host Exchange-VIP[Exchange-VIP])

The MAILGW instance uses these parameters :

$ postconf -nfc /usr/local/etc/postfix-mailgw
authorized_submit_users =
command_directory = /usr/local/sbin
compatibility_level = 2
config_directory = /usr/local/etc/postfix-mailgw
daemon_directory = /usr/local/libexec/postfix
data_directory = /var/db/postfix-mailgw
debug_peer_level = 2
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd
    $daemon_directory/$process_name $process_id & sleep 5
empty_address_recipient = MAILER-DAEMON
header_checks = regexp:/usr/local/etc/postfix-mailgw/header_checks
html_directory = /usr/local/share/doc/postfix
inet_interfaces = all
inet_protocols = ipv4
local_recipient_maps =
local_transport = error:local mail delivery is disabled
mail_owner = postfix
mailbox_size_limit = 1000000000
mailq_path = /usr/local/bin/mailq
manpage_directory = /usr/local/man
master_service_disable =
message_size_limit = 21000000
multi_instance_enable = yes
multi_instance_name = postfix-mailgw
mydestination =
myhostname = mailgw.domain
mynetworks_style = host
myorigin = $myhostname
newaliases_path = /usr/local/bin/newaliases
parent_domain_matches_subdomains =
queue_directory = /var/spool/postfix-mailgw
readme_directory = /usr/local/share/doc/postfix
recipient_bcc_maps = hash:/usr/local/etc/postfix-mailgw/recipient_bcc
relay_domains = (many domains)
sample_directory = /usr/local/etc/postfix
sendmail_path = /usr/local/sbin/sendmail
setgid_group = maildrop
smtp_connection_cache_on_demand = no
smtp_destination_concurrency_limit = 50
smtp_destination_recipient_limit = 15
smtpd_banner = $myhostname ESMTP
smtpd_client_restrictions = check_client_access
    hash:/usr/local/etc/postfix-mailgw/client_access reject
smtpd_recipient_restrictions = reject_unauth_destination
smtpd_timeout = 300
smtputf8_autodetect_classes = verify
transport_maps = hash:/usr/local/etc/postfix-mailgw/transport
unknown_local_recipient_reject_code = 450
virtual_alias_domains = hugo.domain domain
virtual_alias_maps = hash:/usr/local/etc/postfix-mailgw/virtual_alias


I'm suspecting something fishy with Amavisd, but I can totally rule out a mis-setting on my Postfix servers.

Any help appreciated,
patpro

Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Matus UHLAR - fantomas
On 17.06.20 14:37, Patrick Proniewski wrote:
>I have at work a Postfix infrastructure that sits between Internet and our Exchange servers. Postfix is used for MX and SMTP roles, ensure filtering with Amavisd/Clamav/etc.
>For some time now I notice that some messages, either originating from Internet or from internal servers are bounced when they arrive on the last hop: Exchange.

AFAIK this happens with messages that contain 8-bit headers.

>Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256:
> to=<[hidden email]>, orig_to=<userpart@domain>,
> relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0,
> dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by
> host Exchange-VIP[Exchange-VIP])

I have this problem too a few times a day.

>I'm suspecting something fishy with Amavisd, but I can totally rule out a mis-setting on my Postfix servers.

amavisd is outta here.

apparently it _could_ reformat 8-bit headers but I'm not entirely sure if it
should and if it supports that.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Microsoft dick is soft to do no harm
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Patrick Proniewski
Hi,


> On 17 juin 2020, at 15:08, Matus UHLAR - fantomas <[hidden email]> wrote:
>
> On 17.06.20 14:37, Patrick Proniewski wrote:
>> I have at work a Postfix infrastructure that sits between Internet and our Exchange servers. Postfix is used for MX and SMTP roles, ensure filtering with Amavisd/Clamav/etc.
>> For some time now I notice that some messages, either originating from Internet or from internal servers are bounced when they arrive on the last hop: Exchange.
>
> AFAIK this happens with messages that contain 8-bit headers.
>
>> Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256:
>> to=<[hidden email]>, orig_to=<userpart@domain>,
>> relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0,
>> dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by
>> host Exchange-VIP[Exchange-VIP])
>
> I have this problem too a few times a day.


It becomes quite frequent these days unfortunately, and we've experienced it with internal emails too which is more problematic. Currently I count about 150 to 180 bounces per week.


>
>> I'm suspecting something fishy with Amavisd, but I can totally rule out a mis-setting on my Postfix servers.
>
> amavisd is outta here.
>
> apparently it _could_ reformat 8-bit headers but I'm not entirely sure if it
> should and if it supports that.


On our MX servers I've setup a partial debug (log debug only for some particular recipient) in order to gather stats like DNS timing during analysis, etc. It allowed me to see this kind of things:

Jun 11 08:13:23 amavis[82395]: (82395-16) orcpt_encode rfc822, foo@domain, smtputf8
Jun 11 08:13:23 amavis[82395]: (82395-16) orcpt_encode rfc822, bar@domain, smtputf8

I don't know if it's meaningful or not (those particular messages won't bounce), and it might just as well be a notice because the current Postfix involved supports smtputf8.

regards,
patpro
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Bastian Blank-3
In reply to this post by Patrick Proniewski
On Wed, Jun 17, 2020 at 02:37:23PM +0200, Patrick Proniewski wrote:
> For some time now I notice that some messages, either originating from Internet or from internal servers are bounced when they arrive on the last hop: Exchange.
> Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256: to=<[hidden email]>, orig_to=<userpart@domain>, relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0, dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by host Exchange-VIP[Exchange-VIP])

You have the two options to
- make Exchange support SMTPUTF8 or
- disable SMTPUTF8 in Postfix.

Set "smtputf8_enable = no".

Bastian

--
        "Get back to your stations!"
        "We're beaming down to the planet, sir."
                -- Kirk and Mr. Leslie, "This Side of Paradise",
                   stardate 3417.3
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Wietse Venema
In reply to this post by Patrick Proniewski
Patrick Proniewski:
> Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256: to=<[hidden email]>, orig_to=<userpart@domain>, relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0, dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by host Exchange-VIP[Exchange-VIP])
>

It is required (in this case, by RFC 6531..6533).

With "smtputf8_enable = yes" (the default) Postfix must return mail
as undeliverable when:

- a sending SMTP client requested SMTPUTF8 in the MAIL FROM command(*),

- and a receiving SMTP server does not announce SMTPUTF8 support.

However, Postfix ignores this requirement when a message has no
UTF8 in the header, sender, or recipient envelope address.

Options:

- SMTPUTF8 support in Microsoft Exchange Server protocol revision
  14.0 (from 2018) or later.
  https://en.wikipedia.org/wiki/Extended_SMTP#SMTPUTF8

- Postfix: set "smtputf8_enable = no" so that Postfix will not be
  able to receive such email messages (i.e. they will be returned
  to the sender before they reach Postfix).

There is no defined conversion from SMTPUTF8 mail to historical
email, and I decided not to invent one because it would solve only
a temporary problem.

> smtputf8_autodetect_classes = verify

(*) smtputf8_autodetect_classes enables SMTPUTF8 autodetection for
the smtpd service. this 'detects' UTF8 in MAIL FROM, RCPT TO, and
in the header. But you don't have it enabled for any Postfix service
that receives email, therefore smtputf8_autodetect_classes was not
part of your problem.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Patrick Proniewski
In reply to this post by Bastian Blank-3
Hi,

> On 17 juin 2020, at 15:42, Bastian Blank <bastian+postfix-users=[hidden email]> wrote:
>
> On Wed, Jun 17, 2020 at 02:37:23PM +0200, Patrick Proniewski wrote:
>> For some time now I notice that some messages, either originating from Internet or from internal servers are bounced when they arrive on the last hop: Exchange.
>> Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256: to=<[hidden email]>, orig_to=<userpart@domain>, relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0, dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by host Exchange-VIP[Exchange-VIP])
>
> You have the two options to
> - make Exchange support SMTPUTF8 or


Not possible yet. A flag exists for Exchange 2019 but we are running 2016 now and upgrade is not scheduled for now.


> - disable SMTPUTF8 in Postfix.


That means disabling it everywhere and let messages bounce on MX servers. Would not really change anything in the end.

patpro
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Viktor Dukhovni
On Wed, Jun 17, 2020 at 10:00:32PM +0200, Patrick Proniewski wrote:

> > - disable SMTPUTF8 in Postfix.
>
> That means disabling it everywhere and let messages bounce on MX servers. Would not really change anything in the end.

Yes, but it is the right thing to do.  Better than generating
backscatter.  When you don't have (end-to-end) SMTPUTF8 support for a
non-trivial fraction of your inbound traffic, DO NOT advertise support
for SMTPUTF8, or dedicate separate inbound MX hosts for the domains that
need and do fully support SMTPUTF8.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

@lbutlr
In reply to this post by Patrick Proniewski
On 17 Jun 2020, at 14:00, Patrick Proniewski <[hidden email]> wrote:
> Not possible yet. A flag exists for Exchange 2019 but we are running 2016 now and upgrade is not scheduled for now.

Perhaps showing the bouncing emails to whomever is in charge of this schedule will change it, especially if any of the bounced emails are personal to that person or deemed to be otherwise important.

> That means disabling it everywhere and let messages bounce on MX servers. Would not really change anything in the end.

The difference is the server receiving the mail in the first place will not claim (falsely) to accept SMTPUTF8.

--
Charlie don't surf!
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Patrick Proniewski
Hello,

> On 17 juin 2020, at 22:48, @lbutlr <[hidden email]> wrote:
>
> On 17 Jun 2020, at 14:00, Patrick Proniewski <[hidden email]> wrote:
>> Not possible yet. A flag exists for Exchange 2019 but we are running 2016 now and upgrade is not scheduled for now.
>
> Perhaps showing the bouncing emails to whomever is in charge of this schedule will change it, especially if any of the bounced emails are personal to that person or deemed to be otherwise important.


I'm in charge actually, but our current efforts are elsewhere (COVID19 related evolutions of digital tools for our staff & students), and upgrading 8 exchange servers and transferring +45 TB of mailboxes is not something one can do quick&dirty ;)


> The difference is the server receiving the mail in the first place will not claim (falsely) to accept SMTPUTF8.

yep, that's true.

patpro
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Patrick Proniewski
In reply to this post by Wietse Venema
Hello,

> On 17 juin 2020, at 16:28, Wietse Venema <[hidden email]> wrote:
>
> Patrick Proniewski:
>> Jun 17 12:34:20 postfix-mailgw/smtp[77347]: 57F56EB256: to=<[hidden email]>, orig_to=<userpart@domain>, relay=Exchange-VIP[Exchange-VIP]:25, delay=0.01, delays=0.01/0/0/0, dsn=5.6.7, status=bounced (SMTPUTF8 is required, but was not offered by host Exchange-VIP[Exchange-VIP])
>>
>
> It is required (in this case, by RFC 6531..6533).
>
> With "smtputf8_enable = yes" (the default) Postfix must return mail
> as undeliverable when:
>
> - a sending SMTP client requested SMTPUTF8 in the MAIL FROM command(*),
>
> - and a receiving SMTP server does not announce SMTPUTF8 support.
>
> However, Postfix ignores this requirement when a message has no
> UTF8 in the header, sender, or recipient envelope address.
>
> Options:
>
> - SMTPUTF8 support in Microsoft Exchange Server protocol revision
>  14.0 (from 2018) or later.
>  https://en.wikipedia.org/wiki/Extended_SMTP#SMTPUTF8


I'll have to wait for this one, at least few months.


> - Postfix: set "smtputf8_enable = no" so that Postfix will not be
>  able to receive such email messages (i.e. they will be returned
>  to the sender before they reach Postfix).
>
> There is no defined conversion from SMTPUTF8 mail to historical
> email, and I decided not to invent one because it would solve only
> a temporary problem.


Makes perfect sense.
I'll go for "smtputf8_enable = no" until we can upgrade Exchange.

thanks
patpro
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Patrick Proniewski
In reply to this post by Viktor Dukhovni
On 17 juin 2020, at 22:05, Viktor Dukhovni <[hidden email]> wrote:

>
> On Wed, Jun 17, 2020 at 10:00:32PM +0200, Patrick Proniewski wrote:
>
>>> - disable SMTPUTF8 in Postfix.
>>
>> That means disabling it everywhere and let messages bounce on MX servers. Would not really change anything in the end.
>
> Yes, but it is the right thing to do.  Better than generating
> backscatter.  When you don't have (end-to-end) SMTPUTF8 support for a
> non-trivial fraction of your inbound traffic, DO NOT advertise support
> for SMTPUTF8, or dedicate separate inbound MX hosts for the domains that
> need and do fully support SMTPUTF8.


I'll do that until we can upgrade Exchange to a version that supports SMTPUTF8.

thanks,
patpro
Reply | Threaded
Open this post in threaded view
|

Re: SMTPUTF8 problem with Exchange servers

Gerard E. Seibert
On Thu, 18 Jun 2020 10:20:48 +0200, Patrick Proniewski stated:

>On 17 juin 2020, at 22:05, Viktor Dukhovni
><[hidden email]> wrote:
>>
>> On Wed, Jun 17, 2020 at 10:00:32PM +0200, Patrick Proniewski wrote:
>>  
>>>> - disable SMTPUTF8 in Postfix.  
>>>
>>> That means disabling it everywhere and let messages bounce on MX
>>> servers. Would not really change anything in the end.  
>>
>> Yes, but it is the right thing to do.  Better than generating
>> backscatter.  When you don't have (end-to-end) SMTPUTF8 support for a
>> non-trivial fraction of your inbound traffic, DO NOT advertise
>> support for SMTPUTF8, or dedicate separate inbound MX hosts for the
>> domains that need and do fully support SMTPUTF8.  
>
>
>I'll do that until we can upgrade Exchange to a version that supports
>SMTPUTF8.

That would be " Microsoft Exchange Server 2019"

https://docs.microsoft.com/en-us/Exchange/new-features/new-features?view=exchserver-2019

--
Jerry