About messages bounced due name resolution issues using IPv6

[Sergio Belkin]

Hi folks,

I have a postfix 2.10 with the following parameters set:

smtp_address_preference = any
inet_protocols = all

Sometimes users get this kind of errors:

<snip>

This is the mail system at host groupware.example.com.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<[hidden email]>: Host or domain name not found. Name
service error for
name=another-example.com.mail.protection.outlook.com type=AAAA: Host
found but no data record of requested type

</snip>

AFAIK if DNS over IPv6 fails it tries over IPv4, doesn't it?

I wonder if beyond this bouncing the smtp uses then IPv4 and sends the messages anyway. Please could you clarify this for me?

Thanks in advance

--

[Wietse Venema]

Sergio Belkin:
> smtp_address_preference = any
> inet_protocols = all
...
> <[hidden email]>: Host or domain name not found. Name
> service error for
> name=another-example.com.mail.protection.outlook.com type=AAAA: Host
> found but no data record of requested type
> </snip>
>
> AFAIK if DNS over IPv6 fails it tries over IPv4, doesn't it?

Postfix will send A and AAAA queries, but it will report an error
only for the last query that it tried. You might fimnd more details
in the Postfix logging.

So, the complete error message would be :

    "I made DNS queries with type A and AAAA for the name
    another-example.com.mail.protection.outlook.com. All queries
    failed. The last query that failed had type AAAA. The last error
    was "name exists but there is no AAAA record".

But we reall don't want to send THAT in an email bounce message.

> I wonder if beyond this bouncing the smtp uses then IPv4 and sends the
> messages anyway. Please could you clarify this for me?

All A and AAAA queries failed.

        Wietse

[Sergio Belkin]

El jue, 3 dic 2020 a las 16:37, Wietse Venema (<[hidden email]>) escribió:

Sergio Belkin:
> smtp_address_preference = any
> inet_protocols = all
...
> <[hidden email]>: Host or domain name not found. Name
> service error for
> name=another-example.com.mail.protection.outlook.com type=AAAA: Host
> found but no data record of requested type
> </snip>
>
> AFAIK if DNS over IPv6 fails it tries over IPv4, doesn't it?

Postfix will send A and AAAA queries, but it will report an error
only for the last query that it tried. You might fimnd more details
in the Postfix logging.

So, the complete error message would be :

    "I made DNS queries with type A and AAAA for the name
    another-example.com.mail.protection.outlook.com. All queries
    failed. The last query that failed had type AAAA. The last error
    was "name exists but there is no AAAA record".

But we reall don't want to send THAT in an email bounce message.

> I wonder if beyond this bouncing the smtp uses then IPv4 and sends the
> messages anyway. Please could you clarify this for me?

All A and AAAA queries failed.

        Wietse

Thanks Wietse for your answer

Is quite interesting that I find the following in logs:

Dec  2 23:53:09 muteriver postfix/smtp[28063]: warning: no MX host for another-example.com has a valid address record

And then:

Dec  2 23:53:09 muteriver postfix/smtp[28063]: ED1CF1813C56F: to=<[hidden email]>, relay=none, delay=5.9, delays=0.17/0/5.8/0, dsn=5.4.4, status=bounced (Host or domain name not found. Name service error for name=another-example.com.mail.protection.outlook.com type=AAAA: Host found but no data record of requested type)

and finally:

Dec  2 23:53:10 muteriver postfix/qmgr[1528]: ED1CF1813C56F: removed

That last line led me to wonder if the message was finally sent...

If I try to resolve another-example.com.mail.protection.outlook.com manually on the mail server works fine with IPv4.

What do you think?

Was the message never sent?

If so, is there a way that postfix retries later if there is a temporary resolution?

Thanks in advance!

--

[Viktor Dukhovni]

On Thu, Dec 03, 2020 at 05:34:45PM -0300, Sergio Belkin wrote:

> Dec  2 23:53:09 muteriver postfix/smtp[28063]: ED1CF1813C56F: to=<
> [hidden email]>, relay=none, delay=5.9, delays=0.17/0/5.8/0,
> dsn=5.4.4, status=bounced (Host or domain name not found. Name service
> error for name=another-example.com.mail.protection.outlook.com type=AAAA:
> Host found but no data record of requested type)

This was not a transient error, an authoritative "no such host or
address" response was received as is seen from the hard "5.4.4" DSN.

    https://tools.ietf.org/html/rfc3463#section-3.5

With transient DNS errors, the DSN would have been 4.4.4.  You should
also note the "status=bounced".  The message was "returned to sender,
address unknown".

> Dec  2 23:53:10 muteriver postfix/qmgr[1528]: ED1CF1813C56F: removed

The original message was therefore removed.

--
    Viktor.

[Wietse Venema]

> So, the complete error message would be :
>
>     "I made DNS queries with type A and AAAA for the name
>     another-example.com.mail.protection.outlook.com. All queries
>     failed. The last query that failed had type AAAA. The last error
>     was "name exists but there is no AAAA record".
>
> But we reall don't want to send THAT in an email bounce message.
>
> > I wonder if beyond this bouncing the smtp uses then IPv4 and sends the
> > messages anyway. Please could you clarify this for me?
>
> All A and AAAA queries failed.

Sergio Belkin:
>
> Thanks Wietse for your answer
>
> Is quite interesting that I find the following in logs:
> Dec  2 23:53:09 muteriver postfix/smtp[28063]: warning: no MX host for
> another-example.com has a valid address record

Indeed, these details are not revealed in the bounce message but
can be found in Postfix logs.

> And then:
>
> Dec  2 23:53:09 muteriver postfix/smtp[28063]: ED1CF1813C56F: to=<
> [hidden email]>, relay=none, delay=5.9, delays=0.17/0/5.8/0,
> dsn=5.4.4, status=bounced (Host or domain name not found. Name service
> error for name=another-example.com.mail.protection.outlook.com type=AAAA:
> Host found but no data record of requested type)
>
> and finally:
>
> Dec  2 23:53:10 muteriver postfix/qmgr[1528]: ED1CF1813C56F: removed
>
> That last line led me to wonder if the message was finally sent...

The message ED1CF1813C56F is deleted ONLY after Postfix successfully
injects the non-delivery notifcation into the Postfix mail queue.

> If I try to resolve another-example.com.mail.protection.outlook.com
> manually on the mail server works fine with IPv4.
>
> What do you think?

What comes to mind:

1) You ran the command as root, and the Postfix SMTP client does
not run as root. Name resution fails when the necessary files are
not accessible.

2) You ran the command outside the Postfix chroot jail, and the
Postfix SMTP client runs inside the Postfix chroot jail. Name
resolution fails inside the chroot jail when files are missing,
have wrong permissions, or have wrong contents.

3) Some "security" configuration is breaking Postfix. For exammple
SeLiux or AppArmor.

4) Some other permisssion or configuration problem.

To find out if name resolution fails due to missing files or bad
permissions, run the Postfix SMTP client under strace as described
in http://www.postfix.org/DEBUG_README.html

        Wietse

[Bastian Blank-3]

Hi Sergio

On Thu, Dec 03, 2020 at 05:34:45PM -0300, Sergio Belkin wrote:
> Is quite interesting that I find the following in logs:
> Dec  2 23:53:09 muteriver postfix/smtp[28063]: warning: no MX host for
> another-example.com has a valid address record

Well, more serious: another-example.com does not even have _any_ MX
entries:

| % drill another-example.com mx
| ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 55819
| ;; another-example.com. IN MX
|
| ;; ANSWER SECTION:
|
| ;; AUTHORITY SECTION:
| another-example.com. 900 IN SOA ns-1861.awsdns-40.co.uk.  awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

Bastian

--
Captain's Log, star date 21:34.5...

[Sergio Belkin]

El jue, 3 dic 2020 a las 17:59, Wietse Venema (<[hidden email]>) escribió:

> So, the complete error message would be :
>
>     "I made DNS queries with type A and AAAA for the name
>     another-example.com.mail.protection.outlook.com. All queries
>     failed. The last query that failed had type AAAA. The last error
>     was "name exists but there is no AAAA record".
>
> But we reall don't want to send THAT in an email bounce message.
>
> > I wonder if beyond this bouncing the smtp uses then IPv4 and sends the
> > messages anyway. Please could you clarify this for me?
>
> All A and AAAA queries failed.

Sergio Belkin:
>
> Thanks Wietse for your answer
>
> Is quite interesting that I find the following in logs:
> Dec  2 23:53:09 muteriver postfix/smtp[28063]: warning: no MX host for
> another-example.com has a valid address record

Indeed, these details are not revealed in the bounce message but
can be found in Postfix logs.

> And then:
>
> Dec  2 23:53:09 muteriver postfix/smtp[28063]: ED1CF1813C56F: to=<
> [hidden email]>, relay=none, delay=5.9, delays=0.17/0/5.8/0,
> dsn=5.4.4, status=bounced (Host or domain name not found. Name service
> error for name=another-example.com.mail.protection.outlook.com type=AAAA:
> Host found but no data record of requested type)
>
> and finally:
>
> Dec  2 23:53:10 muteriver postfix/qmgr[1528]: ED1CF1813C56F: removed
>
> That last line led me to wonder if the message was finally sent...

The message ED1CF1813C56F is deleted ONLY after Postfix successfully
injects the non-delivery notifcation into the Postfix mail queue.

> If I try to resolve another-example.com.mail.protection.outlook.com
> manually on the mail server works fine with IPv4.
>
> What do you think?

What comes to mind:

1) You ran the command as root, and the Postfix SMTP client does
not run as root. Name resution fails when the necessary files are
not accessible.

2) You ran the command outside the Postfix chroot jail, and the
Postfix SMTP client runs inside the Postfix chroot jail. Name
resolution fails inside the chroot jail when files are missing,
have wrong permissions, or have wrong contents.

3) Some "security" configuration is breaking Postfix. For exammple
SeLiux or AppArmor.

I have SELinux disables, and have no AppArmor

4) Some other permisssion or configuration problem.

It's weird because it only happens with a few domains...

 

To find out if name resolution fails due to missing files or bad
permissions, run the Postfix SMTP client under strace as described
in http://www.postfix.org/DEBUG_README.html

        Wietse

Thanks

--

[Wietse Venema]

Sergio Belkin:

> > What comes to mind:
> >
> > 1) You ran the command as root, and the Postfix SMTP client does
> > not run as root. Name resution fails when the necessary files are
> > not accessible.
> >
> > 2) You ran the command outside the Postfix chroot jail, and the
> > Postfix SMTP client runs inside the Postfix chroot jail. Name
> > resolution fails inside the chroot jail when files are missing,
> > have wrong permissions, or have wrong contents.
> >
> > 3) Some "security" configuration is breaking Postfix. For exammple
> > SeLiux or AppArmor.
> >
>
> I have SELinux disables, and have no AppArmor
>
> >
> > 4) Some other permisssion or configuration problem.
>
> It's weird because it only happens with a few domains...

It would be perfectly consistent with cases 1, 2, or 4 above. You
can start with a network sniffer and verify that Postfix sends its
the right DNS queries to the 'right' server.

        Wietse

[Sergio Belkin]

El jue, 3 dic 2020 a las 18:18, Wietse Venema (<[hidden email]>) escribió:

Sergio Belkin:
> > What comes to mind:
> >
> > 1) You ran the command as root, and the Postfix SMTP client does
> > not run as root. Name resution fails when the necessary files are
> > not accessible.
> >
> > 2) You ran the command outside the Postfix chroot jail, and the
> > Postfix SMTP client runs inside the Postfix chroot jail. Name
> > resolution fails inside the chroot jail when files are missing,
> > have wrong permissions, or have wrong contents.
> >
> > 3) Some "security" configuration is breaking Postfix. For exammple
> > SeLiux or AppArmor.
> >
>
> I have SELinux disables, and have no AppArmor
>
> >
> > 4) Some other permisssion or configuration problem.
>
> It's weird because it only happens with a few domains...

It would be perfectly consistent with cases 1, 2, or 4 above. You
can start with a network sniffer and verify that Postfix sends its
the right DNS queries to the 'right' server.

        Wietse

OK Wietse, somewhat to take in account, such  domain name has not  a record AAAA for its MX, but only has a record A.

--

[Sergio Belkin]

El jue, 3 dic 2020 a las 18:04, Bastian Blank (<bastian+postfix-users=[hidden email]>) escribió:

Hi Sergio

On Thu, Dec 03, 2020 at 05:34:45PM -0300, Sergio Belkin wrote:
> Is quite interesting that I find the following in logs:
> Dec  2 23:53:09 muteriver postfix/smtp[28063]: warning: no MX host for
> another-example.com has a valid address record

Well, more serious: another-example.com does not even have _any_ MX
entries:

| % drill another-example.com mx
| ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 55819
| ;; another-example.com.       IN      MX
|
| ;; ANSWER SECTION:
|
| ;; AUTHORITY SECTION:
| another-example.com.  900     IN      SOA     ns-1861.awsdns-40.co.uk.  awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

Bastian

Thanks Bastian, I had to obfuscate the domain name due to privacy issues :-/

But I told to Wietse, that domain name has a MX record with an 'A' record associated, but lacks of 'AAAA' record.

What is weird for me is that it happens sometimes not always with that domain...

 

Thanks in advance

--
Captain's Log, star date 21:34.5...

--

[Viktor Dukhovni]

> On Dec 3, 2020, at 7:28 PM, Sergio Belkin <[hidden email]> wrote:
>
> Thanks Bastian, I had to obfuscate the domain name due to privacy issues :-/
> But I told to Wietse, that domain name has a MX record with an 'A' record associated, but lacks of 'AAAA' record.
> What is weird for me is that it happens sometimes not always with that domain...

Even after Wietse and I prowled around in all the
dark places in the code, we still failed to find
a plausible way for a soft error in IPv4 lookups
to fail to suppress a hard NODATA for IPv6 lookups.

The only possible explanations are:

   - Your resolver erroneously reported NODATA for IPv4
   - The authoritative nameservers reported NODATA for IPv4
   - Neither of us was able to spot a subtle bug

To distinguish between these, it would be helpful if you
set:

   debug_peer_list = another-example-com.mail.protection.outlook.com
   debug_peer_level = 1

and if/when the problem happens again, either posted a sanitised
log (with the guilty domain name obscured as appropriate) to the
list, or sent a copy to just me and Wietse off list.

I'd like to see the actual queries and responses logged by the
Postfix SMTP delivery agent.

--
        Viktor.

[Sergio Belkin]

El jue, 3 dic 2020 a las 22:23, Viktor Dukhovni (<[hidden email]>) escribió:

> On Dec 3, 2020, at 7:28 PM, Sergio Belkin <[hidden email]> wrote:
>
> Thanks Bastian, I had to obfuscate the domain name due to privacy issues :-/
> But I told to Wietse, that domain name has a MX record with an 'A' record associated, but lacks of 'AAAA' record.
> What is weird for me is that it happens sometimes not always with that domain...

Even after Wietse and I prowled around in all the
dark places in the code, we still failed to find
a plausible way for a soft error in IPv4 lookups
to fail to suppress a hard NODATA for IPv6 lookups.

The only possible explanations are:

   - Your resolver erroneously reported NODATA for IPv4
   - The authoritative nameservers reported NODATA for IPv4
   - Neither of us was able to spot a subtle bug

To distinguish between these, it would be helpful if you
set:

   debug_peer_list = another-example-com.mail.protection.outlook.com
   debug_peer_level = 1

and if/when the problem happens again, either posted a sanitised
log (with the guilty domain name obscured as appropriate) to the
list, or sent a copy to just me and Wietse off list.

I'd like to see the actual queries and responses logged by the
Postfix SMTP delivery agent.

--
        Viktor.

Thanks Viktor and Wietse, up to now the error didn't happen again.

Some information about my software that may be useful:

libc-2.17-260.el7_6.4.x86_64
glibc-2.17-260.el7_6.4.i686
dnsmasq-2.76-7.el7.x86_64

--

[Sergio Belkin]

El vie, 4 dic 2020 a las 1:42, Sergio Belkin (<[hidden email]>) escribió:

El jue, 3 dic 2020 a las 22:23, Viktor Dukhovni (<[hidden email]>) escribió:

> On Dec 3, 2020, at 7:28 PM, Sergio Belkin <[hidden email]> wrote:
>
> Thanks Bastian, I had to obfuscate the domain name due to privacy issues :-/
> But I told to Wietse, that domain name has a MX record with an 'A' record associated, but lacks of 'AAAA' record.
> What is weird for me is that it happens sometimes not always with that domain...

Even after Wietse and I prowled around in all the
dark places in the code, we still failed to find
a plausible way for a soft error in IPv4 lookups
to fail to suppress a hard NODATA for IPv6 lookups.

The only possible explanations are:

   - Your resolver erroneously reported NODATA for IPv4
   - The authoritative nameservers reported NODATA for IPv4
   - Neither of us was able to spot a subtle bug

To distinguish between these, it would be helpful if you
set:

   debug_peer_list = another-example-com.mail.protection.outlook.com
   debug_peer_level = 1

and if/when the problem happens again, either posted a sanitised
log (with the guilty domain name obscured as appropriate) to the
list, or sent a copy to just me and Wietse off list.

I'd like to see the actual queries and responses logged by the
Postfix SMTP delivery agent.

--
        Viktor.

Thanks Viktor and Wietse, up to now the error didn't happen again.

Some information about my software that may be useful:

libc-2.17-260.el7_6.4.x86_64
glibc-2.17-260.el7_6.4.i686
dnsmasq-2.76-7.el7.x86_64

Just in case: the first package is  glibc-2.17-260.el7_6.4.x86_64

--

[Viktor Dukhovni]

On Fri, Dec 04, 2020 at 01:42:57AM -0300, Sergio Belkin wrote:

> Thanks Viktor and Wietse, up to now the error didn't happen again.
> Some information about my software that may be useful:
> libc-2.17-260.el7_6.4.x86_64
> glibc-2.17-260.el7_6.4.i686
> dnsmasq-2.76-7.el7.x86_64

Is there a compelling reason to run a stripped-down (and typically not
adequately standards-conformant) DNS resolvers on a mail server?

I'd steer well clear of "dnsmasq", "systemd-resolved" and other
"DNS made simple" resolvers.

Install "unbound", "bind" or "knot", whichever you're most
comfortable with.

--
    Viktor.

[Sergio Belkin]

El vie, 4 dic 2020 a las 2:15, Viktor Dukhovni (<[hidden email]>) escribió:

On Fri, Dec 04, 2020 at 01:42:57AM -0300, Sergio Belkin wrote:

> Thanks Viktor and Wietse, up to now the error didn't happen again.
> Some information about my software that may be useful:
> libc-2.17-260.el7_6.4.x86_64
> glibc-2.17-260.el7_6.4.i686
> dnsmasq-2.76-7.el7.x86_64

Is there a compelling reason to run a stripped-down (and typically not
adequately standards-conformant) DNS resolvers on a mail server?

I use mainly for caching purposes

 

I'd steer well clear of "dnsmasq", "systemd-resolved" and other
"DNS made simple" resolvers.

Install "unbound", "bind" or "knot", whichever you're most
comfortable with.

--
    Viktor.

--

[Wietse Venema]

Sergio Belkin:
> Some information about my software that may be useful:
> libc-2.17-260.el7_6.4.x86_64
> glibc-2.17-260.el7_6.4.i686
> dnsmasq-2.76-7.el7.x86_64

Are you running a RHEL port of Postfix 2.10? They sometimes back-port
features from a later Postfix release. The Postfix code that handles
DNS errors is very careful about giving 'try again' errors precedence
over 'hard' errors, and it is very easy to screw that up.

Does it have the Postfix 3.0 smtp_dns_reply_filter feature?

        Wietse

[Matus UHLAR - fantomas]

>> On Fri, Dec 04, 2020 at 01:42:57AM -0300, Sergio Belkin wrote:
>> > Thanks Viktor and Wietse, up to now the error didn't happen again.
>> > Some information about my software that may be useful:
>> > libc-2.17-260.el7_6.4.x86_64
>> > glibc-2.17-260.el7_6.4.i686
>> > dnsmasq-2.76-7.el7.x86_64

>El vie, 4 dic 2020 a las 2:15, Viktor Dukhovni
>(<[hidden email]>) escribió:
>> Is there a compelling reason to run a stripped-down (and typically not
>> adequately standards-conformant) DNS resolvers on a mail server?

On 04.12.20 08:41, Sergio Belkin wrote:
>I use mainly for caching purposes

that's not the point. the point is, especially with allow/blocklists and spam
filters, using own DNS resolvers is important, since shared DNS servers are
often blocked by public DNS lists and the effectivity of filtering lowers.
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"Where do you want to go to die?" [Microsoft]

[Sebastian Wiesinger-10]

* Matus UHLAR - fantomas <[hidden email]> [2020-12-04 15:08]:

> > El vie, 4 dic 2020 a las 2:15, Viktor Dukhovni
> > (<[hidden email]>) escribió:
> > > Is there a compelling reason to run a stripped-down (and typically not
> > > adequately standards-conformant) DNS resolvers on a mail server?
>
> On 04.12.20 08:41, Sergio Belkin wrote:
> > I use mainly for caching purposes
>
> that's not the point. the point is, especially with allow/blocklists and spam
> filters, using own DNS resolvers is important, since shared DNS servers are
> often blocked by public DNS lists and the effectivity of filtering lowers.

The point was that his choice of software is probably not the best
choice for caching on box.

Regards

Sebastian

--
GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A  9D82 58A2 D94A 93A0 B9CE)
'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE.
            -- Terry Pratchett, The Fifth Elephant

[Sergio Belkin]

El vie, 4 dic 2020 a las 9:59, Wietse Venema (<[hidden email]>) escribió:

Sergio Belkin:
> Some information about my software that may be useful:
> libc-2.17-260.el7_6.4.x86_64
> glibc-2.17-260.el7_6.4.i686
> dnsmasq-2.76-7.el7.x86_64

Are you running a RHEL port of Postfix 2.10? They sometimes back-port
features from a later Postfix release. The Postfix code that handles
DNS errors is very careful about giving 'try again' errors precedence
over 'hard' errors, and it is very easy to screw that up.

Does it have the Postfix 3.0 smtp_dns_reply_filter feature?

        Wietse

Wietse, Sadly it's not a backports, because is CentOS 7. It lacks that option

--

[Viktor Dukhovni]

> On Dec 4, 2020, at 9:41 AM, Sergio Belkin <[hidden email]> wrote:
>
> > dnsmasq-2.76-7.el7.x86_64
>
> Is there a compelling reason to run a stripped-down (and typically not
> adequately standards-conformant) DNS resolvers on a mail server?
>
> I use mainly for caching purposes

That's a reason to run a local caching resolver, which is highly
recommended on an MTA, *but* it is not a reason to choose a shoddy
resolver like dnsmasq or systemd-resolved, ... which is barely
good enough for laptop use, where integration with DHCP, roaming
etc. provides some "pros" to offset the "cons", but on a server
MTA there is no plausible reason to choose these.

> I'd steer well clear of "dnsmasq", "systemd-resolved" and other
> "DNS made simple" resolvers.

Steer well clear of "dnsmasq", "systemd-resolved", ...

> Install "unbound", "bind" or "knot", whichever you're most
> comfortable with.

Install "unbound", "BIND" or "Knot", whichever you're most
familiar with.

But before you do that, it would be useful to stick with
what you have for a little longer, with "debug_peer_list"
enabled in the hope that we can narrow down the source of
the problem.  My present bet is on a bug in "dnsmasq".  But
it is possible that some DNS load-balancer or other in some
Microsoft datacentre is unexpectedly responding NODATA for
IPv4.

A bug in Postfix soft-error handling sure looks unlikely.
All the relevant functions note soft lookup errors and do
not let subsequent hard lookup errors erase a preceding soft
error status.

--
        Viktor.