Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Jozef Matický
Hello,

I've been struggling with this for about a week now.
In smtpd_recipient_restrictions I have reject_unverified_recipient.
For recipient address verification I'm using Dovecot's LMTP.
Everything is working as expected when address_verify_negative_cache =
yes - unknown recipients are rejected with 550 (NOQUEUE: reject), for
known the mail is delivered.

The problem I have is when I set address_verify_negative_cache = no.
It goes like this:

- Sender connects to Postfix
+ Postfix is checking address with Dovecot
+ Dovecot responds (almost instantly) with 550 5.1.1 User doesn't exist;
status=undeliverable-but-not-cached
- Above + points are repeated as many as address_verify_poll_count times
(in my case 5 times, with default it happened 3 times)
- Postfix then replies to sender with 450 Recipient address rejected:
unverified address: Address verification in progress.
- After a while sender is trying to deliver the same e-mail again and
the same thing is happening - it is deffered with 450
- This goes on and on and on

It looks like when there is status=undeliverable-but-not-cached Postfix
is trying to verify the recipient address address_verify_poll_count
times and doesn't understand the Dovecot's 550 reply
(status=undeliverable-but-not-cached), after which it should reject
sender with 550 (NOQUEUE: reject).

Am I doing something wrong or is this some kind of a bug?
The reason I'm trying to turn off negative cache is due to the catch-all.
For example, user is trying to send an email to mailbox that doesn't
exist. The e-mail will be rejected with 550.
Then user creates a catch-all for the domain and will send e-mail to the
same address.
Due to the negative cache it again will be rejected with 550 despite the
fact there now is catch-all configured.

This is with Postfix 3.2.3 which I just upgraded from 3.1.6 (on which it
was exactly the same). I also have unverified_recipient_reject_code =
550 set.
If something else is needed, like logs or postconf, please let me know.
Don't want to spam mailing list if this is a problem between the chair
and the keyboard.

Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

A. Schulze


Am 14.10.2017 um 23:23 schrieb Jozef Matický:

> Hello,
>
> I've been struggling with this for about a week now.
> In smtpd_recipient_restrictions I have reject_unverified_recipient.
> For recipient address verification I'm using Dovecot's LMTP.
> Everything is working as expected when address_verify_negative_cache = yes - unknown recipients are rejected with 550 (NOQUEUE: reject), for known the mail is delivered.
>
> The problem I have is when I set address_verify_negative_cache = no.
> It goes like this:
>
> - Sender connects to Postfix
> + Postfix is checking address with Dovecot
> + Dovecot responds (almost instantly) with 550 5.1.1 User doesn't exist; status=undeliverable-but-not-cached
> - Above + points are repeated as many as address_verify_poll_count times (in my case 5 times, with default it happened 3 times)
> - Postfix then replies to sender with 450 Recipient address rejected: unverified address: Address verification in progress.
> - After a while sender is trying to deliver the same e-mail again and the same thing is happening - it is deffered with 450
> - This goes on and on and on
>
> It looks like when there is status=undeliverable-but-not-cached Postfix is trying to verify the recipient address address_verify_poll_count times and doesn't understand the Dovecot's 550 reply (status=undeliverable-but-not-cached), after which it should reject sender with 550 (NOQUEUE: reject).
>
> Am I doing something wrong or is this some kind of a bug?
> The reason I'm trying to turn off negative cache is due to the catch-all.
> For example, user is trying to send an email to mailbox that doesn't exist. The e-mail will be rejected with 550.
> Then user creates a catch-all for the domain and will send e-mail to the same address.
> Due to the negative cache it again will be rejected with 550 despite the fact there now is catch-all configured.
>
> This is with Postfix 3.2.3 which I just upgraded from 3.1.6 (on which it was exactly the same). I also have unverified_recipient_reject_code = 550 set.
> If something else is needed, like logs or postconf, please let me know.
> Don't want to spam mailing list if this is a problem between the chair and the keyboard.

Josef,

Could you explain why you completely disable address_verify_negative_cache?
I personally would only shorten address_verify_negative_refresh_time if caching would be an issue.

Andreas
>
> Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Jozef Matický
Hello Andreas,

Had some users complaining about mailboxes (and catch-alls) they created
were not accepting e-mails - Postfix, due to caching, was rejecting the
e-mail as if the mailbox wouldn't exist (apparently due to a fact
someone was trying to send an e-mail to that mailbox before it even
existed - maybe spammer, maybe a legit user).

The below statement I'm not 100% sure about as I was playing with
multiple config options, will have to check this again, however:
I believe I also played with negative_refresh_time (setting it to 1s),
however I think what was happening was that e-mail was 450 deferred
(address ver. in prog.) for the first time the sender connected and
tried to deliver to already neg. cached address. Only afterwards the
cache was refreshed - causing the e-mail to defer with 450 once.
This would have caused delays in delivery (or lost e-mails with bad
implementations) and more complaints from users saying there is a
problem with e-mails.

Anyways, I really believe there is a bug with
address_verify_negative_cache = no implementation.

Thank you.

Best regards,
Jozef.


On 15. 10. 2017 12:50, A. Schulze wrote:

>
>
> Am 14.10.2017 um 23:23 schrieb Jozef Matický:
>> Hello,
>>
>> I've been struggling with this for about a week now.
>> In smtpd_recipient_restrictions I have reject_unverified_recipient.
>> For recipient address verification I'm using Dovecot's LMTP.
>> Everything is working as expected when address_verify_negative_cache = yes - unknown recipients are rejected with 550 (NOQUEUE: reject), for known the mail is delivered.
>>
>> The problem I have is when I set address_verify_negative_cache = no.
>> It goes like this:
>>
>> - Sender connects to Postfix
>> + Postfix is checking address with Dovecot
>> + Dovecot responds (almost instantly) with 550 5.1.1 User doesn't exist; status=undeliverable-but-not-cached
>> - Above + points are repeated as many as address_verify_poll_count times (in my case 5 times, with default it happened 3 times)
>> - Postfix then replies to sender with 450 Recipient address rejected: unverified address: Address verification in progress.
>> - After a while sender is trying to deliver the same e-mail again and the same thing is happening - it is deffered with 450
>> - This goes on and on and on
>>
>> It looks like when there is status=undeliverable-but-not-cached Postfix is trying to verify the recipient address address_verify_poll_count times and doesn't understand the Dovecot's 550 reply (status=undeliverable-but-not-cached), after which it should reject sender with 550 (NOQUEUE: reject).
>>
>> Am I doing something wrong or is this some kind of a bug?
>> The reason I'm trying to turn off negative cache is due to the catch-all.
>> For example, user is trying to send an email to mailbox that doesn't exist. The e-mail will be rejected with 550.
>> Then user creates a catch-all for the domain and will send e-mail to the same address.
>> Due to the negative cache it again will be rejected with 550 despite the fact there now is catch-all configured.
>>
>> This is with Postfix 3.2.3 which I just upgraded from 3.1.6 (on which it was exactly the same). I also have unverified_recipient_reject_code = 550 set.
>> If something else is needed, like logs or postconf, please let me know.
>> Don't want to spam mailing list if this is a problem between the chair and the keyboard.
>
> Josef,
>
> Could you explain why you completely disable address_verify_negative_cache?
> I personally would only shorten address_verify_negative_refresh_time if caching would be an issue.
>
> Andreas
>>
>> Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Wietse Venema
In reply to this post by A. Schulze
A. Schulze:
> > The problem I have is when I set address_verify_negative_cache = no.
...
> > It looks like when there is status=undeliverable-but-not-cached

There is no such reply. In reality, the reply is that the address
probe is in progress (DEL_RCPT_STAT_TODO).

This is what happens:

1. SMTP daemon queries verify service for an address probe result.
2. There is no address probe result. This triggers a verify probe.
3. Repeat 1 until the addres probe poll count is reached.

I don't see how one would avoid this, other than by blocking the
SMTP daemon's query while a probe is in progress or until a time
limit is reached.

If you want to create accounts on the fly, don't break address
verification.

Instead, change your local_recipient_maps, virtual_alias_maps,
relay_recipient_maps or virtual_mailbox_maps. At the end of those
maps, a tcp_table or socketmap service that handles the queries for
non-existent addresses. This service replies with 5xx if the account
should not exist, or it replies with 4xx and creates the account.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

A. Schulze
In reply to this post by Jozef Matický


Am 15.10.2017 um 15:29 schrieb Jozef Matický:
> Hello Andreas,
>
> Had some users complaining about mailboxes (and catch-alls) they created were not accepting e-mails - Postfix, due to caching, was rejecting the e-mail as if the mailbox wouldn't exist (apparently due to a fact someone was trying to send an e-mail to that mailbox before it even existed - maybe spammer, maybe a legit user).

That's a use case we where also faced years ago. We solved it by documentation. Also we "train" our users: whatever you configure, wait an hour!
(No matter if it' really required or not)

Then we consequently arrange our systems so every change is visible within one hour: address_verify_negative_refresh_time = 30m in this case.
Simple but understandable for our users

Andreas
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Wietse Venema
In reply to this post by Jozef Matický
Jozef Matick?:

> Hello Andreas,
>
> Had some users complaining about mailboxes (and catch-alls) they created
> were not accepting e-mails - Postfix, due to caching, was rejecting the
> e-mail as if the mailbox wouldn't exist (apparently due to a fact
> someone was trying to send an e-mail to that mailbox before it even
> existed - maybe spammer, maybe a legit user).
>
> The below statement I'm not 100% sure about as I was playing with
> multiple config options, will have to check this again, however:
> I believe I also played with negative_refresh_time (setting it to 1s),
> however I think what was happening was that e-mail was 450 deferred
> (address ver. in prog.) for the first time the sender connected and
> tried to deliver to already neg. cached address. Only afterwards the
> cache was refreshed - causing the e-mail to defer with 450 once.
> This would have caused delays in delivery (or lost e-mails with bad
> implementations) and more complaints from users saying there is a
> problem with e-mails.

Summary: if you want to make you scheme work with address verification,
then you need a delivery agent that creates mailboxes instead of
replying with 'user unknown'. But that is not all.

Some choices are:

- The SMTP server replies with 5xx or 4xx, because the delivery
  agent replies with 'user unknown'. You don't want that.

- The SMTP server replies with 4xx, because the verify service
  replies with 'address verification in progress'. You don't want
  that.

- The SMTP server replies with 2xx, even though the delivery agent
  replies with 'user unknown'. The SMTP server must accept the email
  message, and delivery will fail immediately. You don't want that.

- The SMTP server replies with 2xx, because the delivery agent
  replies with 'user exists'. This means that when a mailbox does
  not exist, the delivery agent must reply AFTER the mailbox is
  created.

Since the time to create mailboxes increases with server load, that
last variant also requires removing the loop with address_verify_poll_count
and address_verify_poll_delay, and replacing that loop with a
blocking query.

See a different response for hooking the auto-create feature into
the recipient maps for recipient validation.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Jozef Matický
In reply to this post by Wietse Venema
Hello Wietse,

Not sure I understand.
The problem I have is when I have address_verify_negative_cache = no.
Postfix is then deferring sender with 450 Address verification in
progress despite the fact Dovecot (LMTP) is replying to Postfix with 550
User doesn't exists (with smtpd_recipient_restrictions =
reject_unverified_recipient as I don't want to generate backscatter).
I assumed when Dovecot replies with 550, then Postfix should reject the
email, not to deffer the delivery.
This is the reason I'm not able to turn address_verify_negative_cache
off as e-mail for users that don't exist will be deferred instead of
rejected, thus it will be floating somewhere between the servers for
couple of days after which sender will be notified that there is a
problem with delivery (and not because user doesn't exists but because
recipient's server is doing address verification for multiple days now).

When I set address_verify_negative_cache = yes, then all is working as
expected - Postfix rejects the sender with 550 when Dovecot's LMTP
replies with 550 User doesn't exists.
When Dovecot replies User does exists then mail is taken for delivery.

Thank you.

Best regards,
Jozef.


On 15. 10. 2017 16:31, Wietse Venema wrote:

> A. Schulze:
>>> The problem I have is when I set address_verify_negative_cache = no.
> ...
>>> It looks like when there is status=undeliverable-but-not-cached
>
> There is no such reply. In reality, the reply is that the address
> probe is in progress (DEL_RCPT_STAT_TODO).
>
> This is what happens:
>
> 1. SMTP daemon queries verify service for an address probe result.
> 2. There is no address probe result. This triggers a verify probe.
> 3. Repeat 1 until the addres probe poll count is reached.
>
> I don't see how one would avoid this, other than by blocking the
> SMTP daemon's query while a probe is in progress or until a time
> limit is reached.
>
> If you want to create accounts on the fly, don't break address
> verification.
>
> Instead, change your local_recipient_maps, virtual_alias_maps,
> relay_recipient_maps or virtual_mailbox_maps. At the end of those
> maps, a tcp_table or socketmap service that handles the queries for
> non-existent addresses. This service replies with 5xx if the account
> should not exist, or it replies with 4xx and creates the account.
>
> Wietse
>
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Wietse Venema
Jozef Matick?:
> Hello Wietse,
>
> Not sure I understand.
> The problem I have is when I have address_verify_negative_cache = no.
> Postfix is then deferring sender with 450 Address verification in
> progress despite the fact Dovecot (LMTP) is replying to Postfix with 550
> User doesn't exists (with smtpd_recipient_restrictions =
> reject_unverified_recipient as I don't want to generate backscatter).

When a mailbox needs to be created on-demand, you don't want the
4xx response (verification in progress) before the mailbox is
created, but you want the 5xx response (mailbox does not exist)
which causes the message to be returned as undeliverable.

If you really want that, just set a short expire time for negative
verification results.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Strange issue with reject_unverified_recipient (LMTP/Dovecot)

Jozef Matický
Okay, will do it that way then.

Thank you!

Best regards,
Jozef.

On 16. 10. 2017 0:42, Wietse Venema wrote:

> Jozef Matick?:
>> Hello Wietse,
>>
>> Not sure I understand.
>> The problem I have is when I have address_verify_negative_cache = no.
>> Postfix is then deferring sender with 450 Address verification in
>> progress despite the fact Dovecot (LMTP) is replying to Postfix with 550
>> User doesn't exists (with smtpd_recipient_restrictions =
>> reject_unverified_recipient as I don't want to generate backscatter).
>
> When a mailbox needs to be created on-demand, you don't want the
> 4xx response (verification in progress) before the mailbox is
> created, but you want the 5xx response (mailbox does not exist)
> which causes the message to be returned as undeliverable.
>
> If you really want that, just set a short expire time for negative
> verification results.
>
> Wietse
>