Yahoo rate limit (again...)

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Yahoo rate limit (again...)

gaia45500
Hi,

Sorry about a such boring subject : we have problems with Yahoo for sending mails.
OUr company have to send "massive" newsletter to yahoo users, but we are frequently (always ?) deferred, with only one "simple" explanation :

0.05/0/0.22/0.03, dsn=4.7.0, status=deferred (host mx-eu.mail.am0.yahoodns.net[188.125.72.73] said: 421 4.7.0 [TSS04] Messages from X.X.X.X temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html (in reply to MAIL FROM command))

After contacting Yahoo (no other response than the automate one...) and searching for slow down process, we found many docs about how to slow down sending for a domain.

We found config like 

yahoo_initial_destination_concurrency=3
yahoo_destination_concurrency_limit=2
yahoo_destination_recipient_limit=2
yahoo_destination_rate_delay=12s

with corresponding master.cf config.

It seems to work, as mails for yahoo are send with "large" intervals. On normal behaviour, mails are accepted.
BUT
when mails are blocked, there are stored in deferred queue. And when Postfix "replay" these deferred mails, slow down does'nt apply.

That is : 
- when all goes well, mails are sent and accepted
- when somethinfg is "wrong" (flooding from a user), IP is blocked
- we change IP, and all goes well again, but as Postix replay the deferred queue, everything is again considered as spam and IP is BL, and again, and again...

Why does slow down not apply to deferred replay ?
Is it possible to prevent Postfix to "flush" all when it replays mail ?

Regards,

Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

Wietse Venema
Emmanuel BILLOT:
> BUT
> when mails are blocked, there are stored in deferred queue. And when
> Postfix "replay" these deferred mails, slow down does'nt apply.

THIS IS ABSOLUTELY INCORRECT. The Postfix delivery rate does not
depend on whether a message comes from the deferred queue (new mail)
or from the deferred queue.

However, if a site has multiple MXes, and one MX rejects mail with
4XX, then Postfix will immediately try an alternate MX.

You can limit the number of MX sessions to just one with:

slow       unix  -       -       y       -       -       smtp
    -o smtp_mx_session_limit=1
    -o smtp_mx_address_limit=5

With this Postfix will give up after the first session, but it will
still try five addresses if some address does not respond at all.

        Wietse

> That is :
> - when all goes well, mails are sent and accepted
> - when somethinfg is "wrong" (flooding from a user), IP is blocked
> - we change IP, and all goes well again, but as Postix replay the deferred
> queue, everything is again considered as spam and IP is BL, and again, and
> again...
>
> Why does slow down not apply to deferred replay ?
> Is it possible to prevent Postfix to "flush" all when it replays mail ?
>
> Regards,
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

gaia45500
Thanks for replying.

Ok no problem with this explanation, i'll try this smtp_mx_session_limit
parameter.
Howerver, and it has still happened few minutes ago, does it prevent Postfix
to "vomit" all deferred mails and make the IP blocke again ?

You say that slow donw apply wherever mails come from, ok. But the exact
behaviour i can watch is :
- when deferred queue is empty, with mails arriving from classic incoming
queue, sending is "delayed"
- when deferred queue has mails (even not many), Postfix requeue it, and IP
is BL few minutes later

It is possible that something is not tuned in our config, but what ?

Regards,



--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

Wietse Venema
gaia45500:
> Thanks for replying.
>
> Ok no problem with this explanation, i'll try this smtp_mx_session_limit
> parameter.
> Howerver, and it has still happened few minutes ago, does it prevent Postfix
> to "vomit" all deferred mails and make the IP blocke again ?

You have failed to set POSTFIX OUTPUT RATE LIMITS low enough that
Postfix cannot send too much mail.

Instead, you are relying on INPUT RATE LIMIT from your sending app
to control the Postfix output rate.

> You say that slow donw apply wherever mails come from, ok. But the exact
> behaviour i can watch is :
> - when deferred queue is empty, with mails arriving from classic incoming
> queue, sending is "delayed"
> - when deferred queue has mails (even not many), Postfix requeue it, and IP
> is BL few minutes later

That is because you are depending on the INPUT RATE LMIT from your
sending app, and you have not properly set the OUTPUT RATE CONTROLS
that are built into Postfix.

Once again,

You have failed to set POSTFIX OUTPUT RATE LIMITS low enough that
Postfix cannot send too much mail.

Instead, you are relying on INPUT RATE LIMIT from your sending app
to control the Postfix output rate.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

gaia45500
I can't do anything about sending app.
Considering i can only work on Postfix, and that it is exclusivly used to
send outgoing messaging for dedicated domains, i folled this

http://steam.io/2013/04/01/postfix-rate-limiting/

what could we do more ?
What should be other ways to control POSTFIX OUTPUT RATE LIMITS ?

Thanks for your time and help,



--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

Wietse Venema
gaia45500:
> I can't do anything about sending app.

You haven't properly configureed POSTFIX rate limits. You just got
lucky because the SENDING APP LIMITS kept you under the Yahoo limits.

> Considering i can only work on Postfix, and that it is exclusivly used to
> send outgoing messaging for dedicated domains, i folled this
>
> http://steam.io/2013/04/01/postfix-rate-limiting/
>
> what could we do more ?
> What should be other ways to control POSTFIX OUTPUT RATE LIMITS ?

1) main.cf:xxx_destination_recipient_limit not lower than 2

2) main.cf:xxx_transport_rate_delay = 1 or more

3) main.cf:xxx_destination_concurrency_limit does not matter if you
configure xxx_transport_rate_delay > 0.

4) (in master.cf, the xxx delivery agent) -o smtpd_mx_session_limit=1
(so that Postfix won't connect to another MX when an MX rejects
mail).

5) (in master.cf, the xxx delivery agent) -o smtpd_mx_address_limit
(so that Postfix won't make more than one TCP connection).

With that there is no way that Postfix can have more than one Yahoo
TCP connection per xxx_transport_rate_delay seconds.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

gaia45500
Many thanks for your explanations and your patience.



--
Sent from: http://postfix.1071664.n5.nabble.com/Postfix-Users-f2.html
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

@lbutlr
On 26 Jan 2020, at 23:19, gaia45500 <[hidden email]> wrote:
> Many thanks for your explanations and your patience.

While you have solved one problem, for now, you will almost certainly continue to have problems with yahoo because they are really bad at email.



--
"Oh damn", said Maladict.


Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

Phil Stracchino
On 2020-01-27 12:19, @lbutlr wrote:
> On 26 Jan 2020, at 23:19, gaia45500 <[hidden email]> wrote:
>> Many thanks for your explanations and your patience.
>
> While you have solved one problem, for now, you will almost certainly continue to have problems with yahoo because they are really bad at email.

Honestly, it is a source of continuing amazement, bafflement, and
mystified wonder to me that Yahoo continues to survive at all, in any
form.  They are not just bad at email, they are bad at EVERYTHING, and
they have leaked essentially all of the personal information of
essentially all of their users, multiple times.  And that was even
BEFORE they were bought (gods only know why) by Verizon.  (...Well, OK,
by a Verizon subsidiary.)

Honestly the best thing to do would have been to just let Yahoo DIE.
It's long past time.



--
  Phil Stracchino
  Babylon Communications
  [hidden email]
  [hidden email]
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

James Moe
In reply to this post by gaia45500
On 2020-01-26 12:57 PM, Emmanuel BILLOT wrote:

> status=deferred (host mx-eu.mail.am0.yahoodns.net
> Messages from X.X.X.X temporarily deferred due to user complaints
>
  It would seem you recipients do not appreciate your "massive" newsletter.
  I do not see how rate limiting would solve this user issue.

--
James Moe
moe dot james at sohnen-moe dot com
520.743.3936
Think.


signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Yahoo rate limit (again...)

@lbutlr
On 27 Jan 2020, at 10:41, James Moe <[hidden email]> wrote:
> On 2020-01-26 12:57 PM, Emmanuel BILLOT wrote:
>
>> status=deferred (host mx-eu.mail.am0.yahoodns.net
>> Messages from X.X.X.X temporarily deferred due to user complaints

>  It would seem you recipients do not appreciate your "massive" newsletter.
>  I do not see how rate limiting would solve this user issue.

Because Yahoo lies and gives an error claiming user complaints that is entirely made-up and has nothing to do with users.

Really, they are worthless and I made the decision when they had their FIRST massive breech to simply pretend they do not exist (My server will neither send nor receive mail from Yahoo.com) giving an error message explaining why.




--
"Are you pondering what I'm pondering?"
"I think so, Brain! But shouldn't we let the silk worms finish the
        boxer shorts before we put them on?”