Comcast 421 throttling multiple recipients

[Bob Proulx]

Question about a different system.  Pretty much every question of mine
is related to a different oddball case.  Here I am helping a friend
out and they encountered this problem.  I'll change the 3rd party
addresses so as not to annoy them but the data is otherwise
verbatim.

    Sep 23 14:38:23 yuki postfix/smtp[21081]: CDC7E1438D: to=<[hidden email]>, relay=mx2.comcast.net[68.87.20.5]:25, delay=2.7, delays=0.08/0.06/2.5/0.05, dsn=4.1.0, status=deferred (host mx2.comcast.net[68.87.20.5] said: 421 4.1.0 104.200.24.137 Throttled - try again later. Please see http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
    Sep 23 14:38:23 yuki postfix/smtp[21081]: CDC7E1438D: to=<[hidden email]>, relay=mx2.comcast.net[68.87.20.5]:25, delay=2.7, delays=0.08/0.06/2.5/0.05, dsn=4.1.0, status=deferred (host mx2.comcast.net[68.87.20.5] said: 421 4.1.0 104.200.24.137 Throttled - try again later. Please see http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
    Sep 23 14:38:23 yuki postfix/smtp[21081]: CDC7E1438D: to=<[hidden email]>, relay=mx2.comcast.net[68.87.20.5]:25, delay=2.7, delays=0.08/0.06/2.5/0.05, dsn=4.1.0, status=deferred (host mx2.comcast.net[68.87.20.5] said: 421 4.1.0 104.200.24.137 Throttled - try again later. Please see http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))

This was one email sent to three recipients where all three just
happened to be at comcast.net and therefore they were grouped into one
SMTP transaction.  All seemingly normal.  But apparently Comcast has
decided to throttle based upon this.

My friend poked around and eventually deleted this message from the
queue and sent three messages one to each recipent and Comcast
accepted each of those three immediately.  Therefore we conclude that
Comcast does not like multiple recipients in the same transaction.
Anyone think otherwise?

It seems that the way to add limits when sending to Comcast would be
to set up a transport map such as perhaps this.  (And knowing that
Yerdle is a turtle.)

    comcast.net  yerdle:

Then add to the master.cf file.

    smtp       unix  - - - - - smtp
    yerdle     unix  - - - - - smtp

And add this to the main.cf file.

    yerdle_destination_concurrency_limit = 1
    yerdle_destination_rate_delay = 1s
    yerdle_destination_recipient_limit = 1

That it would allow one connection at a time, with one recipient per
message at a time, and then a small delay between sending of messages.

Does that seem about right?

Bob

[Viktor Dukhovni]

On Thu, Sep 24, 2020 at 12:59:44AM -0600, Bob Proulx wrote:

> Question about a different system.  Pretty much every question of mine
> is related to a different oddball case.  Here I am helping a friend
> out and they encountered this problem.  I'll change the 3rd party
> addresses so as not to annoy them but the data is otherwise
> verbatim.
>
>     Sep 23 14:38:23 yuki postfix/smtp[21081]: CDC7E1438D: to=<[hidden email]>, relay=mx2.comcast.net[68.87.20.5]:25, delay=2.7, delays=0.08/0.06/2.5/0.05, dsn=4.1.0, status=deferred (host mx2.comcast.net[68.87.20.5] said: 421 4.1.0 104.200.24.137 Throttled - try again later. Please see http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
>     Sep 23 14:38:23 yuki postfix/smtp[21081]: CDC7E1438D: to=<[hidden email]>, relay=mx2.comcast.net[68.87.20.5]:25, delay=2.7, delays=0.08/0.06/2.5/0.05, dsn=4.1.0, status=deferred (host mx2.comcast.net[68.87.20.5] said: 421 4.1.0 104.200.24.137 Throttled - try again later. Please see http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
>     Sep 23 14:38:23 yuki postfix/smtp[21081]: CDC7E1438D: to=<[hidden email]>, relay=mx2.comcast.net[68.87.20.5]:25, delay=2.7, delays=0.08/0.06/2.5/0.05, dsn=4.1.0, status=deferred (host mx2.comcast.net[68.87.20.5] said: 421 4.1.0 104.200.24.137 Throttled - try again later. Please see http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))

Look carefully at the log entry.  The "421" is send in response to "MAIL
FROM", not "RCPT TO".  So the recipient limit does not look entirely
plausible.  A good test would be to disable "pipelining" in a custom
smtp(8) transport, and use that for Comcast.  That would definitely
rule out recipient count limits if the reject is still at "MAIL FROM".

Furthermore, looking at (the reject did say "Please see ..."

    http://postmaster.comcast.net/smtp-error-codes.php#RL000001

suggests a reputation issue and rate limits based on that.  I don't see
anything about recipient concurrency.

> It seems that the way to add limits when sending to Comcast would be
> to set up a transport map such as perhaps this.  (And knowing that
> Yerdle is a turtle.)

Since that's not demonstrated (or very likely) to be the problem,
changing the recipient limits is NOT a good idea.

>     smtp       unix  - - - - - smtp
>     yerdle     unix  - - - - - smtp
>
> And add this to the main.cf file.
>
>     yerdle_destination_concurrency_limit = 1
>     yerdle_destination_rate_delay = 1s
>     yerdle_destination_recipient_limit = 1

A recipient limit of 1 is "magic" and will not do what you want,
instead, it will drive up delivery concurrency, making the
problem likely worse.

> That it would allow one connection at a time, with one recipient per
> message at a time, and then a small delay between sending of messages.

The problem is the IP reputation...

--
    Viktor.

[Bob Proulx]

Viktor Dukhovni wrote:
> Bob Proulx wrote:
> > ... http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
>
> Look carefully at the log entry.  The "421" is send in response to "MAIL
> FROM", not "RCPT TO".  So the recipient limit does not look entirely
> plausible.

If this was other than Comcast I would accept that immediately at face
value but a long history says that Comcast is an unreliable narrator
in this story.  :-(

Also the empirical testing showed that forcing one recipient per
message succeeds while three per message was rejected.  Interlaced.

So while Comcast may be putting sites into buckets and some buckets
are allowed to send to three at the same time and others are not
knowing what kind of rule is being applied does not help work with it.

> A good test would be to disable "pipelining" in a custom
> smtp(8) transport, and use that for Comcast.  That would definitely
> rule out recipient count limits if the reject is still at "MAIL FROM".

Sounds like a good thing to test.  Will give that a try.

> Furthermore, looking at (the reject did say "Please see ..."
>
>     http://postmaster.comcast.net/smtp-error-codes.php#RL000001
>
> suggests a reputation issue and rate limits based on that.  I don't see
> anything about recipient concurrency.

Comcast is an unreliable narrator.  Same site and same sender can
reproduce the problem by sending individually or to three recipients
and the messages to three recipients are not accepted.

And these messages are not bulk email.  My friend is just talking
about movies with other friends.

> > That it would allow one connection at a time, with one recipient per
> > message at a time, and then a small delay between sending of messages.
>
> The problem is the IP reputation...

I agree with you.  Because of lack of volume Comcast appears to be
rejecting multiple recipients but allowing single recipients.  Since
single recipients are being accepted okay.

But then how do we configure Postfix to do this automatically so that
we can gain enough reputation to send more than one recipient at a
time?  Because Comcast is not rejecting all mail.  Comcast is only
rejecting mail with multiple recipients.  Comcast is accepting mail
with single recipients.  It's an annoying situation.

Will try a test with pipelining disabled.

Bob

[techlist06-2]

>But then how do we configure Postfix to do this automatically so that
>we can gain enough reputation to send more than one recipient at a
>time?  Because Comcast is not rejecting all mail.  Comcast is only
>rejecting mail with multiple recipients.  Comcast is accepting mail
>with single recipients.  It's an annoying situation.

Bob, I have the same problem with them.  I ran down the lead through their reputation provider, which is mentioned somewhere in their referring URL.  I just don't have enough volume to maintain a steady reputation of more than neutral.  They (CC) have a table published somewhere that shows the rate they will accept relative to your reputation from the third party reputation company.  I lowered how much I push to them at once, and I try to make sure I send to them at least once a month to keep my "reputation" with the 3rd party high enough.  My mail probably goes to a few hundred CC recipients.  It's a ~3k member opt-in maillist for a racing organization, no ads, no links, clean announcements only, like for race cancellations, or reminders for a race that weekend, rule changes, etc.  I used to even send in plain text but read somewhere that was bad and would cause a ding, so I do HTML now.

Comcast appears to have washed their hands of it and just puts their full trust in the other company's provided reputation.  And the other company isn't counting little guys much.  No way to beg for a whitelist I can tell.

S

[Viktor Dukhovni]

On Thu, Sep 24, 2020 at 02:06:05PM -0600, Bob Proulx wrote:

> > > ... http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
> >
> > Look carefully at the log entry.  The "421" is send in response to "MAIL
> > FROM", not "RCPT TO".  So the recipient limit does not look entirely
> > plausible.
>
> If this was other than Comcast I would accept that immediately at face
> value but a long history says that Comcast is an unreliable narrator
> in this story.  :-(

The (in reply to MAIL FROM command) is added by Postfix, not Comcast, so
that is not subject to doubt.  Because the recipients were not yet
processed when "MAIL FROM" is received, it seems unlikely that the
recipient count is the issue, rather simply trying later solved the
problem.

The Comcast message is pretty clear, they're rate limiting on total
recipients per unit time, based on IP reputation.  I see no evidence
of hiding anything, the link in question publishes the numbers, ...

Some legitimate senders may find it difficult to deliver the email they
expect to send this policy, but that does not mean that the policy is
any way something other than what it appears to be.

> Also the empirical testing showed that forcing one recipient per
> message succeeds while three per message was rejected.  Interlaced.

Which only proves that sending at some later time did not run into the
rate limit.  Nothing else.

> So while Comcast may be putting sites into buckets and some buckets
> are allowed to send to three at the same time and others are not
> knowing what kind of rule is being applied does not help work with it.

There is zero evidence that per message recipient counts are pertinent,
unless the rate limit quantum goes all the way down to 1 recipient
per unit time, in which case you have a bit of a problem.

> > A good test would be to disable "pipelining" in a custom
> > smtp(8) transport, and use that for Comcast.  That would definitely
> > rule out recipient count limits if the reject is still at "MAIL FROM".
>
> Sounds like a good thing to test.  Will give that a try.

Do the pipelining test.  The first recipient should be accepted, but
then if the subsequent recipients "421" (instead of 450 or similar)
that'd be unfortunate, I might reach out to comcast email engineers for
a chat about that...

> Comcast is an unreliable narrator.

That's hyperbole.  There's no evidence for that.

> I agree with you.  Because of lack of volume Comcast appears to be
> rejecting multiple recipients but allowing single recipients.

You have no evidence of that, indeed the evidence so far argues
against this.

> But then how do we configure Postfix to do this automatically so that
> we can gain enough reputation to send more than one recipient at a
> time?

When you set a recipient limit of 1, you get three concurrent messages
with one recipient, rather than 1 message with three recipients, and you
also break the destination concurrency limits, which become per-user
rather than per-domain.

Do not set the recipient limit below 2, unless you also have a dedicated
single-domain transport, with a low process limit in master.cf.

> Because Comcast is not rejecting all mail.  Comcast is only
> rejecting mail with multiple recipients.

Again, this is speculative, and not presently supported by the
evidence.

--
    Viktor.

[Bob Proulx]

Viktor Dukhovni wrote:

> Bob Proulx wrote:
> > > > ... http://postmaster.comcast.net/smtp-error-codes.php#RL000001 (in reply to MAIL FROM command))
> > >
> > > Look carefully at the log entry.  The "421" is send in response to "MAIL
> > > FROM", not "RCPT TO".  So the recipient limit does not look entirely
> > > plausible.
> >
> > If this was other than Comcast I would accept that immediately at face
> > value but a long history says that Comcast is an unreliable narrator
> > in this story.  :-(
>
> The (in reply to MAIL FROM command) is added by Postfix, not Comcast, so
> that is not subject to doubt.  Because the recipients were not yet
> processed when "MAIL FROM" is received, it seems unlikely that the
> recipient count is the issue, rather simply trying later solved the
> problem.

But without the results from trying it without pipelining then don't
we not yet know exactly when the rejection occurred?  Because of
pipelining the entire set will be sent and then the results from the
set returned?

I haven't heard back the results from the experiment with turning off
pipelining.  So I can't make a report either way on the result.

> The Comcast message is pretty clear, they're rate limiting on total
> recipients per unit time, based on IP reputation.  I see no evidence
> of hiding anything, the link in question publishes the numbers, ...

I don't see how three recipients in an hour can in any way exceed
their published rate limits.  That's the evidence that they are doing
something that they have not published.

> Some legitimate senders may find it difficult to deliver the email they
> expect to send this policy, but that does not mean that the policy is
> any way something other than what it appears to be.

They publish that with an "N/A" SenderScore that they will accept 120
recipients per hour.

    https://postmaster.comcast.net/smtp-error-codes.php#RL000001

Sending 3 recipients per hour, or day, or week in this case, is well
below that published threshold.  In the category of my friend's IP
reputation likely having a "N/A" score it should be allowed to send up
to 120 recipients per hour.  There is no way that three would exceed
this limit.

> > Also the empirical testing showed that forcing one recipient per
> > message succeeds while three per message was rejected.  Interlaced.
>
> Which only proves that sending at some later time did not run into the
> rate limit.  Nothing else.

Sorry if I failed to describe this adequately.  Let me describe this
again.

The three-recipient messages were always given a 421 even after the
single-recipient messages were accepted.  We could send test messages
with a single recipient and they were immediately accepted without
delay.  Repeatedly.  But multiple attempts to send three-recipient
messages resulted in a 421 response, and those messages staying in the
queue and retrying until message expiration.  After the
single-recipient messages were accepted.

I understand a 421 is not a permanent failure.  But retrying until the
message expires and generates a bounce notification is still not a
successful delivery.  Over the course of a couple of days this was the
timeline.

    1. Three recipients: 421 repeatedly
    2. Single recipients: status=sent immediately
    3. Three recipients: 421 repeatedly
    4. Single recipients: status=sent immediately

Seems like at some level they are bucketing senders allowing the
sending of single-recipient messages but blocking multiple-recipient
messages.

And 3 is less than their published 120 / hour lowest rate limit.  Even
with all of the testing we are in the range of a dozen or twenty
messages in the logs over the last week or so now so definitely this
would be a very low volume server by any measurement.

Any proposed failure model must include machinery such as to account
for this type of behavior or it does not match the behavior that has
been experimentally seen.

> > So while Comcast may be putting sites into buckets and some buckets
> > are allowed to send to three at the same time and others are not
> > knowing what kind of rule is being applied does not help work with it.
>
> There is zero evidence that per message recipient counts are pertinent,
> unless the rate limit quantum goes all the way down to 1 recipient
> per unit time, in which case you have a bit of a problem.

I'll emphasize that the evidence did show this as I repeated above.

And yes I think it does show that we have a bit of a problem. :-(

> > > A good test would be to disable "pipelining" in a custom
> > > smtp(8) transport, and use that for Comcast.  That would definitely
> > > rule out recipient count limits if the reject is still at "MAIL FROM".
> >
> > Sounds like a good thing to test.  Will give that a try.
>
> Do the pipelining test.  The first recipient should be accepted, but
> then if the subsequent recipients "421" (instead of 450 or similar)
> that'd be unfortunate, I might reach out to comcast email engineers for
> a chat about that...

I do not have any results from the pipeline test but will push my
friend to do the test.

However as a workaround we have arranged to relay the very few
messages out through my server instead.  I have enough traffic to
Comcast servers that by deduction the SenderScore IP reputation on my
outbound must be non-zero in their table somewhere by the traffic
counts I can see in my logs.  And is not currently being rate limited.
Therefore it's possible that we will stop poking at things.  Since we
have a workaround.

But it would still be nice to have a good Postfix documented Best
Practice for dealing with situations such as this one.

> > Comcast is an unreliable narrator.
>
> That's hyperbole.  There's no evidence for that.

I agree that was my opinion based upon long experience.  But ask
anyone who has had to deal with Comcast over years and I believe this
would be the majority opinion.  Search the web for ratings of their
customer service and see what you find.  It's not good!

And actually I received direct mail responses from a few others who
read this list that commiserated with me and related similar problems.
And suggested their own workarounds they have implemented to deal with
the issue.  It's comforting to know we are not the only ones with this
problem.

Having things mostly resolved on our end now though I am going to go
back to letting sleeping dogs lie on this again.

Bob