Postscreen concurrency limits

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Postscreen concurrency limits

Alex Regan
Hi,

We had a Mimecast user report today that their mail was being rejected
with a 4.7.0 "too many connections" error. This is a "soft" error, in
that the mail client will later attempt to resend, correct?

Isn't the default of 50 concurrent connections sufficient for most
environments? Is there really any reason I should consider increasing
that limit? Isn't it a bit much for them to expect to be able to make
50 separate connections in the first place? Maybe those are subjective
questions, but I'm interested in what people normally do.

Here are a few of the logs from both postscreen and smtpd.

Dec 13 17:01:18 mail03 postfix/postscreen[15590]: NOQUEUE: reject:
CONNECT from [216.205.24.100]:33254: too many connections
Dec 13 17:01:18 mail03 postfix/postscreen[15590]: DISCONNECT
[216.205.24.100]:33254
Dec 13 17:01:18 mail03 postfix/smtpd[27153]: connect from
us-smtp-delivery-100.mimecast.com[216.205.24.100]
Dec 13 17:01:18 mail03 postfix/smtpd[27141]: connect from
us-smtp-delivery-100.mimecast.com[216.205.24.100]
Dec 13 17:01:18 mail03 postfix/smtpd[27153]: warning: Connection
concurrency limit exceeded: 5 from us-smtp-delivery-100.mimecast.com[2
16.205.24.100] for service smtpd
Dec 13 17:01:18 mail03 postfix/smtpd[27141]: warning: Connection
concurrency limit exceeded: 6 from us-smtp-delivery-100.mimecast.com[2
16.205.24.100] for service smtpd

Are there other parameters I should consider? This is with postfix-3.3.1.

postconf -d|grep smtpd_client_connection_count_limit
postscreen_client_connection_count_limit = $smtpd_client_connection_count_limit
smtpd_client_connection_count_limit = 50

Thanks,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Viktor Dukhovni


> On Dec 13, 2018, at 8:25 PM, Alex <[hidden email]> wrote:
>
> We had a Mimecast user report today that their mail was being rejected
> with a 4.7.0 "too many connections" error. This is a "soft" error, in
> that the mail client will later attempt to resend, correct?

Should be.

> Isn't the default of 50 concurrent connections sufficient for most
> environments? Is there really any reason I should consider increasing
> that limit?

Do read the log you post...

> Dec 13 17:01:18 mail03 postfix/smtpd[27153]: warning: Connection
> concurrency limit exceeded: 5 from us-smtp-delivery-100.mimecast.com[2
> 16.205.24.100] for service smtpd

5 < 50.

> Dec 13 17:01:18 mail03 postfix/smtpd[27141]: warning: Connection
> concurrency limit exceeded: 6 from us-smtp-delivery-100.mimecast.com[2
> 16.205.24.100] for service smtpd

6 < 50.

> postconf -d|grep smtpd_client_connection_count_limit
> postscreen_client_connection_count_limit = $smtpd_client_connection_count_limit
> smtpd_client_connection_count_limit = 50

The "postconf -d" command reports compiled-in defaults, not your
settings, which are reported with "postconf -n" or just "postconf"
if you also want to see any unchanged defaults.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

allenc


On 14/12/2018 06:13, Viktor Dukhovni wrote:

>
>
>> On Dec 13, 2018, at 8:25 PM, Alex <[hidden email]> wrote:
>>
>> We had a Mimecast user report today that their mail was being rejected
>> with a 4.7.0 "too many connections" error. This is a "soft" error, in
>> that the mail client will later attempt to resend, correct?
>
> Should be.
>
>> Isn't the default of 50 concurrent connections sufficient for most
>> environments? Is there really any reason I should consider increasing
>> that limit?
>
> Do read the log you post...
>
>> Dec 13 17:01:18 mail03 postfix/smtpd[27153]: warning: Connection
>> concurrency limit exceeded: 5 from us-smtp-delivery-100.mimecast.com[2
>> 16.205.24.100] for service smtpd
>
> 5 < 50.

I have a hunch that this is an excess count.  Some while ago I had a mini-DoS attack.  As I recall, the server accepted
the first 50 connections and then started counting the refusals.

Earlier log entries should prove it one way or the other.

Allen C
Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Viktor Dukhovni


> On Dec 14, 2018, at 4:27 PM, Allen Coates <[hidden email]> wrote:
>
> I have a hunch that this is an excess count.

It is not.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Alex Regan
Hi,

On Sat, Dec 15, 2018 at 1:42 PM Viktor Dukhovni
<[hidden email]> wrote:
> > On Dec 14, 2018, at 4:27 PM, Allen Coates <[hidden email]> wrote:
> >
> > I have a hunch that this is an excess count.
>
> It is not.

The issue was that I had one mail host with the parameter set to 5
while the one I checked did not have it set at all, resulting it being
set to the default of 50. I didn't expect that they were different.

I've reverted the one host back to the default. The original reason I
had set it in the first place was to try and control the amount of
email the bulk senders like constantcontact, mailchimp, etc, could
send at once, filling our queues with thousands of messages at once.
This does not appear to be that parameter :-)

Ideas for better throttling these bulk senders would be appreciated.

Thanks,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Viktor Dukhovni
On Mon, Dec 17, 2018 at 10:47:02AM -0500, Alex wrote:

> The original reason I had set it in the first place was to try and control
> the amount of email the bulk senders like constantcontact, mailchimp, etc,
> could send at once, filling our queues with thousands of messages at once.
> This does not appear to be that parameter :-)

Inbound email should not generally result in queue congestion, it
is typically delivered to mailboxes faster than it arrives from
the remote system.  Are you accepting email for non-existent
recipients and sending bounces?  Best to not do that...

> Ideas for better throttling these bulk senders would be appreciated.

Throttling is often counter-productive,  it just means that even
more email arrives later.  The examples you give are not botnet
email senders, but mailstream commercial email delivery companies,
so they honour "unsubscribe".  If you're receiving unwanted email
from them, you or your users should unsubscribe.

Postfix rate limits are not an anti-spam mechanism, rather they are
safety mechanisms for accidental DoS by one or a few clients when
buggy software sends email in a tight loop.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Alex Regan
Hi,

On Mon, Dec 17, 2018 at 12:18 PM Viktor Dukhovni
<[hidden email]> wrote:

>
> On Mon, Dec 17, 2018 at 10:47:02AM -0500, Alex wrote:
>
> > The original reason I had set it in the first place was to try and control
> > the amount of email the bulk senders like constantcontact, mailchimp, etc,
> > could send at once, filling our queues with thousands of messages at once.
> > This does not appear to be that parameter :-)
>
> Inbound email should not generally result in queue congestion, it
> is typically delivered to mailboxes faster than it arrives from
> the remote system.  Are you accepting email for non-existent
> recipients and sending bounces?  Best to not do that...

No, we're not doing that. We have check_recipient_access maps that
contain a list of all valid users.

The problem is that this is a domain with thousands of recipients, and
mailchimp and others send mass newsletters to thousands of those
recipients at once to our relyhosts, which first scan the emails for
spam/viruses and only then forward on. This causes somewhat of a
significant delay at times as the server processes all the emails.

I recall reading that more recently people are configuring their mail
systems to scan the email as it's received without queuing it, and I
assume reject temporarily any mail that can't be processed at the time
it's received?

Thanks,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Viktor Dukhovni
> On Dec 18, 2018, at 6:48 PM, Alex <[hidden email]> wrote:
>
> The problem is that this is a domain with thousands of recipients, and
> mailchimp and others send mass newsletters to thousands of those
> recipients at once to our relyhosts, which first scan the emails for
> spam/viruses and only then forward on. This causes somewhat of a
> significant delay at times as the server processes all the emails.
>
> I recall reading that more recently people are configuring their mail
> systems to scan the email as it's received without queuing it, and I
> assume reject temporarily any mail that can't be processed at the time
> it's received?

So you have a CPU-limited throughput cap that is currently post-queue,
so that mail can arrive faster than it can be scanned.

Throughput = concurrency / latency, and the problem is that the
input stage has low latency, and can accept mail quickly, but
at the same concurrency, the filter stage has higher (CPU-bound)
latency, and noticeably lower throughput.

The solution is perhaps in part to throw some more CPU at the
problem, but alternatively, assuming that mailchimp et. al.
are not abusing reasonable concurrency limits, you can reduce
the impedance mismatch by increasing the input latency, by
doing the A/V scan "on-line' via a milter.

This does carry a greater risk of overall DoS (all your
SMTP servers busy competing for limited CPU), but by
reducing the throughput per concurrent connection, and
assuming no attack, a combination of some more CPU,
and pre-queue A/V scanning might do the job.

You could attempt per-client message rate limits, or
slightly lower per-client concurrency limits, but these
can backfire if set too low, by just deferring ever more
traffic if you make the pipe too narrow.

And of course, if possible, encourage your users to subscribe
to less junk mail.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

@lbutlr
On 18 Dec 2018, at 16:58, Viktor Dukhovni <[hidden email]> wrote:
> The solution is perhaps in part to throw some more CPU at the
> problem, but alternatively, assuming that mailchimp et. al.
> are not abusing reasonable concurrency limits, you can reduce
> the impedance mismatch by increasing the input latency, by
> doing the A/V scan "on-line' via a milter.

Am I wrong in thinking that doing an A/V scan on mail from Mailchimp and/or cosntantcontact is a waste of time?

They are not sending viruses. Hell, they are not even sending spam.

--
What the hell's goin' on in the engine room? Were there monkeys? Some
terrifying space monkeys maybe got loose?

Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

Viktor Dukhovni
> On Dec 20, 2018, at 1:04 PM, @lbutlr <[hidden email]> wrote:
>
> Am I wrong in thinking that doing an A/V scan on mail from Mailchimp and/or cosntantcontact is a waste of time?
>
> They are not sending viruses. Hell, they are not even sending spam.

Viruses can come from any source.  And message origin authentication
is fragile.  But if you find a way to whitelist them that's not easily
spoofed and works for you, that's your call...

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Postscreen concurrency limits

@lbutlr
On 20 Dec 2018, at 11:08, Viktor Dukhovni <[hidden email]> wrote:
> Viruses can come from any source.

OK, But I am pretty sure I’ve never seen a virus from mail chimp.

I don’t have a large enough load to worry about not scanning, but if I did the first thing I would stop scanning is gmail incoming and the large remailers like mail chimp as they both are highly motivated not to send viruses.

Granted, since most my mail accounts use Macs, I care a lot less.

--
Honesty may be the best policy, but it's important to remember that
apparently, by elimination, dishonesty is the second-best policy.