Increasing number of connections?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Increasing number of connections?

Alex Regan
Hi,
I built a dual-Xeon quad-core box with 8GB using fedora15 and
postfix-v2.8.5 and during various times during the day connections to
port 25 timeout or are very slow. The majority of times this happens
is under peak loads, but even times when it's not at capacity it may
do this.

I have a similar box with lesser hardware, which I believe processes
as much mail, and it never occurs there, using very similar
configuration. Where should I look to troubleshoot something like
this?

In master.cf I have tried to adjust the number of smtp and smtpd
processes to between equal-to and double the number of processor
cores, but it doesn't seem to make any difference.

Could this be a TCP limit, or is it most assuredly a postfix limit?

I hope this isn't a FAQ because I've searched a bit, but confused by
all the potential options and not sure where to look specifically.
Tuning advice and general guidance would be appreciated. I've attached
my postfinger output below.

postfinger - postfix configuration on Wed Nov 16 20:52:22 EST 2011
version: 1.30

--System Parameters--
mail_version = 2.8.5
hostname = mail01.example.com
uname = Linux mail01.example.com 2.6.40.6-0.fc15.x86_64 #1 SMP Tue Oct
4 00:39:50 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

--Packaging information--
looks like this postfix comes from RPM package: postfix-2.8.5-1.fc15.x86_64

--main.cf non-default parameters--
alias_maps = hash:/etc/aliases
allow_mail_to_files = alias,forward
always_bcc = bcc-user
biff = no
body_checks = regexp:/etc/postfix/body_checks.pcre
content_filter = smtp-amavis:[127.0.0.1]:10024
default_process_limit = 140
delay_warning_time = 4h
disable_vrfy_command = yes
header_checks =
pcre:/etc/postfix/header_checks.pcre pcre:/etc/postfix/header_checks-jimsun.pcre
initial_destination_concurrency = 20
mailbox_command = /usr/bin/procmail
mailbox_size_limit = 2000000000
manpage_directory = /usr/share/man
maximal_queue_lifetime = 2d
message_size_limit = 13312000
mydestination = $myhostname, localhost.$mydomain
mynetworks = 127.0.0.0/8, 192.168.1.0/24, 192.168.6.0/24,
68.XXX.YYY.40/29, 64.XXX.YYY.0/27, 206.XXX.YYY.45/32,
206.XXX.ZZZ.45/32,160.XXX.YYY.1
rbl_reply_maps = ${stress?hash:/etc/postfix/rbl_reply_maps}
readme_directory = /usr/share/doc/postfix-2.8.5/README_FILES
relay_domains = $mydestination, $transport_maps, example1.com,
dom2.example.com, dom1.example.com, example.com
sample_directory = /usr/share/doc/postfix-2.8.5/samples
smtpd_recipient_restrictions =
reject_non_fqdn_recipient, check_client_access
hash:/etc/postfix/client_checks_special, check_sender_access
hash:/etc/postfix/sender_checks_special, reject_non_fqdn_sender, reject_unlisted_recipient, permit_mynetworks, reject_unauth_destination, reject_unknown_sender_domain, reject_unknown_recipient_domain, check_helo_access
pcre:/etc/postfix/helo_checks.pcre, reject_invalid_helo_hostname, check_client_access
hash:/etc/postfix/client_checks, check_sender_access
hash:/etc/postfix/sender_checks, check_recipient_access
pcre:/etc/postfix/relay_recips_segtravel, check_recipient_access
pcre:/etc/postfix/relay_recips_access, check_recipient_access
pcre:/etc/postfix/property_recip_map, check_recipient_access
pcre:/etc/postfix/recipient_checks, check_recipient_access
pcre:/etc/postfix/bwi_relay_recip_checks, check_recipient_access
pcre:/etc/postfix/relay_recips_ecartis, reject_rbl_client
zen.spamhaus.org, reject_rbl_client psbl.surriel.com, permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = $myhostname, mail01.example.com
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = reject_sender_login_mismatch
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/pki/dovecot/certs/dovecot.pem
smtpd_tls_key_file = /etc/pki/dovecot/private/dovecot.pem
smtpd_tls_received_header = yes
smtpd_tls_security_level = may
smtpd_tls_session_cache_database =
btree:/var/lib/postfix/smtpd_tls_session_cache
smtp_tls_CAfile = /etc/pki/tls/cacert.pem
smtp_use_tls = yes
transport_maps = hash:/etc/postfix/transport
virtual_alias_maps = hash:/etc/postfix/virtual,
hash:/etc/postfix/virtual-segtravel

--master.cf--
smtp      inet  n       -       n       -       -       smtpd
        -o receive_override_options=no_address_mappings
submission inet n       -       n       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
pickup    fifo  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
        -o smtp_fallback_relay=
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
smtp-amavis unix    -       -       n       -       6     smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
127.0.0.1:10025 inet n    -       n       -       6     smtpd
    -o content_filter=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o smtpd_restriction_classes=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
    -o local_header_rewrite_clients=
bwi    unix    -       -       n       -       -       smtp
        -o fallback_relay=[206.XXX.ZZZ.20]
csbwi    unix    -       -       n       -       -       smtp
        -o fallback_relay=[206.XXX.YYY.20]

-- end of postfinger output --

Thanks,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Noel Jones-2
On 11/16/2011 7:56 PM, Alex wrote:
> Hi,
> I built a dual-Xeon quad-core box with 8GB using fedora15 and
> postfix-v2.8.5 and during various times during the day connections to
> port 25 timeout or are very slow. The majority of times this happens
> is under peak loads, but even times when it's not at capacity it may
> do this.

Often slow smtpd connections are caused by not having enough smtpd
processes running.

On your hardware, postfix will support thousands of smtpd processes.
 Use netstat or lsof to see how many connections postfix is handling
when you experience slowdowns.
http://www.postfix.org/documentation.html
http://www.postfix.org/TUNING_README.html

If zombie spambots are using up most of your available connections,
postscreen will likely help.
http://www.postfix.org/POSTSCREEN_README.html

Other things to check for are your DNS speed and hard drive performance.

At any rate, you need to examine your system to find the bottleneck.



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Alex Regan
Hi,

>> I built a dual-Xeon quad-core box with 8GB using fedora15 and
>> postfix-v2.8.5 and during various times during the day connections to
>> port 25 timeout or are very slow. The majority of times this happens
>> is under peak loads, but even times when it's not at capacity it may
>> do this.
>
> Often slow smtpd connections are caused by not having enough smtpd
> processes running.
>
> On your hardware, postfix will support thousands of smtpd processes.
>  Use netstat or lsof to see how many connections postfix is handling
> when you experience slowdowns.

It's in the hundreds. There is also some amount of iowait, but I don't
think that's the issue.

When using amavisd-new, shouldn't the number of processes match the
number of smtpd processes?

I think what I'm concerned about is having postfix receive more
messages than amavisd can process?

> If zombie spambots are using up most of your available connections,
> postscreen will likely help.
> http://www.postfix.org/POSTSCREEN_README.html

Yes, looks like this would be a good thing to do is a general idea.

Thanks again,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Claudio Kuenzler-2
It's possible that amavisd slows down your postfix.
You can try to increase the number of amavis processes in the config:

$max_servers = 5;            # number of pre-forked children

The number of amavisd processes is independent of the smtp processes.

On Thu, Nov 17, 2011 at 7:15 AM, Alex <[hidden email]> wrote:
Hi,

>> I built a dual-Xeon quad-core box with 8GB using fedora15 and
>> postfix-v2.8.5 and during various times during the day connections to
>> port 25 timeout or are very slow. The majority of times this happens
>> is under peak loads, but even times when it's not at capacity it may
>> do this.
>
> Often slow smtpd connections are caused by not having enough smtpd
> processes running.
>
> On your hardware, postfix will support thousands of smtpd processes.
>  Use netstat or lsof to see how many connections postfix is handling
> when you experience slowdowns.

It's in the hundreds. There is also some amount of iowait, but I don't
think that's the issue.

When using amavisd-new, shouldn't the number of processes match the
number of smtpd processes?

I think what I'm concerned about is having postfix receive more
messages than amavisd can process?

> If zombie spambots are using up most of your available connections,
> postscreen will likely help.
> http://www.postfix.org/POSTSCREEN_README.html

Yes, looks like this would be a good thing to do is a general idea.

Thanks again,
Alex

Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Noel Jones-2
In reply to this post by Alex Regan
On 11/17/2011 12:15 AM, Alex wrote:

> Hi,
>
>>> I built a dual-Xeon quad-core box with 8GB using fedora15 and
>>> postfix-v2.8.5 and during various times during the day connections to
>>> port 25 timeout or are very slow. The majority of times this happens
>>> is under peak loads, but even times when it's not at capacity it may
>>> do this.
>>
>> Often slow smtpd connections are caused by not having enough smtpd
>> processes running.
>>
>> On your hardware, postfix will support thousands of smtpd processes.
>>  Use netstat or lsof to see how many connections postfix is handling
>> when you experience slowdowns.
>
> It's in the hundreds.

The question is: is the number of connections your system is
handling at peak nearly equal to the number of connections
configured?  If yes, then you need to configure more connections.

If many of the connections are spambots, postscreen will help.
That's what it is designed for.

> When using amavisd-new, shouldn't the number of processes match the
> number of smtpd processes?

When using amavisd-new as a content_filter, the number of postfix
smtp->amavisd feeder processes should be equal to (or maybe one less
than for monitoring) the number of amavisd processes.  This is
independent of the number of smtpd input processes.


> I think what I'm concerned about is having postfix receive more
> messages than amavisd can process?

Yes, that's a concern when using any content_filter, but really a
separate issue.

If you limit the input by using too few smtpd processes, connecting
clients will get timeouts.

Your system should handle 20~30 amavisd-new processes.  Check the
amavisd-new and spamassassin documentation and user lists for
performance tips (90%+ of amavisd-new processing time is spent in
spamassassin).



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Jeroen Geilman
In reply to this post by Alex Regan
On 2011-11-17 07:15, Alex wrote:

> Hi,
>
>>> I built a dual-Xeon quad-core box with 8GB using fedora15 and
>>> postfix-v2.8.5 and during various times during the day connections to
>>> port 25 timeout or are very slow. The majority of times this happens
>>> is under peak loads, but even times when it's not at capacity it may
>>> do this.
>> Often slow smtpd connections are caused by not having enough smtpd
>> processes running.
>>
>> On your hardware, postfix will support thousands of smtpd processes.
>>   Use netstat or lsof to see how many connections postfix is handling
>> when you experience slowdowns.
> It's in the hundreds. There is also some amount of iowait, but I don't
> think that's the issue.
>
> When using amavisd-new, shouldn't the number of processes match the
> number of smtpd processes?

Hell no, amavisd can kill your system dead.
It will take 100MB per process easily, and each of these takes much more
time to complete than any comparable SMTP transaction - or postfix queue
process.
If you are using more than about 50 amavisd threads, you'll be depleting
those 8GB very quickly.

As said previously, postscreen to stop the 90% spam connections, a few
hundred smtpds, and 50 or so amavisd threads should be doable.

Mail will be queued before amavis can get to it, but that is the nature
of the beast - 8GB is very little memory for a modern server system.

Max out the board to 32 or 48GB, whatever it takes - it will cost far
less than any other solution.


--
J.

Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Henrik K
On Thu, Nov 17, 2011 at 07:46:26PM +0100, Jeroen Geilman wrote:
>
> Hell no, amavisd can kill your system dead.
> It will take 100MB per process easily, and each of these takes much

Terrible misinformation. Amavisd-new preloads pretty much everything before
forking, which means childs just share that common 100MB chunk
(copy-on-write). Additional memory is only consumed when child is actively
processing something (maybe 10MB tops).

Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Alex Regan
In reply to this post by Noel Jones-2
Hi,

>> When using amavisd-new, shouldn't the number of processes match the
>> number of smtpd processes?
>
> When using amavisd-new as a content_filter, the number of postfix
> smtp->amavisd feeder processes should be equal to (or maybe one less
> than for monitoring) the number of amavisd processes.  This is
> independent of the number of smtpd input processes.

I think I may be still a bit confused.

Can you confirm that this is the relevant section I should be
adjusting to prevent timeouts connecting to port 25:

smtp      inet  n       -       n       -       -       smtpd
        -o receive_override_options=no_address_mappings

I also increased the number of amavisd feeder processes to 24 today,
and it seemed to handle the load:

smtp-amavis unix    -       -       n       -       24     smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

127.0.0.1:10025 inet n    -       n       -       24     smtpd
    -o content_filter=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o smtpd_restriction_classes=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
    -o local_header_rewrite_clients=

Thanks again,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Noel Jones-2
On 11/17/2011 7:37 PM, Alex wrote:

> Hi,
>
>>> When using amavisd-new, shouldn't the number of processes match the
>>> number of smtpd processes?
>>
>> When using amavisd-new as a content_filter, the number of postfix
>> smtp->amavisd feeder processes should be equal to (or maybe one less
>> than for monitoring) the number of amavisd processes.  This is
>> independent of the number of smtpd input processes.
>
> I think I may be still a bit confused.
>
> Can you confirm that this is the relevant section I should be
> adjusting to prevent timeouts connecting to port 25:
>
> smtp      inet  n       -       n       -       -       smtpd
>         -o receive_override_options=no_address_mappings

Yes, adjust the above service to control how many network smtpd
listeners you have.


>
> I also increased the number of amavisd feeder processes to 24 today,
> and it seemed to handle the load:
>
> smtp-amavis unix    -       -       n       -       24     smtp
>...

The above controls the number of feeder processes from postfix to
amavisd.  This number should match the number of amavisd-new
processes (or optionally one less than the amavisd-new process
limit).  This number should never be higher than the number of
amavisd-new processes.


> 127.0.0.1:10025 inet n    -       n       -       24     smtpd
> ...

This smtpd listener is for amavisd-new to reinject mail back to
postfix.  The process limit on this service must be equal to or
greater than the number of amavisd-new processes.  You can safely
leave this service's process count at the "-" default setting
(assuming your main.cf $default_process_limit (default 100) is
greater than the number of amavisd-new processes).


  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Increasing number of connections?

Alex Regan
Hi,

>> Can you confirm that this is the relevant section I should be
>> adjusting to prevent timeouts connecting to port 25:
>>
>> smtp      inet  n       -       n       -       -       smtpd
>>         -o receive_override_options=no_address_mappings
>
> Yes, adjust the above service to control how many network smtpd
> listeners you have.
[snipped]

Thanks so much for your help.
Best,
Alex