Configure Postfix for High Volume

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Configure Postfix for High Volume

Tech Gurus
Hello,

I have single mail server that send relatively large amounts of emails at least 3 times a day ranging from 15K to 50K each time ..

80% of emails are going to one domain owned by my company (Domain1).. The current mail flow does around 1K-1.2K per minute , this is CentOS 7 VM ( 4 CPU/8GB Memory) running on SSD Datastore.

Upon digging into logs, I found lots of  postfix/anvil statistics DNS lookup misses for (Domain1) , I made some changes which produced small improvements (AVG/Minute went from 1K ~ 1.2K)..

Any guidance is appreciated

Thanks

Jim

The changes below produced small improvements

- Created static transport for internal domain to forward email to local "VIP" ,VIP RR to 8 servers behind , the 8 servers are the MX host for the company owned domain.
- Changed default_destination_concurrency_limit to 100 and default_destination_recipient_limit = 1500

Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

anvartay
What version of postfix are you using?

Anvar Kuchkartaev 
[hidden email]
From: Tech Gurus
Sent: jueves, 25 de enero de 2018 21:50
Subject: Configure Postfix for High Volume

Hello,

I have single mail server that send relatively large amounts of emails at least 3 times a day ranging from 15K to 50K each time ..

80% of emails are going to one domain owned by my company (Domain1).. The current mail flow does around 1K-1.2K per minute , this is CentOS 7 VM ( 4 CPU/8GB Memory) running on SSD Datastore.

Upon digging into logs, I found lots of  postfix/anvil statistics DNS lookup misses for (Domain1) , I made some changes which produced small improvements (AVG/Minute went from 1K ~ 1.2K)..

Any guidance is appreciated

Thanks

Jim

The changes below produced small improvements

- Created static transport for internal domain to forward email to local "VIP" ,VIP RR to 8 servers behind , the 8 servers are the MX host for the company owned domain.
- Changed default_destination_concurrency_limit to 100 and default_destination_recipient_limit = 1500


Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Tech Gurus
I am using 2.10.1-6 RPM.

On Thu, Jan 25, 2018 at 6:17 PM, Anvar Kuchkartaev <[hidden email]> wrote:
What version of postfix are you using?

Anvar Kuchkartaev 
[hidden email]
From: Tech Gurus
Sent: jueves, 25 de enero de 2018 21:50
Subject: Configure Postfix for High Volume

Hello,

I have single mail server that send relatively large amounts of emails at least 3 times a day ranging from 15K to 50K each time ..

80% of emails are going to one domain owned by my company (Domain1).. The current mail flow does around 1K-1.2K per minute , this is CentOS 7 VM ( 4 CPU/8GB Memory) running on SSD Datastore.

Upon digging into logs, I found lots of  postfix/anvil statistics DNS lookup misses for (Domain1) , I made some changes which produced small improvements (AVG/Minute went from 1K ~ 1.2K)..

Any guidance is appreciated

Thanks

Jim

The changes below produced small improvements

- Created static transport for internal domain to forward email to local "VIP" ,VIP RR to 8 servers behind , the 8 servers are the MX host for the company owned domain.
- Changed default_destination_concurrency_limit to 100 and default_destination_recipient_limit = 1500



Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Viktor Dukhovni
In reply to this post by Tech Gurus
On Thu, Jan 25, 2018 at 03:50:24PM -0500, Tech Gurus wrote:

> I have single mail server that send relatively large amounts of emails at
> least 3 times a day ranging from 15K to 50K each time ..

This is in general terms not a lot of mail for Postfix to handle.
Medium volume Postfix servers run at a leisurely sustained pace of
around 10--20 msgs/sec handling around 1--2 million messages a day.

High volume Postfix servers handle well north of 100 msgs/sec,
handling 10s of millions of messages a day.

> 80% of emails are going to one domain owned by my company *(Domain1)*.. The
> current mail flow does around 1K-1.2K per minute , this is CentOS 7 VM ( 4
> CPU/8GB Memory) running on SSD Datastore.

1200/minute is 20/sec, so this is a modest though not low message
rate.  If your system is not going any faster despite a backlog of
mail to send, most likely the problem is on the receiving side.
You can't deliver any faster than the rate at which the receiving
system is able to accept mail.

So first find out whether the receiving system is running at full
capacity.  If not, try to understand is how long each delivery
takes, and what your concurrency settings are.  You can't tune a
system blind-folded, nor anyone help you without any quantitative
information about the mail flow.  You'll need to understand the
distributions of the "delays=a/b/c/d" data in your logs.  In
particular throughput is largely determined by your concurrency
divided by the typical "c+d" value.

> - Changed default_destination_concurrency_limit to 100 and

If raising concurrency is not helping much then the destination is
likely saturated or you have network bandwidth problems.

> default_destination_recipient_limit = 1500

For both parameters, instead of changing the numbers for all
destinations, configure a dedicated transport for the congested
destination, and adjust the variables just for that:

    foo_destination_concurrency_limit = 100
    foo_destination_recipient_limit = 1500

with "foo" a clone of the "smtp" or "relay" transport definition
in "master.cf".

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

anvartay
In reply to this post by Tech Gurus
You need to optimize 2 topics
1. DNS
2. Connections

For dns optimizations I recommend to put other domain1 MX hosts IP addresses to the /etc/hosts file and disable postfix dns lookups so you will reduce volume of dns lookups dramatically. And defining static transport entry also reduces MX lookups.
Second you must use connection caching in Postfix to reduce performance overheat.

Both optimizations possible in Postfix 2.10. But not earlier versions.

Anvar Kuchkartaev 
[hidden email]
From: Tech Gurus
Sent: viernes, 26 de enero de 2018 02:35
To: Anvar Kuchkartaev
Subject: Re: Configure Postfix for High Volume

I am using 2.10.1-6 RPM.

On Thu, Jan 25, 2018 at 6:17 PM, Anvar Kuchkartaev <[hidden email]> wrote:
What version of postfix are you using?

Anvar Kuchkartaev 
[hidden email]
From: Tech Gurus
Sent: jueves, 25 de enero de 2018 21:50
Subject: Configure Postfix for High Volume

Hello,

I have single mail server that send relatively large amounts of emails at least 3 times a day ranging from 15K to 50K each time ..

80% of emails are going to one domain owned by my company (Domain1).. The current mail flow does around 1K-1.2K per minute , this is CentOS 7 VM ( 4 CPU/8GB Memory) running on SSD Datastore.

Upon digging into logs, I found lots of  postfix/anvil statistics DNS lookup misses for (Domain1) , I made some changes which produced small improvements (AVG/Minute went from 1K ~ 1.2K)..

Any guidance is appreciated

Thanks

Jim

The changes below produced small improvements

- Created static transport for internal domain to forward email to local "VIP" ,VIP RR to 8 servers behind , the 8 servers are the MX host for the company owned domain.
- Changed default_destination_concurrency_limit to 100 and default_destination_recipient_limit = 1500




Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Viktor Dukhovni


> On Jan 25, 2018, at 8:51 PM, Anvar Kuchkartaev <[hidden email]> wrote:
>
> For dns optimizations I recommend to put other domain1 MX hosts IP addresses
> to the /etc/hosts file and disable postfix dns lookups so you will reduce
> volume of dns lookups dramatically.

This is not good advice, it breaks delivery to other domains.  Much better
to run a local caching resolver.  Note also that the OP reports that raising
concurrency does not improve throughput by much.  If DNS lookups were slow
higher concurrency would lead to a significant throughput increase.

> And defining static transport entry also reduces MX lookups.

If there are no MX records, and the destination is served by a single host,
one can indeed gain modest efficiency by putting in a transport entry with
the nexthop in [].

> Second you must use connection caching in Postfix to reduce performance
> overhead.

Postfix will by default cache non-TLS connections on demand (when there's
a backlog, which presumably applies to the OP's traffic).  If the
destination is using TLS, then connection caching is not an option.  We
don't yet a connection cache for TLS>

> Both optimizations possible in Postfix 2.10. But not earlier versions.

This is not the case.  Connection caching dates back to around Postfix
2.2, and bypassing DNS lookups has been around even longer and still
not a good idea except in very special cases.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Stephen Satchell
On 01/25/2018 05:58 PM, Viktor Dukhovni wrote:
> This is not good advice, it breaks delivery to other domains.  Much better
> to run a local caching resolver.  Note also that the OP reports that raising
> concurrency does not improve throughput by much.  If DNS lookups were slow
> higher concurrency would lead to a significant throughput increase.

+1

In the dim, dark past, when I was mail administrator for a hosting
company, I configured a PostFix instance (bare metal, not VM) that
smart-hosted (I'm guessing) 40-50 instances of qmail and exim in Web
control panel systems.  The outgoing mail volume was on the order of
tens of thousands per hour.  (That server did per-domain throttling for
the major mail services, to avoid being nailed by the traffic monitors
on those services.)  At peak outgoing load, it still loafed.

On that outbound MX server, I configured a local caching DNS server.
The key to success was to configure the size of the memory cache up and
up and up.  That limited the number of recursive look-ups that had to go
off-system.

For the incoming MX (on a separate box) I did something similar, yet
another local caching DNS server, to ease the DNS resolver traffic for
PostFix, DNSBLs, and spam assassin.

The reason I don't recall the actual size of the CNS cache is that I
"tuned" the size of each DNS cache until the amount of outbound query
traffic was acceptable to me.  Neither box had minimum hold times set
up, so it didn't do all that much for domains with short (~300 seconds
or less) hold times, but those were a small percentage of the look-ups
that were cached.

N.B.:  Before doing the smarthost consolidation, my main DNS servers
were running at red-line.
Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Tech Gurus
Just checking back if there is recommendation to increase outbound mail delivery  .

On Fri, Jan 26, 2018 at 11:04 AM, Stephen Satchell <[hidden email]> wrote:
On 01/25/2018 05:58 PM, Viktor Dukhovni wrote:
This is not good advice, it breaks delivery to other domains.  Much better
to run a local caching resolver.  Note also that the OP reports that raising
concurrency does not improve throughput by much.  If DNS lookups were slow
higher concurrency would lead to a significant throughput increase.

+1

In the dim, dark past, when I was mail administrator for a hosting company, I configured a PostFix instance (bare metal, not VM) that smart-hosted (I'm guessing) 40-50 instances of qmail and exim in Web control panel systems.  The outgoing mail volume was on the order of tens of thousands per hour.  (That server did per-domain throttling for the major mail services, to avoid being nailed by the traffic monitors on those services.)  At peak outgoing load, it still loafed.

On that outbound MX server, I configured a local caching DNS server. The key to success was to configure the size of the memory cache up and up and up.  That limited the number of recursive look-ups that had to go off-system.

For the incoming MX (on a separate box) I did something similar, yet another local caching DNS server, to ease the DNS resolver traffic for PostFix, DNSBLs, and spam assassin.

The reason I don't recall the actual size of the CNS cache is that I "tuned" the size of each DNS cache until the amount of outbound query traffic was acceptable to me.  Neither box had minimum hold times set up, so it didn't do all that much for domains with short (~300 seconds or less) hold times, but those were a small percentage of the look-ups that were cached.

N.B.:  Before doing the smarthost consolidation, my main DNS servers were running at red-line.

Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Viktor Dukhovni


> On Jan 30, 2018, at 9:44 PM, Tech Gurus <[hidden email]> wrote:
>
> Just checking back if there is recommendation to increase outbound mail delivery

First understand the source of the bottleneck.  To that end, the
"delays" field in your logs are the key data source to try to
understand the origin of the problem.  So far it looks like
the destination is congested.  If so, the solution is there.
It is not possible to deliver mail faster than the remote
end is able to accept it.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Stephen Satchell
In reply to this post by Tech Gurus
On 01/30/2018 06:44 PM, Tech Gurus wrote:
> Just checking back if there is recommendation to increase outbound mail
> delivery  .

Can you characterize the distribution of your mail delivery?  In other
words, if you take each mailpiece, determine the MX, and collate the
results, do you have a lot of mail going to relatively few endpoints, or
do you have a wide spread of targets?

What is the nature of the emails themselves?  Are they small messages,
or are they carriers of huge attachments?  Some mail exchangers will
throttle large volumes of mail from a single IP address.  Large as in
count, or large as in size, or some combination.

Do you throttle your sending in any way to major mailers like Yahoo and
Google?  I know that, in order to get mail to the "free" mail servers, I
had to eliminate any concurrent sendings to avoid triggering their
anti-spam measures.  That means you will need a large hard disk
(although "large" a decade a go is bordering on entry size today) to
hold the mail spool.

Do you have people doing mass mailing?  If so, you need to segregate
those potential abusers onto a separate mail exchanger.  In my prior
practice, I set up individual MX systems for customers who do large mass
mailings; for example, a real-estate clearinghouse sent house listings
to agents all over the place.  One huge newsletter operator needed her
own mail exchanger, too.

Have you examined the subject lines of the mail going out from your
system?  Have you put in any protective messages so that any spam mail
is tossed before it enters your mail queue?  On my systems, I maintained
a header_checks file with all the common spam subjects.  (It helps that
I had a small collection of honeypot mail addresses so I could harvest
those spam subjects.)

Finally, what measures have you taken to avoid outsiders injecting mail?
  If you are a Web host, for example, have you hardened all the mail-out
Web pages against various forms of injection?  Do you accept mail only
from your users, or are you running an open mail relay?

Before you try to come up with a solution, you need to find out the
parameters of your problem.
Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Stephen Satchell
In reply to this post by Tech Gurus
On 01/30/2018 06:44 PM, Tech Gurus wrote:
> Just checking back if there is recommendation to increase outbound mail
> delivery  .

One additional thought:  have you thought about punting the problem, and
configuring PostFix to use a smarthost on a contracted mail service?
One that cares about their reputation, and takes active measures to keep
its MX peers happy?  This could be your cheapest solution.
Reply | Threaded
Open this post in threaded view
|

Re: Configure Postfix for High Volume

Viktor Dukhovni
In reply to this post by Stephen Satchell


> On Jan 31, 2018, at 9:53 AM, Stephen Satchell <[hidden email]> wrote:
>
> Can you characterize the distribution of your mail delivery?  In other words, if you take each mailpiece, determine the MX, and collate the results, do you have a lot of mail going to relatively few endpoints, or do you have a wide spread of targets?

The OP's original message said:

> I have single mail server that send relatively large amounts of emails at least 3 times a day ranging from 15K to 50K each time ..
>
> 80% of emails are going to one domain owned by my company (Domain1).. The current mail flow does around 1K-1.2K per minute , this is CentOS 7 VM ( 4 CPU/8GB Memory) running on SSD Datastore.

With slow delivery to this one domain.  To figure out why, requires a
quantitative approach to the problem, but the OP has not yet measured
the delivery latency or determined whether the delay is in connection
setup (before HELO) or in message delivery (MAIL FROM to DOT).

Postfix records these in the "delays=" field of each delivery log entry.

The OP must also attempt to understand whether the destination is
actually capable of processing email any faster.

DNS issues seem unlikely given that raising concurrency does not
dramatically improve throughput.  This does not mean that DNS
problems (especially with reverse resolution) might not be a
contributing factor.  So it would be a good idea to make sure
that the receiving system is able to reverse resolve the IP
address of the sending system, and is not configured to tarpit
or otherwise slow down input as a misplaced anti-spam feature.

--
        Viktor.