Quantcast

relay server - mass mailing tuning

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

relay server - mass mailing tuning

Zalezny Niezalezny
Hi,

in the next days our external service provider will push to our relay server (Postfix) regulary ~300 000 - 500 000 E-mails with size 60-500kb.

External system provider has a SMTP server farm, so it will send messages from ~20 IP`s to our single Postfix instance.

My question is, do I need to tune anything in our Postfix relay to receive and transport such a big amount of messages in single session ?

What kind of settings should I setup to keep performance on the optimal level and to avoid situation with too many connections etc.etc.


Could You please advice here ?



Thanks in advance for any support.


With kind regards

Konrad Wawryn
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: relay server - mass mailing tuning

Fazzina, Angelo

Hi,

I would think they would have to tell you the volume and or rate at which they intend to send the mail.

By default Postfix lets the queue grow to 20,000 emails I think, and then may stop accepting until it can process some, and start accepting more.

 

Any reason you haven’t been reading this?

http://www.postfix.org/TUNING_README.html

 

 

And making some educated guesses ?  You know the total volume 300-500K. all you care about is Rate, how fast are they going to send them ?

Good Luck.

-ALF

 

-Angelo Fazzina

Operating Systems Programmer / Analyst

University of Connecticut,  UITS, SSG, Server Systems

860-486-9075

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Zalezny Niezalezny
Sent: Tuesday, April 11, 2017 4:24 AM
To: Postfix users <[hidden email]>
Subject: relay server - mass mailing tuning

 

Hi,

in the next days our external service provider will push to our relay server (Postfix) regulary ~300 000 - 500 000 E-mails with size 60-500kb.

External system provider has a SMTP server farm, so it will send messages from ~20 IP`s to our single Postfix instance.

My question is, do I need to tune anything in our Postfix relay to receive and transport such a big amount of messages in single session ?

What kind of settings should I setup to keep performance on the optimal level and to avoid situation with too many connections etc.etc.

Could You please advice here ?


Thanks in advance for any support.

With kind regards

Konrad Wawryn

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: relay server - mass mailing tuning

Wietse Venema
Fazzina, Angelo:
> Hi,
> I would think they would have to tell you the volume and or rate
> at which they intend to send the mail.

He mentioned message count and size, about 100GB of traffic, so that
might take a while, especially when the queue is on spinning disks.

> By default Postfix lets the queue grow to 20,000 emails I think,
> and then may stop accepting until it can process some, and start
> accepting more.

That is incorrect. Postfix keeps accepting mail until queue_minfree
limit is reached. That happens when mail arrives faster than Postfix
can deliver it.

I suggest that the original poster does some experiments with
smtp-source and smtp-sink, to find out where the bottlenecks are.

Things to keep in mind:

- Multiple parallel inbound connections may get the mail faster
into Postfix, but that also overloads the file system and prevents
Postfix from delivering mail. There is an optimal input volume that
depends on file system performance.

- By default, smtpd_client_connection_count_limit is set to half
of the default process limit, so that is something that the senders
need to be aware of.

- If the sender's DNS setup is borked, Postfix will lose time doing
DNS lookup for the SMTP client name/address.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: relay server - mass mailing tuning

Stephen Satchell
On 04/11/2017 07:02 AM, Wietse Venema wrote:
> - If the sender's DNS setup is borked, Postfix will lose time doing
> DNS lookup for the SMTP client name/address.

One way I minimized this issue was to define a very large cache in the
local DNS service for each dedicated incoming mail server.

This was in a web hosting shop with roughly 80K domains across a number
of CPANEL and PLESK hosts, with a central set of beefy PostFix-running
clearinghouse MXes fronting for QMail and Exim.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: relay server - mass mailing tuning

Philip Paeps
In reply to this post by Wietse Venema
On 2017-04-11 10:02:21 (-0400), Wietse Venema <[hidden email]> wrote:
>Fazzina, Angelo:
>>I would think they would have to tell you the volume and or rate at
>>which they intend to send the mail.
>
>He mentioned message count and size, about 100GB of traffic, so that
>might take a while, especially when the queue is on spinning disks.

It is worth repeating that the spinning rust actually matters in this
case: Postfix fsync()s when accepting a message into the queue.  The
time to it takes to enqueue a message is at least the time it takes to
write it to disk, not simply the time it takes to hit the buffer cache.

People occasionally forget that Postfix actually tries to be reliable in
addition to fast. :)

Mounting /var/spool on an SSD is definitely a good idea.  Ideally a
mirrored pair of SSDs for reliability.

If you're running on ZFS, you want your ZIL on fast SSDs (again: plural,
and remember to mirror them, by default ZFS load-balances log devices).

(If you don't care about reliability, mount /var/spool on a tmpfs.  Or
build Postfix without HAS_FSYNC.  If it breaks you get to keep the
pieces though.)

>- If the sender's DNS setup is borked, Postfix will lose time doing
>DNS lookup for the SMTP client name/address.

Running a local resolver (e.g. Unbound) can mitigate most of these
issues.  Also be sure to check that the TTL of the SMTP client's DNS
name / zone are configured sensibly.  If it's got a TTL of 5 seconds,
you can do all the caching you like and you'll still be doing a lot of
waiting for DNS.  (This can be mitigated with the cache-min-ttl setting
in Unbound).

Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: relay server - mass mailing tuning

Viktor Dukhovni

> On Apr 11, 2017, at 1:55 PM, Philip Paeps <[hidden email]> wrote:
>
> It is worth repeating that the spinning rust actually matters in this case:
> Postfix fsync()s when accepting a message into the queue.  The time to it
> takes to enqueue a message is at least the time it takes to write it to
> disk, not simply the time it takes to hit the buffer cache.

For high-volume senders, the main problem is usually getting the mail *out*
faster with less friction from the receiving systems.  Accepting mail faster
is not necessarily a good idea, if that just means a bigger backlog.

The OPs efforts should go towards understanding what it takes to send email
at the intended volume.  Receiving it fast enough is a secondary issue.

It sounds like the OP is largely unprepared for the task at hand.  This
requires real experience and dedicated effort.  Just asking for a few ideas
on this list is unlikely to be sufficient.

A sane approach is to start small, grow gradually, and solve small problems
as they come up, before they become big problems.  Going from nothing to
sending a very large volume of mail is not easy.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: relay server - mass mailing tuning

Philip Paeps
On 2017-04-11 14:04:08 (-0400), Viktor Dukhovni <[hidden email]> wrote:

> On Apr 11, 2017, at 1:55 PM, Philip Paeps <[hidden email]> wrote:
>>It is worth repeating that the spinning rust actually matters in this
>>case: Postfix fsync()s when accepting a message into the queue.  The
>>time to it takes to enqueue a message is at least the time it takes to
>>write it to disk, not simply the time it takes to hit the buffer
>>cache.
>
>For high-volume senders, the main problem is usually getting the mail *out*
>faster with less friction from the receiving systems.  Accepting mail faster
>is not necessarily a good idea, if that just means a bigger backlog.

I'm aware of that. :)  I merely wanted to remind the OP (and anyone
stumbling across this thread in the archives) that the time it takes to
enqueue mail should not be underestimated.  Many common workloads don't
have to care too much about disk latency because they're protected by
the buffer cache.  This is not the case when handling large volumes of
mail.

Ideally you want to be able to get mail out as fast as it's coming in or
at least not very much slower but you don't have complete control over
that (sites you need to deliver to can go down or refuse to take mail
from you for a while, etc).

If you know you need to relay half a million messages in a short period
of time weighing about 100Gbytes in total, you need to be prepared to
enqueue an appreciable fraction of that volume.

>The OPs efforts should go towards understanding what it takes to send
>email at the intended volume.  Receiving it fast enough is a secondary
>issue.
>
>It sounds like the OP is largely unprepared for the task at hand.  This
>requires real experience and dedicated effort.  Just asking for a few
>ideas on this list is unlikely to be sufficient.

I agree.

>A sane approach is to start small, grow gradually, and solve small
>problems as they come up, before they become big problems.  Going from
>nothing to sending a very large volume of mail is not easy.

At the very least, the OP should spend what little time he has before he
needs to handle this load becoming as familiar as possible with the
TUNING_README and probably also QSHAPE_README.

That will hopefully prepare him enough to balance the rate of mail
coming in against the rate he can get it out.  It still might not be
very pretty though.

Philip

--
Philip Paeps
Senior Reality Engineer
Ministry of Information
Loading...