Basic sending concurrency question

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Basic sending concurrency question

Peter Scott
Hello.  I'm very new to Postfix configuration; I switched from Sendmail
because I want to send mail through the Amazon Simple Email Service and
Postfix has concurrency options that were easier to understand.  
However, they're not doing what I want and the mail is going too slow.

Sending mail via Amazon happens via piping it to a program that makes an
HTTP connection.  This takes about 0.5 seconds.  We have a high volume
of mail that needs to be delivered at a rate of at least 10 messages per
second, and our Amazon account supports up to 90/s.  But when I send
mail with sendmail.postfix it appears synchronous, i.e., takes 0.5 s to
return.  Amazon says that concurrency is the way to increase the rate:
make multiple simultaneous connections.

I was hoping that Postfix would do this for me, i.e., that calling
sendmail.postfix would return instantly, queueing the message up and
executing as many parallel connections as it was configured for.  It
doesn't; either I haven't configured it right or I'm wrong about how
sending works.  I could of course background each call to
sendmail.postfix but then I'd have to do my own concurrency management
to throttle the number of simultaneous processes I had running that, and
that seems to me to be exactly what Postfix should be doing for me.

Part of my main.cf:

default_transport = aws-email
default_destination_concurrency_limit = 1000
aws-email_destination_concurrency_limit = 1000
default_process_limit = 1000

Part of my master.cf:

aws-email  unix  -       n       n       -       0       pipe
   flags=R user=mail argv=/path/to/ses-send-email.pl [...] -f ${sender}
${recipient}

I'm sending mail with:

/usr/lib/sendmail.postfix -t -f<fromaddress>

Is it possible to send mail with a faster return than 0.5 seconds and
have Postfix manage the sending queue for me with concurrency?  What am
I missing, please?
Reply | Threaded
Open this post in threaded view
|

Re: Basic sending concurrency question

Wietse Venema
Peter Scott:

> Hello.  I'm very new to Postfix configuration; I switched from Sendmail
> because I want to send mail through the Amazon Simple Email Service and
> Postfix has concurrency options that were easier to understand.  
> However, they're not doing what I want and the mail is going too slow.
>
> Sending mail via Amazon happens via piping it to a program that makes an
> HTTP connection.  This takes about 0.5 seconds.  We have a high volume
> of mail that needs to be delivered at a rate of at least 10 messages per
> second, and our Amazon account supports up to 90/s.  But when I send
> mail with sendmail.postfix it appears synchronous, i.e., takes 0.5 s to
> return.  Amazon says that concurrency is the way to increase the rate:
> make multiple simultaneous connections.

The Postfix sendmail command RECEIVES mail.  The program returns
BEFORE MAIL IS DELIVERED. Mail delivery is handled by other programs
that run in the background.

If it takes 0.5s for you to put mail into Postfix, then that is
your bottleneck.  

Possibilities:

- Perhaps your HTTP-based system is slow.

- Perhaps the storage system is slow. We had a thread a week ago
about a storage system that needed 0.3 to 0.5 seconds to write a
message to the queue.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Basic sending concurrency question

Bastian Blank-3
In reply to this post by Peter Scott
On Mon, Jan 30, 2012 at 02:33:53PM -0800, Peter Scott wrote:
> Sending mail via Amazon happens via piping it to a program that
> makes an HTTP connection.  This takes about 0.5 seconds.

Postfix is a SMTP-Server/-Client, no HTTP-Client.

>                                                           We have a
> high volume of mail that needs to be delivered at a rate of at least
> 10 messages per second, and our Amazon account supports up to 90/s.

Postfix themself can easily handle this rate over SMTP.

>                               Amazon says that concurrency is the
> way to increase the rate: make multiple simultaneous connections.

You are confused. Postfix uses multiple SMTP-Connections if there is
enough mail.

Bastian

--
Oh, that sound of male ego.  You travel halfway across the galaxy and
it's still the same song.
                -- Eve McHuron, "Mudd's Women", stardate 1330.1