Postfix delivery mechanism

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Postfix delivery mechanism

Laszlo Kupor
Hello!

I migrate a Sendmail system to Postfix, but i have
incompatibility/performance problem. The Sendmail catch envelope "rcpt
to:" recipients and if relay is applicable sends mail to "rcpt to:"
addresses email domain part MX mail server. If all the recipients
domains MX host are same Sendmail sends mail in one SMTP session as one
mail for all recipients.
For ex.:

recipients: [hidden email]
            [hidden email]
            [hidden email]
            [hidden email]

If i send mails every recipients, Sendmail query domain1.com,
domain2.com and domain3.com mail handler, and if this same connect ot
mail handler and send email to mail handler at one SMTP session with all
four recipients (one mail in network one process, one resources).

This situation in Postfix environment different.
Postfix after accept mail query all domains (domain1,2,3) MX and sends
email in separate SMTP session per domain name also if same MX of all
domain. The example above sends 3 mail with 2,1,1 recipients in 3 SMTP
session, 3 mail on network,3 processes and resources.
I search documentation, and internet but not found solution to emulate
Sendmail behavior in Postfix.

The question i can emulate this in Postfix?

Thanks,  


--
Udv.: Willy

PGP GNUPG/1.0 ID = 44E7F3A4    Kupor Laszlo Attila <[hidden email]>
Key fingerprint  = 1294 00C9 F7ED AE32 1D2D  B80A D5C9 98D6 44E7 F3A4


Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Stan Hoeppner
Laszlo Kupor put forth on 10/3/2009 5:16 AM:

> Hello!
>
> I migrate a Sendmail system to Postfix, but i have
> incompatibility/performance problem. The Sendmail catch envelope "rcpt
> to:" recipients and if relay is applicable sends mail to "rcpt to:"
> addresses email domain part MX mail server. If all the recipients
> domains MX host are same Sendmail sends mail in one SMTP session as one
> mail for all recipients.
> For ex.:
>
> recipients: [hidden email]
>    [hidden email]
>    [hidden email]
>    [hidden email]

[snip]

I'm not sure this is exactly what you need, but start by taking a look
at the following options.  I do not know what values you need to
specify.  Others here will know, but may not respond for a few hours
yet.  Experiment with thees and see if they do the trick.

http://www.postfix.org/postconf.5.html#smtp_destination_concurrency_limit
http://www.postfix.org/postconf.5.html#smtp_destination_recipient_limit

--
Stan
Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Laszlo Kupor
2009. 10. 3, 05.32 Stan Hoeppner write:

> Laszlo Kupor put forth on 10/3/2009 5:16 AM:
> > Hello!
> >
> > I migrate a Sendmail system to Postfix, but i have
> > incompatibility/performance problem. The Sendmail catch envelope "rcpt
> > to:" recipients and if relay is applicable sends mail to "rcpt to:"
> > addresses email domain part MX mail server. If all the recipients
> > domains MX host are same Sendmail sends mail in one SMTP session as one
> > mail for all recipients.
> > For ex.:
> >
> > recipients: [hidden email]
> >    [hidden email]
> >    [hidden email]
> >    [hidden email]
>
> [snip]
>
> I'm not sure this is exactly what you need, but start by taking a look
> at the following options.  I do not know what values you need to
> specify.  Others here will know, but may not respond for a few hours
> yet.  Experiment with thees and see if they do the trick.
>
> http://www.postfix.org/postconf.5.html#smtp_destination_concurrency_limit
> http://www.postfix.org/postconf.5.html#smtp_destination_recipient_limit
>
> --
> Stan
>

Hello Stan,

Thanks for reply. But i review this parameters again and don't help for
me.
The settings in "smtp_destination_concurrency_limit" this limit set to 1
Postfix send 1 message to same destination.  
"The maximal number of parallel deliveries to the same destination via
the smtp"
The settings in  "smtp_destination_recipient_limit" recipient maximal
number in SMTP session.
"The maximal number of recipients per message for the smtp message
delivery transport"

This can be modify "smtp_destination_concurrency_limit"
If "smtp_destination_recipient_limit" is 1
"smtp_destination_concurrency_limit" same destination == recipient if
greater than 1 same destination = domain.

There is no option to "same destination" == "same mail handler".  



--
Udv.: Willy

PGP GNUPG/1.0 ID = 44E7F3A4    Kupor Laszlo Attila <[hidden email]>
Key fingerprint  = 1294 00C9 F7ED AE32 1D2D  B80A D5C9 98D6 44E7 F3A4

Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Sahil Tandon
In reply to this post by Stan Hoeppner
On Sat, 03 Oct 2009, Stan Hoeppner wrote:

> Laszlo Kupor put forth on 10/3/2009 5:16 AM:
> > Hello!
> >
> > I migrate a Sendmail system to Postfix, but i have
> > incompatibility/performance problem. The Sendmail catch envelope "rcpt
> > to:" recipients and if relay is applicable sends mail to "rcpt to:"
> > addresses email domain part MX mail server. If all the recipients
> > domains MX host are same Sendmail sends mail in one SMTP session as one
> > mail for all recipients.
> > For ex.:
> >
> > recipients: [hidden email]
> >    [hidden email]
> >    [hidden email]
> >    [hidden email]
>
> [snip]
>
> I'm not sure this is exactly what you need, but start by taking a look
> at the following options.  I do not know what values you need to
> specify.  Others here will know, but may not respond for a few hours
> yet.  Experiment with thees and see if they do the trick.
>
> http://www.postfix.org/postconf.5.html#smtp_destination_concurrency_limit
> http://www.postfix.org/postconf.5.html#smtp_destination_recipient_limit

No, these links do not address the OP's question.  Laslzo, you should
review http://www.postfix.org/CONNECTION_CACHE_README.html, especially
the section titled 'Connection cache configuration'.

--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Victor Duchovni
In reply to this post by Laszlo Kupor
On Sat, Oct 03, 2009 at 12:16:26PM +0200, Laszlo Kupor wrote:

> This situation in Postfix environment different.
> Postfix after accept mail query all domains (domain1,2,3) MX and sends
> email in separate SMTP session per domain name also if same MX of all
> domain. The example above sends 3 mail with 2,1,1 recipients in 3 SMTP
> session, 3 mail on network,3 processes and resources.
> I search documentation, and internet but not found solution to emulate
> Sendmail behavior in Postfix.
>
> The question i can emulate this in Postfix?

No. Postfix never sends recipients for different transport:nexthop
combinations (the nexthop is typically the recipient domain) in the
same envelope.

There are good reasons for this, it is difficult to do this and still
have a reasonable queue scheduling algorithm (Sendmail has no global
scheduling at all, which is far worse than occasionally splitting
envelopes).

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Laszlo Kupor
2009. 10. 3,  12.39 Victor Duchovni wrote:

> On Sat, Oct 03, 2009 at 12:16:26PM +0200, Laszlo Kupor wrote:
>
> > This situation in Postfix environment different.
> > Postfix after accept mail query all domains (domain1,2,3) MX and sends
> > email in separate SMTP session per domain name also if same MX of all
> > domain. The example above sends 3 mail with 2,1,1 recipients in 3 SMTP
> > session, 3 mail on network,3 processes and resources.
> > I search documentation, and internet but not found solution to emulate
> > Sendmail behavior in Postfix.
> >
> > The question i can emulate this in Postfix?
>
> No. Postfix never sends recipients for different transport:nexthop
> combinations (the nexthop is typically the recipient domain) in the
> same envelope.
>
> There are good reasons for this, it is difficult to do this and still
> have a reasonable queue scheduling algorithm (Sendmail has no global
> scheduling at all, which is far worse than occasionally splitting
> envelopes).
>

Thanks for your reply, Sendmail stay here.
I can't go with you in this situation. The administrators view: this is
big performance loss in some situation(maybe rare), and can occurs
duplicate mail delivery (i tested, for a day and overhead above 150%
here). I can't see clearly why good this, but i accept this is good.
Just not for me and not for now. If you have a written documentation
about queue mechanism please give me a reference.

Thanks,


--
Udv.: Willy

PGP GNUPG/1.0 ID = 44E7F3A4    Kupor Laszlo Attila <[hidden email]>
Key fingerprint  = 1294 00C9 F7ED AE32 1D2D  B80A D5C9 98D6 44E7 F3A4

Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Sahil Tandon
On Sun, 04 Oct 2009, Laszlo Kupor wrote:

> 2009. 10. 3,  12.39 Victor Duchovni wrote:
> > On Sat, Oct 03, 2009 at 12:16:26PM +0200, Laszlo Kupor wrote:
> >
> > > This situation in Postfix environment different.
> > > Postfix after accept mail query all domains (domain1,2,3) MX and sends
> > > email in separate SMTP session per domain name also if same MX of all
> > > domain. The example above sends 3 mail with 2,1,1 recipients in 3 SMTP
> > > session, 3 mail on network,3 processes and resources.
> > > I search documentation, and internet but not found solution to emulate
> > > Sendmail behavior in Postfix.
> > >
> > > The question i can emulate this in Postfix?
> >
> > No. Postfix never sends recipients for different transport:nexthop
> > combinations (the nexthop is typically the recipient domain) in the
> > same envelope.
> >
> > There are good reasons for this, it is difficult to do this and still
> > have a reasonable queue scheduling algorithm (Sendmail has no global
> > scheduling at all, which is far worse than occasionally splitting
> > envelopes).
>
> Thanks for your reply, Sendmail stay here.
> I can't go with you in this situation. The administrators view: this is
> big performance loss in some situation(maybe rare), and can occurs
> duplicate mail delivery (i tested, for a day and overhead above 150%
> here). I can't see clearly why good this, but i accept this is good.
> Just not for me and not for now. If you have a written documentation
> about queue mechanism please give me a reference.

You've already been linked to one document, but here it is again with
one more.  Please read carefully.

 http://www.postfix.org/CONNECTION_CACHE_README.html
 http://www.postfix.org/SCHEDULER_README.html

--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Laszlo Kupor
2009. 10. 4, vasárnap keltezéssel 11.27-kor Sahil Tandon ezt írta:

> On Sun, 04 Oct 2009, Laszlo Kupor wrote:
>
> > 2009. 10. 3,  12.39 Victor Duchovni wrote:
> > > On Sat, Oct 03, 2009 at 12:16:26PM +0200, Laszlo Kupor wrote:
> > >
> > > > This situation in Postfix environment different.
> > > > Postfix after accept mail query all domains (domain1,2,3) MX and sends
> > > > email in separate SMTP session per domain name also if same MX of all
> > > > domain. The example above sends 3 mail with 2,1,1 recipients in 3 SMTP
> > > > session, 3 mail on network,3 processes and resources.
> > > > I search documentation, and internet but not found solution to emulate
> > > > Sendmail behavior in Postfix.
> > > >
> > > > The question i can emulate this in Postfix?
> > >
> > > No. Postfix never sends recipients for different transport:nexthop
> > > combinations (the nexthop is typically the recipient domain) in the
> > > same envelope.
> > >
> > > There are good reasons for this, it is difficult to do this and still
> > > have a reasonable queue scheduling algorithm (Sendmail has no global
> > > scheduling at all, which is far worse than occasionally splitting
> > > envelopes).
> >
> > Thanks for your reply, Sendmail stay here.
> > I can't go with you in this situation. The administrators view: this is
> > big performance loss in some situation(maybe rare), and can occurs
> > duplicate mail delivery (i tested, for a day and overhead above 150%
> > here). I can't see clearly why good this, but i accept this is good.
> > Just not for me and not for now. If you have a written documentation
> > about queue mechanism please give me a reference.
>
> You've already been linked to one document, but here it is again with
> one more.  Please read carefully.
>
>  http://www.postfix.org/CONNECTION_CACHE_README.html
>  http://www.postfix.org/SCHEDULER_README.html
>

Hello,

Thanks again, but one part of this documents absolutely other view of my
problem, other part is a documentation.

The first one (linked before),  only give effect in very specialized
environment, and not for TLS connections. And not give solution for
duplicate delivery.

SCHEDULER_README:
Concurrency scheduler make steps against pressure of negative SMTP
connections. Absolute useless in my problem

Preemptive scheduler well, this is a good document to introduce queuing
mechanism, and this very interesting not contains restriction
domain=destination, and now much confused, why and how limit postfix
destination/transport nexthop = domain.



Quote:
"Whenever nqmgr moves a queue file into the active queue, the following
happens: It reads all necessary information from the queue file as oqmgr
does, and also reads as many recipients as possible - more on that
later, for now let's just pretend it always reads all recipients.

Then it resolves the recipients as oqmgr does, which means obtaining
(address, nexthop, transport) triple for each recipient. For each
triple, it finds the transport; if it does not exist yet, it
instantiates it (unless it's dead). Within the transport, it finds the
destination queue for given nexthop; if it does not exist yet, it
instantiates it (unless it's dead). The triple is then bound to given
destination queue. This happens in qmgr_resolve() and is basically the
same as in oqmgr.

Then for each triple which was bound to some queue (and thus transport),
the program finds the job which represents the message within that
transport's context; if it does not exist yet, it instantiates it.
Within the job, it finds the peer which represents the bound destination
queue within this jobs context; if it does not exist yet, it
instantiates it. Finally, it stores the address from the resolved triple
to the recipient entry which is appended to both the queue entry list
and the peer entry list. The addresses for same nexthop are batched in
the entries up to recipient_concurrency limit for that transport. This
happens in qmgr_assign() and apart from that it operates with job and
peer structures it is basically the same as in oqmgr.

When the job is instantiated, it is enqueued on the transport's job list
based on the time its message was picked up by nqmgr. For first batch of
recipients this means it is appended to the end of the job list, but the
ordering of the job list by the enqueue time is important as we will see
shortly.
"

Think, a (may rare) example: in a popular mail handler (or mail
handlers) with many (1000+) domains catch the mails from the internet,
and deliver to mailbox backend server(s). If i send email to all hosted
domain [hidden email] email address if SMTP client Postfix and connect
with TLS do 1000 connection to deliver this only one mail.  If this SMTP
client is a Sendmail connect and sends (1000/max concurrent recipient).

In other view. If you have corporal alias pl [hidden email] witch
deliver many postmaster in organization, if one of them send mail to
postmaster@ and other postmaster (because this is a policy) send a reply
to all the original poster receive 2 absolute same  mails with postfix
and 1 mail with Sendmail.

Again, i understand Victor email, and accept. But not understand why
limit nexthop/destination for one domain.
 

--
Udv.: Willy

PGP GNUPG/1.0 ID = 44E7F3A4    Kupor Laszlo Attila <[hidden email]>
Key fingerprint  = 1294 00C9 F7ED AE32 1D2D  B80A D5C9 98D6 44E7 F3A4

Reply | Threaded
Open this post in threaded view
|

Re: Postfix delivery mechanism

Victor Duchovni
In reply to this post by Laszlo Kupor
On Sun, Oct 04, 2009 at 10:48:12AM +0200, Laszlo Kupor wrote:

> > > I search documentation, and internet but not found solution to emulate
> > > Sendmail behavior in Postfix.
> > >
> > > The question i can emulate this in Postfix?
> >
> > No. Postfix never sends recipients for different transport:nexthop
> > combinations (the nexthop is typically the recipient domain) in the
> > same envelope.
> >
> > There are good reasons for this, it is difficult to do this and still
> > have a reasonable queue scheduling algorithm (Sendmail has no global
> > scheduling at all, which is far worse than occasionally splitting
> > envelopes).
> >
>
> Thanks for your reply, Sendmail stay here.

You are welcome to continue to use Sendmail.

> I can't go with you in this situation. The administrators view: this is
> big performance loss in some situation(maybe rare),

Postfix scheduling outperforms Sendmail scheduling in all but the most
exotic configurations. Postfix has no QueueLA/RefuseLA, which are
severely broken. Postfix continues getting work done without stop/start
congestion amplification and without saturating the machine.

> and can occurs
> duplicate mail delivery (i tested, for a day and overhead above 150%
> here).

Your tests don't mean much if you are testing isolated edge cases. If
the bulk of your load is to a set of domains served by a single pool
of MX hosts, you can add transport table entries:

        example.com smtp:example.com
        example.net smtp:example.com
        example.org smtp:example.com

and then mail for all three domains is delivered in one envelope.

> I can't see clearly why good this, but i accept this is good.
> Just not for me and not for now. If you have a written documentation
> about queue mechanism please give me a reference.

http://www.postfix.org/QSHAPE_README.html#active_queue

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.