Postfix mailq priority

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Postfix mailq priority

vince@vheuser.com
Is there any way to change the order of delivery of items in the mail queue?
Is there any way of changing priority of mail when adding to the queue?
Searched the docs and found nqmgr but not this specific question.

Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Viktor Dukhovni
On Mon, Jun 15, 2020 at 04:50:16PM -0400, [hidden email] wrote:

> Is there any way to change the order of delivery of items in the mail queue?

Not as such, but see below.

> Is there any way of changing priority of mail when adding to the queue?

Not as such, but:

    - Mail delivery within a given transport is largely FIFO.

    - But messages for destinations that have reached their destination
      concurrency limit (for the transport) are delayed until enough
      earlier messages are delivered.

    - Mail delivery is round-robin by transport, but deliveries via
      transports that have reached their process limit are delayed
      until free process slots become available.

Therefore, you can prioritise some messages by using a dedicated
transport that is not congested with other low-priority messages and
sends to a destination that is willing to accept messages quickly enough
at the configured concurrency.

Sometimes a dedicated transport is used to separate sluggish
low-priority traffic from normal traffic, so that normal traffic
delivery is unimpeded.

You get to put messages in different queues, some queues may move faster
or slower than others.

The express checkout line in a supermarket is great if most shoppers are
filling carts, and just a few are grabbing a quick couple of items.  But
if one day everyone wants a carton of milk and a pack of chewing gum,
the express line could turn out to be much slower.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

vince@vheuser.com
On 2020/06/15 19:01 PM, Viktor Dukhovni wrote:

> On Mon, Jun 15, 2020 at 04:50:16PM -0400, [hidden email] wrote:
>
>> Is there any way to change the order of delivery of items in the mail queue?
> Not as such, but see below.
>
>> Is there any way of changing priority of mail when adding to the queue?
> Not as such, but:
>
>      - Mail delivery within a given transport is largely FIFO.
>
>      - But messages for destinations that have reached their destination
>        concurrency limit (for the transport) are delayed until enough
>        earlier messages are delivered.
>
>      - Mail delivery is round-robin by transport, but deliveries via
>        transports that have reached their process limit are delayed
>        until free process slots become available.
>
> Therefore, you can prioritise some messages by using a dedicated
> transport that is not congested with other low-priority messages and
> sends to a destination that is willing to accept messages quickly enough
> at the configured concurrency.
>
> Sometimes a dedicated transport is used to separate sluggish
> low-priority traffic from normal traffic, so that normal traffic
> delivery is unimpeded.
>
> You get to put messages in different queues, some queues may move faster
> or slower than others.
>
> The express checkout line in a supermarket is great if most shoppers are
> filling carts, and just a few are grabbing a quick couple of items.  But
> if one day everyone wants a carton of milk and a pack of chewing gum,
> the express line could turn out to be much slower.
>
Thanks, Viktor.

The problem is periodic, completely irregular, mailing list loads.
It would be nice to lower the list priority
so that regular mail got delivered first.

I had to force some lists to lower concurrency
but now they clog the queue.
Guess I had better read up on alternative transports,
but thanks to the response.

Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Noel Jones-2

> On Jun 15, 2020, at 7:13 PM, "[hidden email]" <[hidden email]> wrote:
>
> The problem is periodic, completely irregular, mailing list loads.
> It would be nice to lower the list priority
> so that regular mail got delivered first.
>

Postfix has only one queue, and concurrency and process count is per-destination, not based on where the mail came from.

Consider using a separate postfix instance for delivering mail list messages to prevent them from interfering with regular mail. See MULTI_INSTANCE_README


  — Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Viktor Dukhovni
On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote:

> Postfix has only one queue, and concurrency and process count is
> per-destination, not based on where the mail came from.
>
> Consider using a separate postfix instance for delivering mail list
> messages to prevent them from interfering with regular mail. See
> MULTI_INSTANCE_README

There's one active queue, but within that active queue messages are
queued to a particular (transport,nexthop) pair, and scheduling is
round-robin by transport, and then FIFO, subject to concurrency and
process limits and amortised preëmption of multi-recipient messages by
messages with fewer recipients, so that messages with lots of recipients
don't hog the queue too long before some other messages with fewer
recipients that arrived later get to use a delivery slot.

If one just wants to put messages in multiple "lines" for delivery
scheduling, but otherwise all settings are the same, then using multiple
transports is simpler and often just as effective as multiple instances.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Wietse Venema
Viktor Dukhovni:

> On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote:
>
> > Postfix has only one queue, and concurrency and process count is
> > per-destination, not based on where the mail came from.
> >
> > Consider using a separate postfix instance for delivering mail list
> > messages to prevent them from interfering with regular mail. See
> > MULTI_INSTANCE_README
>
> There's one active queue, but within that active queue messages are
> queued to a particular (transport,nexthop) pair, and scheduling is
> round-robin by transport, and then FIFO, subject to concurrency and
> process limits and amortised pre?mption of multi-recipient messages by
> messages with fewer recipients, so that messages with lots of recipients
> don't hog the queue too long before some other messages with fewer
> recipients that arrived later get to use a delivery slot.
>
> If one just wants to put messages in multiple "lines" for delivery
> scheduling, but otherwise all settings are the same, then using multiple
> transports is simpler and often just as effective as multiple instances.

The more Postfix can do by itself, the better. That could be:

- Adding layer of sender-based round-robin selection. Not sure if
that would explode at large scale.

- Adding some artificial 'cost' value that is computed while delivery
requests are added to per-destination queues. Cost could depend on
the number of delivery requests per sender email address, the message
size, and so on. Then, the scheduler could choose what-to-deliver
based on artificial cost in addition to the things that it already
considers now.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Viktor Dukhovni
> On Jun 16, 2020, at 12:39 PM, Wietse Venema <[hidden email]> wrote:
>
> - Adding layer of sender-based round-robin selection. Not sure if
> that would explode at large scale.

When the input mix approximates steady-state, FIFO is essentially optimal,
with each type of message getting its average share of the output in a fair
and timely manner.  If/when we stray from FIFO, we need to be carefuly not
be simply underserving some fraction of the expected steady-state load
distribution.

This would mean being able to somehow detect a "burst" of traffic and
characterise to distinguish the messages that are contributing to the
burst from other messages.  That's a tricky problem.  The sender address
for bulk traffic is liable to have VERP tags, the client IP of interest
may be a few hops back from the edge system that encounters delays in
delivering an input burst to remote organizations (ADMDs in the language
of RFC5598).

> - Adding some artificial 'cost' value that is computed while delivery
> requests are added to per-destination queues. Cost could depend on
> the number of delivery requests per sender email address, the message
> size, and so on. Then, the scheduler could choose what-to-deliver
> based on artificial cost in addition to the things that it already
> considers now.

Ideally we'd have an algorithm that could group related messages into
a set of logical "bulk" sources, and apply the current bulk message
preëmption algorithm not only to multi-recipient mail, but also to
multi-message bulk sources.  The hard part is the classification,
especially in a single-threaded queue manager that needs to do this
in O(1 millisecond).

Perhaps the best proxy for related messages is origin domain (ignoring
localpart) + approximate message size.  Related messages are likely to
carry similar content of approximately the same size to all recipients.
But completely ignoring the localpart may be too coarse.  It is not
obvious to me how to extract common elements from a per-recipient-salted
localpart...

A traditional multi-recipient message automatically qualifies as a single
source (same message size for all recipients and same origin domain).

We could attempt to group "related" messages in a fuzzy manner as above,
and then apply the existing preëmption algorithm.  Still not sure how to
do a good job of the aggregation.

--
--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Wietse Venema
Viktor Dukhovni:

> > On Jun 16, 2020, at 12:39 PM, Wietse Venema <[hidden email]> wrote:
> >
> > - Adding layer of sender-based round-robin selection. Not sure if
> > that would explode at large scale.
>
> When the input mix approximates steady-state, FIFO is essentially optimal,
> with each type of message getting its average share of the output in a fair
> and timely manner.  If/when we stray from FIFO, we need to be carefuly not
> be simply underserving some fraction of the expected steady-state load
> distribution.

Specifically, we don't want to forever deprioritize a request, and
that is usually done over time by decreasing the thing that causes
a request to be postponed. In the next approach, artificial cost,
we would monotonically decrease the artificial cost over time
so that a delivery request is not permanently starved.

> This would mean being able to somehow detect a "burst" of traffic and
> characterise to distinguish the messages that are contributing to the
> burst from other messages.  That's a tricky problem.  The sender address
> for bulk traffic is liable to have VERP tags, the client IP of interest
> may be a few hops back from the edge system that encounters delays in
> delivering an input burst to remote organizations (ADMDs in the language
> of RFC5598).

Of course any sender-based metrics would be based on sender address
without the address extension (i.e. the part that VERP  adds).

> > - Adding some artificial 'cost' value that is computed while delivery
> > requests are added to per-destination queues. Cost could depend on
> > the number of delivery requests per sender email address, the message
> > size, and so on. Then, the scheduler could choose what-to-deliver
> > based on artificial cost in addition to the things that it already
> > considers now.
>
> Ideally we'd have an algorithm that could group related messages into
> a set of logical "bulk" sources, and apply the current bulk message
> pre?mption algorithm not only to multi-recipient mail, but also to
> multi-message bulk sources.  The hard part is the classification,
> especially in a single-threaded queue manager that needs to do this
> in O(1 millisecond).

With artificial cost done well, it should make little difference
if a message has 10000 recipients in distinct domains, or if those
same recipients are enqueued in 10000 single-recipient submissions.
In both cases we would have 10000 delivery requests, and in both
cases the artificial cost would be similar.

I think that would be an improvement.

I'd like to avoid grouping as too diffult, and consider artificial
cost (with monotonic decrease) as more feasible, provided that the
'current artificial cost' metric is easily updated (for example a
hash table indexed by sender that counts the number of delivery
requests with that sender). Not that the number of delivery requests
measures performance impact better than number of recipients.

Each new delivery request is then assigned an artificial cost that
recflects the 'current cost' metrics at that point in time. Over
time, the artificial cost for that delivery request is decreased
monotonically so that it will not be starved.

> Perhaps the best proxy for related messages is origin domain (ignoring
> localpart) + approximate message size.  Related messages are likely to
> carry similar content of approximately the same size to all recipients.
> But completely ignoring the localpart may be too coarse.  It is not
> obvious to me how to extract common elements from a per-recipient-salted
> localpart...
>
> A traditional multi-recipient message automatically qualifies as a single
> source (same message size for all recipients and same origin domain).

See my above comment abount the equivalence of multi-recipient
list mail versus single-recipient.

Look, Mom! No grouping needed.

        Wietse

> We could attempt to group "related" messages in a fuzzy manner as above,
> and then apply the existing pre?mption algorithm.  Still not sure how to
> do a good job of the aggregation.
>
> --
> --
> Viktor.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

vince@vheuser.com
In reply to this post by Viktor Dukhovni

On 2020/06/16 11:32 AM, Viktor Dukhovni wrote:

>> On Jun 16, 2020, at 12:39 PM, Wietse Venema <[hidden email]> wrote:
>>
>> - Adding layer of sender-based round-robin selection. Not sure if
>> that would explode at large scale.
> When the input mix approximates steady-state, FIFO is essentially optimal,
> with each type of message getting its average share of the output in a fair
> and timely manner.  If/when we stray from FIFO, we need to be carefuly not
> be simply underserving some fraction of the expected steady-state load
> distribution.
>
> This would mean being able to somehow detect a "burst" of traffic and
> characterise to distinguish the messages that are contributing to the
> burst from other messages.  That's a tricky problem.  The sender address
> for bulk traffic is liable to have VERP tags, the client IP of interest
> may be a few hops back from the edge system that encounters delays in
> delivering an input burst to remote organizations (ADMDs in the language
> of RFC5598).
>
>> - Adding some artificial 'cost' value that is computed while delivery
>> requests are added to per-destination queues. Cost could depend on
>> the number of delivery requests per sender email address, the message
>> size, and so on. Then, the scheduler could choose what-to-deliver
>> based on artificial cost in addition to the things that it already
>> considers now.
> Ideally we'd have an algorithm that could group related messages into
> a set of logical "bulk" sources, and apply the current bulk message
> preëmption algorithm not only to multi-recipient mail, but also to
> multi-message bulk sources.  The hard part is the classification,
> especially in a single-threaded queue manager that needs to do this
> in O(1 millisecond).
>
> Perhaps the best proxy for related messages is origin domain (ignoring
> localpart) + approximate message size.  Related messages are likely to
> carry similar content of approximately the same size to all recipients.
> But completely ignoring the localpart may be too coarse.  It is not
> obvious to me how to extract common elements from a per-recipient-salted
> localpart...
>
> A traditional multi-recipient message automatically qualifies as a single
> source (same message size for all recipients and same origin domain).
>
> We could attempt to group "related" messages in a fuzzy manner as above,
> and then apply the existing preëmption algorithm.  Still not sure how to
> do a good job of the aggregation.
>
I've got a lot to learn here.  Didn't expect so much detail!

Ideally, one could flag the big list somehow and not worry about the little ones.
I turned off concurrency because TWC.com was rejecting bundles of emails.
Serious slow down.
Postfix steadily delivers, but would like to de-prioritize the list emails
so regular traffic is not delayed.

I will look up how to create transports and assign mail to them.
But it appears that there would be no way to redirect what is already in the queue.
Is that correct?
(Sheepish grin) Feature request?  (-:

Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

vince@vheuser.com
In reply to this post by Wietse Venema

On 2020/06/16 10:39 AM, Wietse Venema wrote:

> Viktor Dukhovni:
>> On Mon, Jun 15, 2020 at 08:25:31PM -0500, Noel Jones wrote:
>>
>>> Postfix has only one queue, and concurrency and process count is
>>> per-destination, not based on where the mail came from.
>>>
>>> Consider using a separate postfix instance for delivering mail list
>>> messages to prevent them from interfering with regular mail. See
>>> MULTI_INSTANCE_README
>> There's one active queue, but within that active queue messages are
>> queued to a particular (transport,nexthop) pair, and scheduling is
>> round-robin by transport, and then FIFO, subject to concurrency and
>> process limits and amortised pre?mption of multi-recipient messages by
>> messages with fewer recipients, so that messages with lots of recipients
>> don't hog the queue too long before some other messages with fewer
>> recipients that arrived later get to use a delivery slot.
>>
>> If one just wants to put messages in multiple "lines" for delivery
>> scheduling, but otherwise all settings are the same, then using multiple
>> transports is simpler and often just as effective as multiple instances.
> The more Postfix can do by itself, the better. That could be:
>
> - Adding layer of sender-based round-robin selection. Not sure if
> that would explode at large scale.
>
> - Adding some artificial 'cost' value that is computed while delivery
> requests are added to per-destination queues. Cost could depend on
> the number of delivery requests per sender email address, the message
> size, and so on. Then, the scheduler could choose what-to-deliver
> based on artificial cost in addition to the things that it already
> considers now.
>
> Wietse
>
Obviously I am above my pay grade here,
but can this  "Adding some artificial 'cost' value" currently be done?  How?

Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Wietse Venema
In reply to this post by vince@vheuser.com
[hidden email]:
> Postfix steadily delivers, but would like to de-prioritize the list emails
> so regular traffic is not delayed.

That's what we're talking about  - making that automatic so you
don't have to babysit Postfix. Otherwise, we'd be wasting everyone's
time.

> I will look up how to create transports and assign mail to them.
> But it appears that there would be no way to redirect what is
> already in the queue.
> Is that correct?

The transport is NOT stored in the queue file. The transport map
is read every time the queue manager tries to deliver mail.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Viktor Dukhovni
On Tue, Jun 16, 2020 at 01:25:41PM -0400, Wietse Venema wrote:

> > I will look up how to create transports and assign mail to them.
> > But it appears that there would be no way to redirect what is
> > already in the queue.

If you already have a lot of messages in the "active" queue, then those
already are already resolve to a transport, but you can get the
transport to be recomputed by putting all the messages "on hold", and
then immediately releassing them (into the deferred queue) via
"postsuper -H".

> > Is that correct?
>
> The transport is NOT stored in the queue file. The transport map
> is read every time the queue manager tries to deliver mail.

More precisely, whenever the queue manager imports a message into
the active queue and queues it for delivery.  So recomputed each
time a message is retried (from "deferred" or perhaps "hold").

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Wietse Venema
In reply to this post by vince@vheuser.com
[hidden email]:
> Obviously I am above my pay grade here,
> but can this? "Adding some artificial 'cost' value" currently be done?

No. There is an existing solution that works for multi-recipient
list mail. That solution does not work for single-recipient list mail,
which we suspect is your use case.

Viktor proposes to schedule single-recipient list mail from the
same sender as if it is multi-recipient list mail (post-facto grouping).

I'm proposing to just put a cost on each delivery request, initially
based on how many other requests a sender already has in flight,
that decays over time until the request is selected for delivery.
This guarantees that all mail will eventually be delivered.

This also would give us the option to make the scheduling dependent
on message size, or quality-of-service indicators.

        Wietse
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

vince@vheuser.com
On 2020/06/16 13:42 PM, Wietse Venema wrote:

> [hidden email]:
>> Obviously I am above my pay grade here,
>> but can this? "Adding some artificial 'cost' value" currently be done?
> No. There is an existing solution that works for multi-recipient
> list mail. That solution does not work for single-recipient list mail,
> which we suspect is your use case.
>
> Viktor proposes to schedule single-recipient list mail from the
> same sender as if it is multi-recipient list mail (post-facto grouping).
>
> I'm proposing to just put a cost on each delivery request, initially
> based on how many other requests a sender already has in flight,
> that decays over time until the request is selected for delivery.
> This guarantees that all mail will eventually be delivered.
>
> This also would give us the option to make the scheduling dependent
> on message size, or quality-of-service indicators.
>
> Wietse
Just fyi -
Mailman domain concurrency is a yes/no/full option.
I've limited postfix to 5 and set Mailman to yes
and that sped things up dramatically
with no rejections from TWC.

Reply | Threaded
Open this post in threaded view
|

Re: Postfix mailq priority

Wietse Venema
[hidden email]:

> On 2020/06/16 13:42 PM, Wietse Venema wrote:
> > [hidden email]:
> >> Obviously I am above my pay grade here,
> >> but can this? "Adding some artificial 'cost' value" currently be done?
> > No. There is an existing solution that works for multi-recipient
> > list mail. That solution does not work for single-recipient list mail,
> > which we suspect is your use case.
> >
> > Viktor proposes to schedule single-recipient list mail from the
> > same sender as if it is multi-recipient list mail (post-facto grouping).
> >
> > I'm proposing to just put a cost on each delivery request, initially
> > based on how many other requests a sender already has in flight,
> > that decays over time until the request is selected for delivery.
> > This guarantees that all mail will eventually be delivered.
> >
> > This also would give us the option to make the scheduling dependent
> > on message size, or quality-of-service indicators.
> >
> > Wietse
> Just fyi -
> Mailman domain concurrency is a yes/no/full option.
> I've limited postfix to 5 and set Mailman to yes
> and that sped things up dramatically
> with no rejections from TWC.

Perhaps you set some Postfix xxx_destination_recipient_limit=1.  As
documented that makes the Postfix concurrency limit a per-recipient
property, which increases the total concurrency for a destinaion domain.

        Wietse