Milter calls based on conditions and deliberate delays

[Ralph Seichter-3]

Hello list,

I am trying to configure Postfix to process inbound mail in the following
manner:

1. A sender connect to Postfix smtpd to deliver a message.

2. Postfix calls Milters A, B, C. Each of these Milters can either (a)
reject the message, (b) accept the message without changes or (c) add
a new header "X-SomeHeader: SomeValue" and then accept the message.

3. If any of the milters rejects the message (case 2a) Postfix rejects
the delivery attempt in progress with code 5xx.

4. In case 2b (all milters accept without changes) Postfix signals 250
to the sender, terminates the inbound connection, looks up a "next hop"
transport and passes the message to that destination.

The tricky part is case 2c. Messages marked with one or more additional
Headers, and only these messages, need to be passed to a fourth milter D
after a delay (!) of somewhere between 90s and 5m (current estimate).
Milter D then either accepts the message as is or adds yet another new
header before accepting.

As calling milter D depends on the presence and content of headers, I
assume that it might be difficult to achieve the desired behaviour with
built-in Postfix features of a single Postfix instance. I am currently
exploring the idea of writing a transport service which FIFO-queues all
inbound messages for case 2c, delaying them for X seconds before passing
them to a second Postfix instance which talks to milter D.

However, I'd rather use Postfix's own robust mechanisms (perhaps the
deferred queue?) instead of implementing a queing transport from scratch.
Also, I estimate that whatever queue I use will grow to hold approx. 1-5
million messages at peak times, so performance and robustness are key.

If you have ideas that could help me solve this, I'd appreciate you
letting me know.

-Ralph

[Benny Pedersen-2]

On 2021-04-07 20:10, Ralph Seichter wrote:

> If you have ideas that could help me solve this, I'd appreciate you
> letting me know.

maybe https://github.com/milter-manager/milter-manager ?

[Wietse Venema]

Ralph Seichter:

> Hello list,
>
> I am trying to configure Postfix to process inbound mail in the following
> manner:
>
> 1. A sender connect to Postfix smtpd to deliver a message.
>
> 2. Postfix calls Milters A, B, C. Each of these Milters can either (a)
> reject the message, (b) accept the message without changes or (c) add
> a new header "X-SomeHeader: SomeValue" and then accept the message.
>
> 3. If any of the milters rejects the message (case 2a) Postfix rejects
> the delivery attempt in progress with code 5xx.
>
> 4. In case 2b (all milters accept without changes) Postfix signals 250
> to the sender, terminates the inbound connection, looks up a "next hop"
> transport and passes the message to that destination.
>
> The tricky part is case 2c. Messages marked with one or more additional
> Headers, and only these messages, need to be passed to a fourth milter D
> after a delay (!) of somewhere between 90s and 5m (current estimate).
> Milter D then either accepts the message as is or adds yet another new
> header before accepting.
>
> As calling milter D depends on the presence and content of headers, I
> assume that it might be difficult to achieve the desired behaviour with
> built-in Postfix features of a single Postfix instance. I am currently
> exploring the idea of writing a transport service which FIFO-queues all
> inbound messages for case 2c, delaying them for X seconds before passing
> them to a second Postfix instance which talks to milter D.
>
> However, I'd rather use Postfix's own robust mechanisms (perhaps the
> deferred queue?) instead of implementing a queing transport from scratch.
> Also, I estimate that whatever queue I use will grow to hold approx. 1-5
> million messages at peak times, so performance and robustness are key.
>
> If you have ideas that could help me solve this, I'd appreciate you
> letting me know.

What problem are you trying to solve? Please describe the problem
(why not deliver mail when it is received), not the solution (labeling
a message, and diverting labeled messages).

        Wietse

[Ralph Seichter-3]

* Wietse Venema:

> What problem are you trying to solve?

Milters A, B and C in my example scenario can trigger asynchronous
actions in backend systems, the results of which become available only
after a delay caused by processing, which takes about 3 minutes. These
results are required by milter D, and it is not feasible to have Postfix
hold the original sender's session in an active state for this amount of
time.

My idea is to hold the affected messages in a FIFO-queue-like component
for a configurable amount of time, long enough to ensure that the
backend processing has finished when milter D is called. Other than
inducing a delay, the transport queue can be "dumb". That whay, I see no
need for any form of synchronisation between Postfix and the third-party
backend processes.

-Ralph

[John Stoffel-2]

>>>>> "Ralph" == Ralph Seichter <[hidden email]> writes:

Ralph> * Wietse Venema:
>> What problem are you trying to solve?

Ralph> Milters A, B and C in my example scenario can trigger
Ralph> asynchronous actions in backend systems, the results of which
Ralph> become available only after a delay caused by processing, which
Ralph> takes about 3 minutes. These results are required by milter D,
Ralph> and it is not feasible to have Postfix hold the original
Ralph> sender's session in an active state for this amount of time.

So what happens if you get 1,000 emails coming into your system?  This
smells like a Denial of Service problem for you and your incoming
email.  That 3 minutes is a *long* time to wait for a reply on your
milters.

Ralph> My idea is to hold the affected messages in a FIFO-queue-like
Ralph> component for a configurable amount of time, long enough to
Ralph> ensure that the backend processing has finished when milter D
Ralph> is called. Other than inducing a delay, the transport queue can
Ralph> be "dumb". That whay, I see no need for any form of
Ralph> synchronisation between Postfix and the third-party backend
Ralph> processes.

So basically you want to accept all emails, feed them through a chain
of milters which take forever to process, and then deliver or not?

Are you're milters filtering out attachements?  Or looking inside
attachments for spam and/or viruses?

Can you go up a level and talk about the problem you're trying to
solve, not HOW you're trying to solve it.  What email threat are you
looking to solve?

John

[Wietse Venema]

Ralph Seichter:

> * Wietse Venema:
>
> > What problem are you trying to solve?
>
> Milters A, B and C in my example scenario can trigger asynchronous
> actions in backend systems, the results of which become available only
> after a delay caused by processing, which takes about 3 minutes. These
> results are required by milter D, and it is not feasible to have Postfix
> hold the original sender's session in an active state for this amount of
> time.

Use milter_header_checks to trigger a FILTER action. This will divery
a message to a content_filter that upon re-injection into
Postfix

    internet -> smtpd(with milters A-C) -> cleanup -> queue

If the message has no magical headeer, it is delivered as usual.

    queue -> smtp -> network
    queue -> local -> mailbox

Otherwise it is sent though a content filter than can be empty

    queoe -> smtp(with large timeout) ->
        smtpd(with milter D, with large milter timeout) -> cleanup -> queue

and then it is delivered as usual.

This requires that Milter D blocks until the nedessary info is available.

        Wietse

[Wietse Venema]

Wietse Venema:

> Ralph Seichter:
> > * Wietse Venema:
> >
> > > What problem are you trying to solve?
> >
> > Milters A, B and C in my example scenario can trigger asynchronous
> > actions in backend systems, the results of which become available only
> > after a delay caused by processing, which takes about 3 minutes. These
> > results are required by milter D, and it is not feasible to have Postfix
> > hold the original sender's session in an active state for this amount of
> > time.

Edited for clarity, and one additional suggestion:

> Use milter_header_checks to trigger a FILTER action.
>
>     internet -> smtpd(with milters A-C) -> cleanup -> queue
>
> If the message has no magical headeer, it is delivered as usual.
>
>     queue -> smtp -> network
>     queue -> local -> mailbox
>
> Otherwise it is sent though a content filter than can be empty
>
>     queoe -> smtp(with large timeout) ->
> smtpd(with milter D, with large milter timeout) -> cleanup -> queue
>
> and then it is delivered as usual.
>
> This requires that Milter D blocks until the nedessary info is available.

Perhaps better, Milter D defers the message immrediately, and Postfix retries
a few minutes later, until the necessary data is available.

You may need to tweak queue_run_delay, minimal_backoff_time and
maximal_backoff_time.

        Wietse

[Matus UHLAR - fantomas]

>* Wietse Venema:
>
>> What problem are you trying to solve?

On 07.04.21 22:35, Ralph Seichter wrote:
>Milters A, B and C in my example scenario can trigger asynchronous
>actions in backend systems, the results of which become available only
>after a delay caused by processing, which takes about 3 minutes. These
>results are required by milter D, and it is not feasible to have Postfix
>hold the original sender's session in an active state for this amount of
>time.

it it's not feasible to hold sender's session, use D as content_filter, not
milter.

the main reason for milters is to be able to hold sender's session so you
can reject mail during SMTP transaction.


--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I intend to live forever - so far so good.

[Ralph Seichter-3]

* Wietse Venema:

> Perhaps better, Milter D defers the message immrediately, and Postfix
> retries a few minutes later, until the necessary data is available.

Indeed. Having milter D block until the required data becomes available
would tie up too many resources. Better to make D reply with 4xx and to
have Postfix invoke its back-off strategies.

The downside is that changes to milter D would be required, which
happens to be third-party software. It currently relies on the necessary
data being already available when called. Hence my idea to introduce
some form of transport that adds a long enough delay to ensure that the
data is actually present when D is called.

Still, I have asked the authors of milter D if they would even consider
making any changes; an answer is pending.

-Ralph

[Ralph Seichter-3]

* John Stoffel:

> So what happens if you get 1,000 emails coming into your system?

In one form or other, incoming messages flagged by milters A-C need to
be queued until milter D is ready for them. Like I wrote in my original
post, I am expecting a queue size of 1-5 million messages during peak
hours.

> So basically you want to accept all emails, feed them through a chain
> of milters which take forever to process, and then deliver or not?

Not quite, see my OP. Milters A, B and C can either reject a message,
accept it unchanged or accept and flag it by means of adding a header,
and trigger background processing aimed at milter D. At a later time,
milter D can either accept a message unchanged or add yet another flag
header. Rejection can only happen in milters A-C, i.e. the "fast" ones.

> Can you go up a level and talk about the problem you're trying to
> solve, not HOW you're trying to solve it.

I don't have permission to discuss details in public. Suffice it to say
that, unfortunately, milters A-D and the async processing are "condicio
sine quibus non". Introducing a delay into Postfix's message processing
is counterintuitive to me. I am usually more concerned with Postfix
passing the hot potatoes along as quickly as possible.

-Ralph

[Wietse Venema]

Ralph Seichter:

> * Wietse Venema:
>
> > Perhaps better, Milter D defers the message immrediately, and Postfix
> > retries a few minutes later, until the necessary data is available.
>
> Indeed. Having milter D block until the required data becomes available
> would tie up too many resources. Better to make D reply with 4xx and to
> have Postfix invoke its back-off strategies.
>
> The downside is that changes to milter D would be required, which
> happens to be third-party software. It currently relies on the necessary

What does the milter do when data is unavailable?

> data being already available when called. Hence my idea to introduce
> some form of transport that adds a long enough delay to ensure that the
> data is actually present when D is called.

You could put a sleep(500) call in the content filter.

    network -> smtp(milters A-Z) -> cleanup(milter_header_checks) -> queue

No magical milter header:

    -> queue -> delivery as usual

With magical milter header:

    queue -> smtp(long timeouts) -> proxy(with sleep 500) ->
        smtpd(milter D, long milter timeouts) -> queue -> delivery as usual

The proxy sleeps and relays the inputs and outputs from the
smtp and smtpd processes.

        Wietse

> Still, I have asked the authors of milter D if they would even consider
> making any changes; an answer is pending.
>
> -Ralph
>

[Ralph Seichter-3]

* Wietse Venema:

> What does the milter do when data is unavailable?

My understanding is that milter D is unable to decide whether or not to
add its own flag header if the data is unavailable. There might be a
default behaviour, but it would render the milter useless. It would
require milter changes to respond with code 4xx if a decision cannot yet
be made, due to lacking data.

> You could put a sleep(500) call in the content filter.
> [...]
> queue -> smtp(long timeouts) -> proxy(with sleep 500) ->
> smtpd(milter D, long milter timeouts) -> queue -> delivery as usual

Would that not cause a prohibitively large number of open connections
between Postfix's smtp and the sleeping proxy?

-Ralph

[Wietse Venema]

Ralph Seichter:

> * Wietse Venema:
>
> > What does the milter do when data is unavailable?
>
> My understanding is that milter D is unable to decide whether or not to
> add its own flag header if the data is unavailable. There might be a
> default behaviour, but it would render the milter useless. It would
> require milter changes to respond with code 4xx if a decision cannot yet
> be made, due to lacking data.
>
> > You could put a sleep(500) call in the content filter.
> > [...]
> > queue -> smtp(long timeouts) -> proxy(with sleep 500) ->
> > smtpd(milter D, long milter timeouts) -> queue -> delivery as usual
>
> Would that not cause a prohibitively large number of open connections
> between Postfix's smtp and the sleeping proxy?

Limited by the (master.cf) per-delivery-transport process limit,
and the (main.cf) per-delivery-transport concurrency limits.

        Wierse

> On 07 Apr 2021, at 14:35, Ralph Seichter <[hidden email]> wrote:
>
> * Wietse Venema:
>
>> What problem are you trying to solve?
>
> Milters A, B and C in my example scenario can trigger asynchronous
> actions in backend systems, the results of which become available only
> after a delay caused by processing, which takes about 3 minutes. These
> results are required by milter D, and it is not feasible to have Postfix
> hold the original sender's session in an active state for this amount of
> time.
>
> My idea is to hold the affected messages in a FIFO-queue-like component
> for a configurable amount of time, long enough to ensure that the
> backend processing has finished when milter D is called. Other than
> inducing a delay, the transport queue can be "dumb". That whay, I see no
> need for any form of synchronisation between Postfix and the third-party
> backend processes.

I think you would have to create your own quarantine queue to hold those messages and then have D act as a content_filter (not a milter) then re-inject them into the postfix queue. I believe ClamAV does something along these lines though the reinfection¹ requires user interaction in that case.

But if your are spending MINUES looking at a mail message might I suggest that is where your problem lies?

¹ Reinjection, but it was too funny to edit out :)

--
Si Hoc Legere Scis Nimium Eruditionis Habes

[Ralph Seichter-3]

* Wietse Venema:

>> Would that not cause a prohibitively large number of open connections
>> between Postfix's smtp and the sleeping proxy?
>
> Limited by the (master.cf) per-delivery-transport process limit,
> and the (main.cf) per-delivery-transport concurrency limits.

I have written and tested a prototype based on your suggestions, which
uses goroutines to proxy and delay individual connections. The results
look promising so far. I hope that I will get the go-ahead to test this
in an environment with a message load closer to production conditions
over the next week.

One question arose: Postfix documentation [1] mentions that milter
header cheks allow "content inspection of message headers that are
produced by Milter applications". Does Postfix keep a record of headers
which have been added during milter calls, and if so, can the list of
added headers be accessed and/or logged?

[1] http://www.postfix.org/postconf.5.html#milter_header_checks

-Ralph

[Wietse Venema]

Ralph Seichter:

> * Wietse Venema:
>
> >> Would that not cause a prohibitively large number of open connections
> >> between Postfix's smtp and the sleeping proxy?
> >
> > Limited by the (master.cf) per-delivery-transport process limit,
> > and the (main.cf) per-delivery-transport concurrency limits.
>
> I have written and tested a prototype based on your suggestions, which
> uses goroutines to proxy and delay individual connections. The results
> look promising so far. I hope that I will get the go-ahead to test this
> in an environment with a message load closer to production conditions
> over the next week.
>
> One question arose: Postfix documentation [1] mentions that milter
> header cheks allow "content inspection of message headers that are
> produced by Milter applications". Does Postfix keep a record of headers
> which have been added during milter calls, and if so, can the list of
> added headers be accessed and/or logged?

There is no such thing.

When a Milter asks Postfix to add a header to the message, then
Postfix runs that header through milter_header_checks before updating
the queue file (or taking some other action as specified in the
milter_header_checks result).

        Wietse

[Noel Jones-2]

> On Apr 10, 2021, at 11:10 AM, Wietse Venema <[hidden email]> wrote:
>
> Ralph Seichter:
>> * Wietse Venema:
>>
>>>> Would that not cause a prohibitively large number of open connections
>>>> between Postfix's smtp and the sleeping proxy?
>>>
>>> Limited by the (master.cf) per-delivery-transport process limit,
>>> and the (main.cf) per-delivery-transport concurrency limits.
>>
>> I have written and tested a prototype based on your suggestions, which
>> uses goroutines to proxy and delay individual connections. The results
>> look promising so far. I hope that I will get the go-ahead to test this
>> in an environment with a message load closer to production conditions
>> over the next week.
>>
>> One question arose: Postfix documentation [1] mentions that milter
>> header cheks allow "content inspection of message headers that are
>> produced by Milter applications". Does Postfix keep a record of headers
>> which have been added during milter calls, and if so, can the list of
>> added headers be accessed and/or logged?
>
> There is no such thing.
>
> When a Milter asks Postfix to add a header to the message, then
> Postfix runs that header through milter_header_checks before updating
> the queue file (or taking some other action as specified in the
> milter_header_checks result).
>
>    Wietse

You could probably log added headers with a WARN action if that would be useful to you.

/./  WARN


  — Noel Jones

[Wietse Venema]

Noel Jones:

>
> > On Apr 10, 2021, at 11:10 AM, Wietse Venema <[hidden email]> wrote:
> >
> > ?Ralph Seichter:
> >> * Wietse Venema:
> >>
> >>>> Would that not cause a prohibitively large number of open connections
> >>>> between Postfix's smtp and the sleeping proxy?
> >>>
> >>> Limited by the (master.cf) per-delivery-transport process limit,
> >>> and the (main.cf) per-delivery-transport concurrency limits.
> >>
> >> I have written and tested a prototype based on your suggestions, which
> >> uses goroutines to proxy and delay individual connections. The results
> >> look promising so far. I hope that I will get the go-ahead to test this
> >> in an environment with a message load closer to production conditions
> >> over the next week.
> >>
> >> One question arose: Postfix documentation [1] mentions that milter
> >> header cheks allow "content inspection of message headers that are
> >> produced by Milter applications". Does Postfix keep a record of headers
> >> which have been added during milter calls, and if so, can the list of
> >> added headers be accessed and/or logged?
> >
> > There is no such thing.
> >
> > When a Milter asks Postfix to add a header to the message, then
> > Postfix runs that header through milter_header_checks before updating
> > the queue file (or taking some other action as specified in the
> > milter_header_checks result).
> >
> >    Wietse
>
> You could probably log added headers with a WARN action if that would be useful to you.
>
> /./  WARN

Also, Postfix logs (milter_)header_checks actions.

        Wietse