Hello list,
I am trying to configure Postfix to process inbound mail in the following manner: 1. A sender connect to Postfix smtpd to deliver a message. 2. Postfix calls Milters A, B, C. Each of these Milters can either (a) reject the message, (b) accept the message without changes or (c) add a new header "X-SomeHeader: SomeValue" and then accept the message. 3. If any of the milters rejects the message (case 2a) Postfix rejects the delivery attempt in progress with code 5xx. 4. In case 2b (all milters accept without changes) Postfix signals 250 to the sender, terminates the inbound connection, looks up a "next hop" transport and passes the message to that destination. The tricky part is case 2c. Messages marked with one or more additional Headers, and only these messages, need to be passed to a fourth milter D after a delay (!) of somewhere between 90s and 5m (current estimate). Milter D then either accepts the message as is or adds yet another new header before accepting. As calling milter D depends on the presence and content of headers, I assume that it might be difficult to achieve the desired behaviour with built-in Postfix features of a single Postfix instance. I am currently exploring the idea of writing a transport service which FIFO-queues all inbound messages for case 2c, delaying them for X seconds before passing them to a second Postfix instance which talks to milter D. However, I'd rather use Postfix's own robust mechanisms (perhaps the deferred queue?) instead of implementing a queing transport from scratch. Also, I estimate that whatever queue I use will grow to hold approx. 1-5 million messages at peak times, so performance and robustness are key. If you have ideas that could help me solve this, I'd appreciate you letting me know. -Ralph |
On 2021-04-07 20:10, Ralph Seichter wrote:
> If you have ideas that could help me solve this, I'd appreciate you > letting me know. maybe https://github.com/milter-manager/milter-manager ? |
In reply to this post by Ralph Seichter-3
Ralph Seichter:
> Hello list, > > I am trying to configure Postfix to process inbound mail in the following > manner: > > 1. A sender connect to Postfix smtpd to deliver a message. > > 2. Postfix calls Milters A, B, C. Each of these Milters can either (a) > reject the message, (b) accept the message without changes or (c) add > a new header "X-SomeHeader: SomeValue" and then accept the message. > > 3. If any of the milters rejects the message (case 2a) Postfix rejects > the delivery attempt in progress with code 5xx. > > 4. In case 2b (all milters accept without changes) Postfix signals 250 > to the sender, terminates the inbound connection, looks up a "next hop" > transport and passes the message to that destination. > > The tricky part is case 2c. Messages marked with one or more additional > Headers, and only these messages, need to be passed to a fourth milter D > after a delay (!) of somewhere between 90s and 5m (current estimate). > Milter D then either accepts the message as is or adds yet another new > header before accepting. > > As calling milter D depends on the presence and content of headers, I > assume that it might be difficult to achieve the desired behaviour with > built-in Postfix features of a single Postfix instance. I am currently > exploring the idea of writing a transport service which FIFO-queues all > inbound messages for case 2c, delaying them for X seconds before passing > them to a second Postfix instance which talks to milter D. > > However, I'd rather use Postfix's own robust mechanisms (perhaps the > deferred queue?) instead of implementing a queing transport from scratch. > Also, I estimate that whatever queue I use will grow to hold approx. 1-5 > million messages at peak times, so performance and robustness are key. > > If you have ideas that could help me solve this, I'd appreciate you > letting me know. What problem are you trying to solve? Please describe the problem (why not deliver mail when it is received), not the solution (labeling a message, and diverting labeled messages). Wietse |
* Wietse Venema:
> What problem are you trying to solve? Milters A, B and C in my example scenario can trigger asynchronous actions in backend systems, the results of which become available only after a delay caused by processing, which takes about 3 minutes. These results are required by milter D, and it is not feasible to have Postfix hold the original sender's session in an active state for this amount of time. My idea is to hold the affected messages in a FIFO-queue-like component for a configurable amount of time, long enough to ensure that the backend processing has finished when milter D is called. Other than inducing a delay, the transport queue can be "dumb". That whay, I see no need for any form of synchronisation between Postfix and the third-party backend processes. -Ralph |
>>>>> "Ralph" == Ralph Seichter <[hidden email]> writes:
Ralph> * Wietse Venema: >> What problem are you trying to solve? Ralph> Milters A, B and C in my example scenario can trigger Ralph> asynchronous actions in backend systems, the results of which Ralph> become available only after a delay caused by processing, which Ralph> takes about 3 minutes. These results are required by milter D, Ralph> and it is not feasible to have Postfix hold the original Ralph> sender's session in an active state for this amount of time. So what happens if you get 1,000 emails coming into your system? This smells like a Denial of Service problem for you and your incoming email. That 3 minutes is a *long* time to wait for a reply on your milters. Ralph> My idea is to hold the affected messages in a FIFO-queue-like Ralph> component for a configurable amount of time, long enough to Ralph> ensure that the backend processing has finished when milter D Ralph> is called. Other than inducing a delay, the transport queue can Ralph> be "dumb". That whay, I see no need for any form of Ralph> synchronisation between Postfix and the third-party backend Ralph> processes. So basically you want to accept all emails, feed them through a chain of milters which take forever to process, and then deliver or not? Are you're milters filtering out attachements? Or looking inside attachments for spam and/or viruses? Can you go up a level and talk about the problem you're trying to solve, not HOW you're trying to solve it. What email threat are you looking to solve? John |
In reply to this post by Ralph Seichter-3
Ralph Seichter:
> * Wietse Venema: > > > What problem are you trying to solve? > > Milters A, B and C in my example scenario can trigger asynchronous > actions in backend systems, the results of which become available only > after a delay caused by processing, which takes about 3 minutes. These > results are required by milter D, and it is not feasible to have Postfix > hold the original sender's session in an active state for this amount of > time. Use milter_header_checks to trigger a FILTER action. This will divery a message to a content_filter that upon re-injection into Postfix internet -> smtpd(with milters A-C) -> cleanup -> queue If the message has no magical headeer, it is delivered as usual. queue -> smtp -> network queue -> local -> mailbox Otherwise it is sent though a content filter than can be empty queoe -> smtp(with large timeout) -> smtpd(with milter D, with large milter timeout) -> cleanup -> queue and then it is delivered as usual. This requires that Milter D blocks until the nedessary info is available. Wietse |
Wietse Venema:
> Ralph Seichter: > > * Wietse Venema: > > > > > What problem are you trying to solve? > > > > Milters A, B and C in my example scenario can trigger asynchronous > > actions in backend systems, the results of which become available only > > after a delay caused by processing, which takes about 3 minutes. These > > results are required by milter D, and it is not feasible to have Postfix > > hold the original sender's session in an active state for this amount of > > time. Edited for clarity, and one additional suggestion: > Use milter_header_checks to trigger a FILTER action. > > internet -> smtpd(with milters A-C) -> cleanup -> queue > > If the message has no magical headeer, it is delivered as usual. > > queue -> smtp -> network > queue -> local -> mailbox > > Otherwise it is sent though a content filter than can be empty > > queoe -> smtp(with large timeout) -> > smtpd(with milter D, with large milter timeout) -> cleanup -> queue > > and then it is delivered as usual. > > This requires that Milter D blocks until the nedessary info is available. Perhaps better, Milter D defers the message immrediately, and Postfix retries a few minutes later, until the necessary data is available. You may need to tweak queue_run_delay, minimal_backoff_time and maximal_backoff_time. Wietse |
In reply to this post by Ralph Seichter-3
>* Wietse Venema:
> >> What problem are you trying to solve? On 07.04.21 22:35, Ralph Seichter wrote: >Milters A, B and C in my example scenario can trigger asynchronous >actions in backend systems, the results of which become available only >after a delay caused by processing, which takes about 3 minutes. These >results are required by milter D, and it is not feasible to have Postfix >hold the original sender's session in an active state for this amount of >time. it it's not feasible to hold sender's session, use D as content_filter, not milter. the main reason for milters is to be able to hold sender's session so you can reject mail during SMTP transaction. -- Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. I intend to live forever - so far so good. |
In reply to this post by Wietse Venema
* Wietse Venema:
> Perhaps better, Milter D defers the message immrediately, and Postfix > retries a few minutes later, until the necessary data is available. Indeed. Having milter D block until the required data becomes available would tie up too many resources. Better to make D reply with 4xx and to have Postfix invoke its back-off strategies. The downside is that changes to milter D would be required, which happens to be third-party software. It currently relies on the necessary data being already available when called. Hence my idea to introduce some form of transport that adds a long enough delay to ensure that the data is actually present when D is called. Still, I have asked the authors of milter D if they would even consider making any changes; an answer is pending. -Ralph |
In reply to this post by John Stoffel-2
* John Stoffel:
> So what happens if you get 1,000 emails coming into your system? In one form or other, incoming messages flagged by milters A-C need to be queued until milter D is ready for them. Like I wrote in my original post, I am expecting a queue size of 1-5 million messages during peak hours. > So basically you want to accept all emails, feed them through a chain > of milters which take forever to process, and then deliver or not? Not quite, see my OP. Milters A, B and C can either reject a message, accept it unchanged or accept and flag it by means of adding a header, and trigger background processing aimed at milter D. At a later time, milter D can either accept a message unchanged or add yet another flag header. Rejection can only happen in milters A-C, i.e. the "fast" ones. > Can you go up a level and talk about the problem you're trying to > solve, not HOW you're trying to solve it. I don't have permission to discuss details in public. Suffice it to say that, unfortunately, milters A-D and the async processing are "condicio sine quibus non". Introducing a delay into Postfix's message processing is counterintuitive to me. I am usually more concerned with Postfix passing the hot potatoes along as quickly as possible. -Ralph |
In reply to this post by Ralph Seichter-3
Ralph Seichter:
> * Wietse Venema: > > > Perhaps better, Milter D defers the message immrediately, and Postfix > > retries a few minutes later, until the necessary data is available. > > Indeed. Having milter D block until the required data becomes available > would tie up too many resources. Better to make D reply with 4xx and to > have Postfix invoke its back-off strategies. > > The downside is that changes to milter D would be required, which > happens to be third-party software. It currently relies on the necessary What does the milter do when data is unavailable? > data being already available when called. Hence my idea to introduce > some form of transport that adds a long enough delay to ensure that the > data is actually present when D is called. You could put a sleep(500) call in the content filter. network -> smtp(milters A-Z) -> cleanup(milter_header_checks) -> queue No magical milter header: -> queue -> delivery as usual With magical milter header: queue -> smtp(long timeouts) -> proxy(with sleep 500) -> smtpd(milter D, long milter timeouts) -> queue -> delivery as usual The proxy sleeps and relays the inputs and outputs from the smtp and smtpd processes. Wietse > Still, I have asked the authors of milter D if they would even consider > making any changes; an answer is pending. > > -Ralph > |
* Wietse Venema:
> What does the milter do when data is unavailable? My understanding is that milter D is unable to decide whether or not to add its own flag header if the data is unavailable. There might be a default behaviour, but it would render the milter useless. It would require milter changes to respond with code 4xx if a decision cannot yet be made, due to lacking data. > You could put a sleep(500) call in the content filter. > [...] > queue -> smtp(long timeouts) -> proxy(with sleep 500) -> > smtpd(milter D, long milter timeouts) -> queue -> delivery as usual Would that not cause a prohibitively large number of open connections between Postfix's smtp and the sleeping proxy? -Ralph |
Ralph Seichter:
> * Wietse Venema: > > > What does the milter do when data is unavailable? > > My understanding is that milter D is unable to decide whether or not to > add its own flag header if the data is unavailable. There might be a > default behaviour, but it would render the milter useless. It would > require milter changes to respond with code 4xx if a decision cannot yet > be made, due to lacking data. > > > You could put a sleep(500) call in the content filter. > > [...] > > queue -> smtp(long timeouts) -> proxy(with sleep 500) -> > > smtpd(milter D, long milter timeouts) -> queue -> delivery as usual > > Would that not cause a prohibitively large number of open connections > between Postfix's smtp and the sleeping proxy? Limited by the (master.cf) per-delivery-transport process limit, and the (main.cf) per-delivery-transport concurrency limits. Wierse |
In reply to this post by Ralph Seichter-3
> On 07 Apr 2021, at 14:35, Ralph Seichter <[hidden email]> wrote: > > * Wietse Venema: > >> What problem are you trying to solve? > > Milters A, B and C in my example scenario can trigger asynchronous > actions in backend systems, the results of which become available only > after a delay caused by processing, which takes about 3 minutes. These > results are required by milter D, and it is not feasible to have Postfix > hold the original sender's session in an active state for this amount of > time. > > My idea is to hold the affected messages in a FIFO-queue-like component > for a configurable amount of time, long enough to ensure that the > backend processing has finished when milter D is called. Other than > inducing a delay, the transport queue can be "dumb". That whay, I see no > need for any form of synchronisation between Postfix and the third-party > backend processes. I think you would have to create your own quarantine queue to hold those messages and then have D act as a content_filter (not a milter) then re-inject them into the postfix queue. I believe ClamAV does something along these lines though the reinfection¹ requires user interaction in that case. But if your are spending MINUES looking at a mail message might I suggest that is where your problem lies? ¹ Reinjection, but it was too funny to edit out :) -- Si Hoc Legere Scis Nimium Eruditionis Habes |
In reply to this post by Wietse Venema
* Wietse Venema:
>> Would that not cause a prohibitively large number of open connections >> between Postfix's smtp and the sleeping proxy? > > Limited by the (master.cf) per-delivery-transport process limit, > and the (main.cf) per-delivery-transport concurrency limits. I have written and tested a prototype based on your suggestions, which uses goroutines to proxy and delay individual connections. The results look promising so far. I hope that I will get the go-ahead to test this in an environment with a message load closer to production conditions over the next week. One question arose: Postfix documentation [1] mentions that milter header cheks allow "content inspection of message headers that are produced by Milter applications". Does Postfix keep a record of headers which have been added during milter calls, and if so, can the list of added headers be accessed and/or logged? [1] http://www.postfix.org/postconf.5.html#milter_header_checks -Ralph |
Ralph Seichter:
> * Wietse Venema: > > >> Would that not cause a prohibitively large number of open connections > >> between Postfix's smtp and the sleeping proxy? > > > > Limited by the (master.cf) per-delivery-transport process limit, > > and the (main.cf) per-delivery-transport concurrency limits. > > I have written and tested a prototype based on your suggestions, which > uses goroutines to proxy and delay individual connections. The results > look promising so far. I hope that I will get the go-ahead to test this > in an environment with a message load closer to production conditions > over the next week. > > One question arose: Postfix documentation [1] mentions that milter > header cheks allow "content inspection of message headers that are > produced by Milter applications". Does Postfix keep a record of headers > which have been added during milter calls, and if so, can the list of > added headers be accessed and/or logged? There is no such thing. When a Milter asks Postfix to add a header to the message, then Postfix runs that header through milter_header_checks before updating the queue file (or taking some other action as specified in the milter_header_checks result). Wietse |
> On Apr 10, 2021, at 11:10 AM, Wietse Venema <[hidden email]> wrote: > > Ralph Seichter: >> * Wietse Venema: >> >>>> Would that not cause a prohibitively large number of open connections >>>> between Postfix's smtp and the sleeping proxy? >>> >>> Limited by the (master.cf) per-delivery-transport process limit, >>> and the (main.cf) per-delivery-transport concurrency limits. >> >> I have written and tested a prototype based on your suggestions, which >> uses goroutines to proxy and delay individual connections. The results >> look promising so far. I hope that I will get the go-ahead to test this >> in an environment with a message load closer to production conditions >> over the next week. >> >> One question arose: Postfix documentation [1] mentions that milter >> header cheks allow "content inspection of message headers that are >> produced by Milter applications". Does Postfix keep a record of headers >> which have been added during milter calls, and if so, can the list of >> added headers be accessed and/or logged? > > There is no such thing. > > When a Milter asks Postfix to add a header to the message, then > Postfix runs that header through milter_header_checks before updating > the queue file (or taking some other action as specified in the > milter_header_checks result). > > Wietse You could probably log added headers with a WARN action if that would be useful to you. /./ WARN — Noel Jones |
Noel Jones:
> > > On Apr 10, 2021, at 11:10 AM, Wietse Venema <[hidden email]> wrote: > > > > ?Ralph Seichter: > >> * Wietse Venema: > >> > >>>> Would that not cause a prohibitively large number of open connections > >>>> between Postfix's smtp and the sleeping proxy? > >>> > >>> Limited by the (master.cf) per-delivery-transport process limit, > >>> and the (main.cf) per-delivery-transport concurrency limits. > >> > >> I have written and tested a prototype based on your suggestions, which > >> uses goroutines to proxy and delay individual connections. The results > >> look promising so far. I hope that I will get the go-ahead to test this > >> in an environment with a message load closer to production conditions > >> over the next week. > >> > >> One question arose: Postfix documentation [1] mentions that milter > >> header cheks allow "content inspection of message headers that are > >> produced by Milter applications". Does Postfix keep a record of headers > >> which have been added during milter calls, and if so, can the list of > >> added headers be accessed and/or logged? > > > > There is no such thing. > > > > When a Milter asks Postfix to add a header to the message, then > > Postfix runs that header through milter_header_checks before updating > > the queue file (or taking some other action as specified in the > > milter_header_checks result). > > > > Wietse > > You could probably log added headers with a WARN action if that would be useful to you. > > /./ WARN Also, Postfix logs (milter_)header_checks actions. Wietse |
Free forum by Nabble | Edit this page |