Conditional relayhost based on message size

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Conditional relayhost based on message size

AlexT
Hello,

My goal is to conditionally select the relayhost based on the total size
of the outgoing message. The rationale is that I'm using Amazon AWS SES
for the most part. Alas, SES only accepts messages up to 10 MB in size
(this includes images and attachments that are part of the message), and
when the outgoing message exceeds that size, it will be bounced.

Any tips for filters or tools that would help me define another
relayhost based on the outgoing message size would be greatly
appreciated!

Alex
Reply | Threaded
Open this post in threaded view
|

Re: Conditional relayhost based on message size

Viktor Dukhovni
On Sat, Jan 16, 2021 at 08:14:34AM +0000, Alexander wrote:

> My goal is to conditionally select the relayhost based on the total size
> of the outgoing message. The rationale is that I'm using Amazon AWS SES
> for the most part. Alas, SES only accepts messages up to 10 MB in size
> (this includes images and attachments that are part of the message), and
> when the outgoing message exceeds that size, it will be bounced.
>
> Any tips for filters or tools that would help me define another
> relayhost based on the outgoing message size would be greatly
> appreciated!

Since the queue manager has no mechanism for size-dependent selection of
a default transport (with the help of trivial-rewrite), the only way
handle this deterministically in Postfix is to shunt large messages into
a separate instance, which can be done in a number of ways, but is not
particularly simple or elegant.

The simplest is perhaps to allow the message to route to AWS, where it
would typically rejected in response to "MAIL FROM" when Postfix
announces the message size, and to map that response to a tempfail, at
which point the message would go the SMTP fallback relay for the
transport, but this can be complex if amazon publishes many IP addresses
for their submission service, since Postfix would try those first and
may give up after a small number of connections.  You'd therefore need
to also ensure that the AWS relay resolves to just 1 IP address.

That could be done with Python hooks in unbound, or a static IP
(bypassing live DNS) that's updated out of band.

If you want to take the second instance route, then it may be simplest
to place large messages on hold, and within the same filesystem, move
messages from the hold queue to the incoming queue of the second
instance.  Alternatively, an SMTP content_filter could dynamically
choose the reinjection port based on message size, but you'd need
to implement such an SMTP proxy.

Finally, you could always try Exim.  For sufficiently exotic
requirements, though I'm generally loathe to recomment a complex
monolithic MTA with a spotty security track record, it does offer a lot
more conditional logic in its mail processing pipeline.

You might even use Exim as a content_filter for Postfix, and have
Postfix ultimately deliver the outbound message, with Exim routing the
message to one of two Postfix instances.  Though it is rather overkill
if all you want is a size-dependent SMTP switch.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Conditional relayhost based on message size

Viktor Dukhovni
On Sat, Jan 16, 2021 at 04:48:22AM -0500, Viktor Dukhovni wrote:

> On Sat, Jan 16, 2021 at 08:14:34AM +0000, Alexander wrote:
>
> > My goal is to conditionally select the relayhost based on the total size
> > of the outgoing message. The rationale is that I'm using Amazon AWS SES
> > for the most part. Alas, SES only accepts messages up to 10 MB in size
> > (this includes images and attachments that are part of the message), and
> > when the outgoing message exceeds that size, it will be bounced.
> >
> > Any tips for filters or tools that would help me define another
> > relayhost based on the outgoing message size would be greatly
> > appreciated!
>
> Since the queue manager has no mechanism for size-dependent selection of
> a default transport (with the help of trivial-rewrite), the only way
> handle this deterministically in Postfix is to shunt large messages into
> a separate instance, which can be done in a number of ways, but is not
> particularly simple or elegant.

For this to be naturally supported in Postfix, we'd need a new feature,
namely a variant of "FILTER" that overrides *just* the default_transport,
making it possible to use more general criteria than "sender-dependent"
for default_transport selection.

Possible design:

    1.  New access(5) verb taking an associated transport value that
        triggers such an override.
    2.  New queue-file record type holding the default-transport
        override.
    3.  Queue manager loads this record into memory in the same way
        as "FILTER".  If both are present, "FILTER" wins.
    4.  Queue manager sends its value along with all trivial-rewrite
        "resolve" requests.
    5.  trivial-rewrite uses this default transport rather than
        $default_transport when the destination domain is not
        in any of the more specific address classes, and there
        is no explicit transport(5) mapping for the recipient.

This would be primarily useful only when using a relayhost, subject to
secondary criteria to divert some messages to a different transport
and/or nexthop based on ad-hoc message features detected during
message reception.  The difference from "FILTER" is that messages
to local recipients are not affected.

I am not sure whether adding such a feature motivated by just the
presented use-case is warranted.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Conditional relayhost based on message size

Wietse Venema
Viktor Dukhovni:

> On Sat, Jan 16, 2021 at 04:48:22AM -0500, Viktor Dukhovni wrote:
>
> > On Sat, Jan 16, 2021 at 08:14:34AM +0000, Alexander wrote:
> >
> > > My goal is to conditionally select the relayhost based on the total size
> > > of the outgoing message. The rationale is that I'm using Amazon AWS SES
> > > for the most part. Alas, SES only accepts messages up to 10 MB in size
> > > (this includes images and attachments that are part of the message), and
> > > when the outgoing message exceeds that size, it will be bounced.
> > >
> > > Any tips for filters or tools that would help me define another
> > > relayhost based on the outgoing message size would be greatly
> > > appreciated!
> >
> > Since the queue manager has no mechanism for size-dependent selection of
> > a default transport (with the help of trivial-rewrite), the only way
> > handle this deterministically in Postfix is to shunt large messages into
> > a separate instance, which can be done in a number of ways, but is not
> > particularly simple or elegant.
>
> For this to be naturally supported in Postfix, we'd need a new feature,
> namely a variant of "FILTER" that overrides *just* the default_transport,
> making it possible to use more general criteria than "sender-dependent"
> for default_transport selection.
>
> Possible design:
>
>     1.  New access(5) verb taking an associated transport value that
>         triggers such an override.
>     2.  New queue-file record type holding the default-transport
>         override.
>     3.  Queue manager loads this record into memory in the same way
>         as "FILTER".  If both are present, "FILTER" wins.
>     4.  Queue manager sends its value along with all trivial-rewrite
>         "resolve" requests.
>     5.  trivial-rewrite uses this default transport rather than
>         $default_transport when the destination domain is not
>         in any of the more specific address classes, and there
>         is no explicit transport(5) mapping for the recipient.
>
> This would be primarily useful only when using a relayhost, subject to
> secondary criteria to divert some messages to a different transport
> and/or nexthop based on ad-hoc message features detected during
> message reception.  The difference from "FILTER" is that messages
> to local recipients are not affected.
>
> I am not sure whether adding such a feature motivated by just the
> presented use-case is warranted.

My take: not if it only solves only one problem.

For example, I have several notes on making the scheduling (within
a delivery transport) dependent on a concept of 'cost', which could
be one "number of delivery requests with similar sender or domain",
"message size", "number of delivery requests with similar receiver
or domain", or some weighted combination. Just compute a cost and
schedule as appropriate for that cost bucket.

If we make routing dependent on message properties, we could take
a page from trivial-rewrite which has two flavors of routing, one
for probes and one for non-probes.  We could have N flavors of
routing and select the flavor based on the weighted computation of
something.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Conditional relayhost based on message size

@lbutlr
In reply to this post by Viktor Dukhovni
On 16 Jan 2021, at 02:48, Viktor Dukhovni <[hidden email]> wrote:
> Since the queue manager has no mechanism for size-dependent selection of
> a default transport

Postfix does, however, know the size of the message based if it is over or under the message_size_limit, and it must know that before the message hits the question manager, right?

Mightn't it be possible to hook into that size check pre-queue without. Great deal of faff?

--
C code. C code run. Run, code, run.
Reply | Threaded
Open this post in threaded view
|

Re: Conditional relayhost based on message size

Demi M. Obenour
In reply to this post by Wietse Venema
On 1/16/21 5:12 PM, Wietse Venema wrote:

> Viktor Dukhovni:
>> On Sat, Jan 16, 2021 at 04:48:22AM -0500, Viktor Dukhovni wrote:
>>
>>> On Sat, Jan 16, 2021 at 08:14:34AM +0000, Alexander wrote:
>>>
>>>> My goal is to conditionally select the relayhost based on the total size
>>>> of the outgoing message. The rationale is that I'm using Amazon AWS SES
>>>> for the most part. Alas, SES only accepts messages up to 10 MB in size
>>>> (this includes images and attachments that are part of the message), and
>>>> when the outgoing message exceeds that size, it will be bounced.
>>>>
>>>> Any tips for filters or tools that would help me define another
>>>> relayhost based on the outgoing message size would be greatly
>>>> appreciated!
>>>
>>> Since the queue manager has no mechanism for size-dependent selection of
>>> a default transport (with the help of trivial-rewrite), the only way
>>> handle this deterministically in Postfix is to shunt large messages into
>>> a separate instance, which can be done in a number of ways, but is not
>>> particularly simple or elegant.
>>
>> For this to be naturally supported in Postfix, we'd need a new feature,
>> namely a variant of "FILTER" that overrides *just* the default_transport,
>> making it possible to use more general criteria than "sender-dependent"
>> for default_transport selection.
>>
>> Possible design:
>>
>>     1.  New access(5) verb taking an associated transport value that
>>         triggers such an override.
>>     2.  New queue-file record type holding the default-transport
>>         override.
>>     3.  Queue manager loads this record into memory in the same way
>>         as "FILTER".  If both are present, "FILTER" wins.
>>     4.  Queue manager sends its value along with all trivial-rewrite
>>         "resolve" requests.
>>     5.  trivial-rewrite uses this default transport rather than
>>         $default_transport when the destination domain is not
>>         in any of the more specific address classes, and there
>>         is no explicit transport(5) mapping for the recipient.
>>
>> This would be primarily useful only when using a relayhost, subject to
>> secondary criteria to divert some messages to a different transport
>> and/or nexthop based on ad-hoc message features detected during
>> message reception.  The difference from "FILTER" is that messages
>> to local recipients are not affected.
>>
>> I am not sure whether adding such a feature motivated by just the
>> presented use-case is warranted.
>
> My take: not if it only solves only one problem.
I agree.  In particular, we should avoid a pile of features with very
narrow use-cases.

> For example, I have several notes on making the scheduling (within
> a delivery transport) dependent on a concept of 'cost', which could
> be one "number of delivery requests with similar sender or domain",
> "message size", "number of delivery requests with similar receiver
> or domain", or some weighted combination. Just compute a cost and
> schedule as appropriate for that cost bucket.
>
> If we make routing dependent on message properties, we could take
> a page from trivial-rewrite which has two flavors of routing, one
> for probes and one for non-probes.  We could have N flavors of
> routing and select the flavor based on the weighted computation of
> something.
>
> Wietse
Personally, I would like to be able to control this via a plugin of
some sort.  Declarative configuration is awesome, but it eventually
reaches its limits.

Would the existing milter system be a good fit for this?  If not, I
wonder if an embedded Lua interpreter would be a good idea.  Lua is
*tiny* (much smaller than Postfix itself), extremely portable, and
reasonably fast for an interpreted language.  There is also LuaJIT,
which is extremely fast, while still being smaller than Postfix.

Embedding an interpreter would allow users to express their business
rules in a full programming language, rather than being limited to
what Postfix supports natively.  

Sincerely,

Demi

OpenPGP_0xB288B55FFF9C22C1.asc (4K) Download Attachment
OpenPGP_signature (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Conditional relayhost based on message size

Wietse Venema
Demi M. Obenour:
> >> I am not sure whether adding such a feature motivated by just the
> >> presented use-case is warranted.
> >
> > My take: not if it only solves only one problem.
>
> I agree.  In particular, we should avoid a pile of features with very
> narrow use-cases.

I think that this narrow problem can/should be solved with a
combination of smtp_delivery_status_filter that changes 5XX message
size errors into 4XX, plus an smtp_fallback_relay with more generous
size limits.

Or just use that fallback relay all the time.

        Wuetse