Resolve before transport

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Resolve before transport

Peter
Hi guys,

I send my emails via different gateways based on my transport file. Many
domains, however, use the same email providers, such as outlook or
gmail. Is there a way to check the MX records before the email is sent
and transport it using a specific gateway?

Cheers,
Peter
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Noel Jones-2
On 9/29/2017 3:12 AM, Peter wrote:

> Hi guys,
>
> I send my emails via different gateways based on my transport file. Many
> domains, however, use the same email providers, such as outlook or
> gmail. Is there a way to check the MX records before the email is sent
> and transport it using a specific gateway?
>
> Cheers,
> Peter
>

Sorry, no easy way to do this.



  -- Noel Jones
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Marco Pizzoli
In reply to this post by Peter


On Fri, Sep 29, 2017 at 10:12 AM, Peter <[hidden email]> wrote:
Hi guys,

I send my emails via different gateways based on my transport file. Many
domains, however, use the same email providers, such as outlook or
gmail. Is there a way to check the MX records before the email is sent
and transport it using a specific gateway?


 

Cheers,
Peter

Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Viktor Dukhovni
In reply to this post by Noel Jones-2

> On Sep 29, 2017, at 12:45 PM, Noel Jones <[hidden email]> wrote:
>
>> I send my emails via different gateways based on my transport file. Many
>> domains, however, use the same email providers, such as outlook or
>> gmail. Is there a way to check the MX records before the email is sent
>> and transport it using a specific gateway?
>
> Sorry, no easy way to do this.

Very much correct, though creative multi-instance + custom DNS hacks are
possible if the number of target domains whose MX hosts one wants to
special-case is limited to O(10) or less.  With enough determination
and care, one can cause the MX hosts in question to resolve in DNS to
distinct loopback addresses where another Postfix instance receives
just the traffic for the provider in question.  This requires the
"special mail" to be diverted to a second Postfix instance on the
same machine, and that instance then delivers the mail on to the
real MX hosts.  The two instances need different DNS views, so for
that chroot might actually be needed for smtp(8), with a different
/etc/resolv.conf inside at least one of the jails.

Sadly, the legacy res_init() does not provide a means to explicitly
set the namerver.  This is possible with res_ninit() which is now
available on most current systems, but Postfix does not yet have
support for this non-universal DNS library feature.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Wietse Venema
In reply to this post by Peter
Peter:
> Hi guys,
>
> I send my emails via different gateways based on my transport file. Many
> domains, however, use the same email providers, such as outlook or
> gmail. Is there a way to check the MX records before the email is sent
> and transport it using a specific gateway?

Not yet. The scheduler would have to ask a delivery agent to 'resolve'
the next-hop address and then it would have to wait until the
delivery agent produces its answer. If multiple MX hostnames resolve
to the same IP address, we'd have the question of how to verify
server's TLS certificate - all the MX hostnames that resolved to
that IP address?

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Viktor Dukhovni
On Fri, Sep 29, 2017 at 02:35:14PM -0400, Wietse Venema wrote:

> > I send my emails via different gateways based on my transport file. Many
> > domains, however, use the same email providers, such as outlook or
> > gmail. Is there a way to check the MX records before the email is sent
> > and transport it using a specific gateway?
>
> Not yet. The scheduler would have to ask a delivery agent to 'resolve'
> the next-hop address and then it would have to wait until the
> delivery agent produces its answer. If multiple MX hostnames resolve
> to the same IP address, we'd have the question of how to verify
> server's TLS certificate - all the MX hostnames that resolved to
> that IP address?

Perhaps the request is to somehow limit destination concurrency by
destination MX rather than destination domain, more than avoiding
envelope splitting.  This is still difficult because we don't know
which MX host will actually be used once the mail is handed off.

Also, by the time concurrency limits permit access to a pre-resolved
destination, lots of time may have passed, and the DNS TTLs long
expired!

Many domains have a dead primary MX, and working secondaries, how
would we schedule to avoid overloading the secondary?  We'd need
to change smtp(8) to implement two separate service requests, a
resolution request and a delivery request.

The modified smtp(8) would do connection re-use ahead of anything
else, but absent a cached connection, it would just resolve the
destination and return the envelope to the queue manager to re-queue
for the first "[addr]:port" on the list and the (initially empty)
list of addresses already tried.

When the concurrency limit for "[addr]:port" permits, the queue
manager would call smtp(8) again (with the original delivery request
and the list of already used addresses, and smtp(8) would attemt
delivery to the first address, and if still more recipients remain,
refresh the MX host address list, and send back a now 1-longer list
of used addresses, and a new next address to try (that is not among
those already tried).

This would be very complex, I don't expect this to be implemented
in the near term.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Wietse Venema
Viktor Dukhovni:

> On Fri, Sep 29, 2017 at 02:35:14PM -0400, Wietse Venema wrote:
>
> > > I send my emails via different gateways based on my transport file. Many
> > > domains, however, use the same email providers, such as outlook or
> > > gmail. Is there a way to check the MX records before the email is sent
> > > and transport it using a specific gateway?
> >
> > Not yet. The scheduler would have to ask a delivery agent to 'resolve'
> > the next-hop address and then it would have to wait until the
> > delivery agent produces its answer. If multiple MX hostnames resolve
> > to the same IP address, we'd have the question of how to verify
> > server's TLS certificate - all the MX hostnames that resolved to
> > that IP address?
>
> Perhaps the request is to somehow limit destination concurrency by
> destination MX rather than destination domain, more than avoiding
> envelope splitting.  This is still difficult because we don't know
> which MX host will actually be used once the mail is handed off.
>
> Also, by the time concurrency limits permit access to a pre-resolved
> destination, lots of time may have passed, and the DNS TTLs long
> expired!
>
> Many domains have a dead primary MX, and working secondaries, how
> would we schedule to avoid overloading the secondary?  We'd need
> to change smtp(8) to implement two separate service requests, a
> resolution request and a delivery request.
>
> The modified smtp(8) would do connection re-use ahead of anything
> else, but absent a cached connection, it would just resolve the
> destination and return the envelope to the queue manager to re-queue
> for the first "[addr]:port" on the list and the (initially empty)
> list of addresses already tried.
>
> When the concurrency limit for "[addr]:port" permits, the queue
> manager would call smtp(8) again (with the original delivery request
> and the list of already used addresses, and smtp(8) would attemt
> delivery to the first address, and if still more recipients remain,
> refresh the MX host address list, and send back a now 1-longer list
> of used addresses, and a new next address to try (that is not among
> those already tried).
>
> This would be very complex, I don't expect this to be implemented
> in the near term.

It could be simple. Suppose there is a 'popular site' counter table
(the opposite of the 'dead site' list) which is indexed by some
idea of 'destination' and which is updated in real time as delivery
agents announce what they are doing. To determine if a next-hop has
reached its concurrency limit, compare the configured limit for the
transport with the appropriate 'popular site' count.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

@lbutlr
On 30 Sep 2017, at 07:18, Wietse Venema <[hidden email]> wrote:
> It could be simple. Suppose there is a 'popular site' counter table
> (the opposite of the 'dead site' list) which is indexed by some
> idea of 'destination' and which is updated in real time as delivery
> agents announce what they are doing. To determine if a next-hop has
> reached its concurrency limit, compare the configured limit for the
> transport with the appropriate 'popular site' count.

It's sort of a reverse greylist: instead of blocking a site until it passes the greylist criteria you exempt it based on similar criteria.

--
Apple broke AppleScripting signatures in Mail.app, so no random signatures.

Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Viktor Dukhovni
In reply to this post by Wietse Venema
On Sat, Sep 30, 2017 at 09:18:44AM -0400, Wietse Venema wrote:

> > When the concurrency limit for "[addr]:port" permits, the queue
> > manager would call smtp(8) again (with the original delivery request
> > and the list of already used addresses, and smtp(8) would attemt
> > delivery to the first address, and if still more recipients remain,
> > refresh the MX host address list, and send back a now 1-longer list
> > of used addresses, and a new next address to try (that is not among
> > those already tried).
> >
> > This would be very complex, I don't expect this to be implemented
> > in the near term.
>
> It could be simple. Suppose there is a 'popular site' counter table
> (the opposite of the 'dead site' list) which is indexed by some
> idea of 'destination' and which is updated in real time as delivery
> agents announce what they are doing. To determine if a next-hop has
> reached its concurrency limit, compare the configured limit for the
> transport with the appropriate 'popular site' count.

Not sure I understand.  An SMTP nexthop corresponds to an unknown
(to the queue manager) set of destination MX hosts.  Lots of domains
have various google, outlook.com, ... MX hosts, but there's no way
to know this without doing the MX lookup first.  How do you propose
to avoid making more than the destination concurrency of connections
to a Gmail or Microsoft, ... MX host?

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Wietse Venema
Viktor Dukhovni:

> On Sat, Sep 30, 2017 at 09:18:44AM -0400, Wietse Venema wrote:
>
> > > When the concurrency limit for "[addr]:port" permits, the queue
> > > manager would call smtp(8) again (with the original delivery request
> > > and the list of already used addresses, and smtp(8) would attemt
> > > delivery to the first address, and if still more recipients remain,
> > > refresh the MX host address list, and send back a now 1-longer list
> > > of used addresses, and a new next address to try (that is not among
> > > those already tried).
> > >
> > > This would be very complex, I don't expect this to be implemented
> > > in the near term.
> >
> > It could be simple. Suppose there is a 'popular site' counter table
> > (the opposite of the 'dead site' list) which is indexed by some
> > idea of 'destination' and which is updated in real time as delivery
> > agents announce what they are doing. To determine if a next-hop has
> > reached its concurrency limit, compare the configured limit for the
> > transport with the appropriate 'popular site' count.
>
> Not sure I understand.  An SMTP nexthop corresponds to an unknown
> (to the queue manager) set of destination MX hosts.  Lots of domains
> have various google, outlook.com, ... MX hosts, but there's no way
> to know this without doing the MX lookup first.  How do you propose
> to avoid making more than the destination concurrency of connections
> to a Gmail or Microsoft, ... MX host?

If a domain has no 'popularity' stats, apply the concurrency limit
to the domain name only. Once a domain has a delivery request in
progress, and there is another request queued for that domain, apply
the concurrency limit to the domain name and to any popularity stats
associated with the domain (assume that mail exchanger lookup results
will be consistent with available popularity stats). Conceptually,
domain popularity stats are pointers to mail exchanger popularity
stats. They are updated when the delivery agent sends a hint, and
automatically decremented when the queue manager to delivery agent
connection is closed. One in progress, a delivery request is not
preempted.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Viktor Dukhovni

> On Oct 2, 2017, at 7:13 AM, Wietse Venema <[hidden email]> wrote:
>
>> An SMTP nexthop corresponds to an unknown
>> (to the queue manager) set of destination MX hosts.  Lots of domains
>> have various google, outlook.com, ... MX hosts, but there's no way
>> to know this without doing the MX lookup first.  How do you propose
>> to avoid making more than the destination concurrency of connections
>> to a Gmail or Microsoft, ... MX host?
>
> If a domain has no 'popularity' stats, apply the concurrency limit
> to the domain name only. Once a domain has a delivery request in
> progress, and there is another request queued for that domain, apply
> the concurrency limit to the domain name and to any popularity stats
> associated with the domain (assume that mail exchanger lookup results
> will be consistent with available popularity stats). Conceptually,
> domain popularity stats are pointers to mail exchanger popularity
> stats. They are updated when the delivery agent sends a hint, and
> automatically decremented when the queue manager to delivery agent
> connection is closed. One in progress, a delivery request is not
> preempted.

This is a best-effort model, which approximates a per-MX concurrency
limit when when the mail stream contains frequent messages to a set
of domains for which the associations with the underlying MX hosts
don't time out between messages.  Greylisting also makes this more
complicated, since on a first delivery attempt all the MX hosts will
tempfail.  Equal-weight MX hosts will make it difficult to apply the
concurrency limit correctly, since we don't know which MX host will
be chosen by the delivery agent...  Perhaps I am making this too
complicated, but I don't yet see how this comes together.

Mind you, if we're not about to start building this any time soon,
the design discussion is perhaps best deferred.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: Resolve before transport

Wietse Venema
In reply to this post by Wietse Venema
Wietse Venema:

> Viktor Dukhovni:
> > On Sat, Sep 30, 2017 at 09:18:44AM -0400, Wietse Venema wrote:
> >
> > > > When the concurrency limit for "[addr]:port" permits, the queue
> > > > manager would call smtp(8) again (with the original delivery request
> > > > and the list of already used addresses, and smtp(8) would attemt
> > > > delivery to the first address, and if still more recipients remain,
> > > > refresh the MX host address list, and send back a now 1-longer list
> > > > of used addresses, and a new next address to try (that is not among
> > > > those already tried).
> > > >
> > > > This would be very complex, I don't expect this to be implemented
> > > > in the near term.
> > >
> > > It could be simple. Suppose there is a 'popular site' counter table
> > > (the opposite of the 'dead site' list) which is indexed by some
> > > idea of 'destination' and which is updated in real time as delivery
> > > agents announce what they are doing. To determine if a next-hop has
> > > reached its concurrency limit, compare the configured limit for the
> > > transport with the appropriate 'popular site' count.
> >
> > Not sure I understand.  An SMTP nexthop corresponds to an unknown
> > (to the queue manager) set of destination MX hosts.  Lots of domains
> > have various google, outlook.com, ... MX hosts, but there's no way
> > to know this without doing the MX lookup first.  How do you propose
> > to avoid making more than the destination concurrency of connections
> > to a Gmail or Microsoft, ... MX host?
>
> If a domain has no 'popularity' stats, apply the concurrency limit
> to the domain name only. Once a domain has a delivery request in
> progress, and there is another request queued for that domain, apply
> the concurrency limit to the domain name and to any popularity stats
> associated with the domain (assume that mail exchanger lookup results
> will be consistent with available popularity stats). Conceptually,
> domain popularity stats are pointers to mail exchanger popularity
> stats. They are updated when the delivery agent sends a hint, and
> automatically decremented when the queue manager to delivery agent
> connection is closed. One in progress, a delivery request is not
> preempted.

In case this isn't obvious, this enforces the shared concurrency
limits only when Postfix is sending multiple messages to the same
recipient domain (for example, personalized mailing lists).

It may be possible to have the queue manager to say  that a delivery
agent has provided a resolved domain result that indicates that the
mail exchangers are getting too much traffic, but that would be a
refinement. That would allow a single email message to be subject
to shared concurrency limits.

        Wietse