VERP uses the recipient name after virtual_regexp rewriting

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

VERP uses the recipient name after virtual_regexp rewriting

Jesper Dybdal-2
I have just installed a mailing list manager (Mailman) for use with my
Postfix installation (which has just been upgraded to 2.5.5).  I have
patched Mailman to use the XVERP option on MAIL FROM.

This works, but I was surprised to see that when the recipient address
provided by Mailman is rewritten by Postfix' virtual_regexp, then the
recipient address that Postfix encodes in the envelope return path is
the rewritten address, rather than the original subscriber address that
Mailman knows.

Since mailing list software using XVERP needs to recognize the address
from the envelope return path as being equal to the subscribed address,
would it not be better to always use the raw address from RCPT TO,
rather than the rewritten one, when creating the VERP'ed return path?

I have not tested this with the 2.6 experimental release, but the
release notes say nothing about VERP, so I assume the behaviour is the
same in 2.6.

(This is not a serious problem for me: the addresses that are rewritten
in my installation are in practice local addresses and it is extremely
unlikely that they will bounce.  But it surprised me.)
--
Jesper Dybdal, Denmark.
http://www.dybdal.dk (in Danish).
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Wietse Venema
Jesper Dybdal:
> I have just installed a mailing list manager (Mailman) for use with my
> Postfix installation (which has just been upgraded to 2.5.5).  I have
> patched Mailman to use the XVERP option on MAIL FROM.
>
> This works, but I was surprised to see that when the recipient address
> provided by Mailman is rewritten by Postfix' virtual_regexp, then the
> recipient address that Postfix encodes in the envelope return path is
> the rewritten address, rather than the original subscriber address that
> Mailman knows.

With VERP, Postfix embeds the envelope recipient address into the
envelope sender address. If the sender is [hidden email], then
the result of embedding is sender+rcptlocal=[hidden email]
(depending on the delimiter settings; default is +=).

If you change the envelope recipient address, then Postfix embeds
the changed recipient. If you change the envelope sender, then
Postfix embeds the recipient into the changed sender address.  

I do not understand why you would send mail to a recipient address
other than the recipient subscribed to the Mailman list.

> Since mailing list software using XVERP needs to recognize the address
> from the envelope return path as being equal to the subscribed address,
> would it not be better to always use the raw address from RCPT TO,
> rather than the rewritten one, when creating the VERP'ed return path?

If I understand this correctly,

1) When you rewrite the envelope RECIPIENT address, then you expect
Postfix VERP to use the original recipient address instead of the
rewritten one.

2) What if you rewrite the envelope SENDER address? Should Postfix
VERP use the original envelope sender address or the rewritten one?

If 1) and 2) work in opposite ways then my little mind will be
really confused.

> I have not tested this with the 2.6 experimental release, but the
> release notes say nothing about VERP, so I assume the behaviour is the
> same in 2.6.

Yes, this project takes pride in accurate documentatiom :-)

        Wietse

> (This is not a serious problem for me: the addresses that are rewritten
> in my installation are in practice local addresses and it is extremely
> unlikely that they will bounce.  But it surprised me.)
> --
> Jesper Dybdal, Denmark.
> http://www.dybdal.dk (in Danish).
>
>

Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

mouss-2
In reply to this post by Jesper Dybdal-2
Jesper Dybdal a écrit :

> I have just installed a mailing list manager (Mailman) for use with my
> Postfix installation (which has just been upgraded to 2.5.5).  I have
> patched Mailman to use the XVERP option on MAIL FROM.
>
> This works, but I was surprised to see that when the recipient address
> provided by Mailman is rewritten by Postfix' virtual_regexp, then the
> recipient address that Postfix encodes in the envelope return path is
> the rewritten address, rather than the original subscriber address that
> Mailman knows.
>
> Since mailing list software using XVERP needs to recognize the address
> from the envelope return path as being equal to the subscribed address,

Really? AFAIK, most list managers use the From: header.

> would it not be better to always use the raw address from RCPT TO,
> rather than the rewritten one, when creating the VERP'ed return path?
>
> I have not tested this with the 2.6 experimental release, but the
> release notes say nothing about VERP, so I assume the behaviour is the
> same in 2.6.
>
> (This is not a serious problem for me: the addresses that are rewritten
> in my installation are in practice local addresses and it is extremely
> unlikely that they will bounce.  But it surprised me.)

Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Jesper Dybdal-2
In reply to this post by Wietse Venema
On Mon, 29 Dec 2008 16:21:41 -0500 (EST), [hidden email] (Wietse
Venema) wrote:

>I do not understand why you would send mail to a recipient address
>other than the recipient subscribed to the Mailman list.

Example:

My server (for Mailman and Postfix and some local mailboxes) is named
nuser.dybdal.dk.  The Mailman virtual host is lists.dybdal.dk, the list
I am experimenting with is named jdtest.

The subscribed address that Mailman knows is jd-test2 at dybdal.dk.

In virtual_regexp, this address is translated to jd+dybdal.dk+jd-test2
at nuser.dybdal.dk.  I do this in order to have it delivered to the "jd"
mailbox at nuser.dybdal, with an extension that allows my mail client to
recognize the precise virtual domain and address that was used (the same
mailbox receives mail to several addresses in several domains).

When Mailman sends mail to jd-test2 at dybdal.dk, the address in RCPT TO
is "jd-test2 at dybdal.dk", but the envelope sender becomes
jdtest-bounces+jd+dybdal.dk+jd-test2=nuser.dybdal.dk at lists.dybdal.dk.
If this should bounce (which it won't in this example, but it might
possible be rewritten to an address at another machine instead), Mailman
will try to find a subscriber with the address
[hidden email], and fail to do so.

>1) When you rewrite the envelope RECIPIENT address, then you expect
>Postfix VERP to use the original recipient address instead of the
>rewritten one.

I think that would make sense, because the VERP'ed recipient address is
used (only) for comparison with the subscribed address when the mail
bounces.

>2) What if you rewrite the envelope SENDER address? Should Postfix
>VERP use the original envelope sender address or the rewritten one?

I had not considered that, since I have no desire to rewrite the sender
address.  But I think my answer to that question is "no".

>If 1) and 2) work in opposite ways then my little mind will be
>really confused.

I may well have misunderstood something, but it seems to me that:

1) The purpose of the VERP encoded recipient address is (only!) to allow
mailing list software to recognize a subscriber, and it therefore makes
sense to have the VERP encoded recipient address equal to the subscriber
address as the mailing list software knows it; i.e., the RCPT TO
address.  This VERP use of the recipient address is quite different from
the primary purpose of the recipient address, which is to get the mail
to wherever the owner of the address wants it, which of course may
require rewriting.

2) The purpose of the sender address (whether or not is has a VERP part
appended) is to ensure that a bounce is delivered correctly; if any
rewriting is specified for the sender address, surely whoever made the
rewriting rule has ensured that the rewritten address will be delivered
correctly.  The mailing list software does not compare the sender
address with anything; it just notes that it received a message at its
bounce address.

Since my first mail, I have tried an experiment where the rewriting of
the sender address is done by a .forward file instead of by
virtual_regexp; in that case, VERP actually uses the recipient address
before it has been changed by .forward, as I would like it to do.

Perhaps part of my problem is that I don't really see why it should make
a difference to the VERP address whether the recipient address is
changed by virtual_regexp or by .forward.

>> I have not tested this with the 2.6 experimental release, but the
>> release notes say nothing about VERP, so I assume the behaviour is the
>> same in 2.6.
>
>Yes, this project takes pride in accurate documentatiom :-)

You don't really need the smiley - the pride is very appropriate.
Accurate documentation, including complete release notes, is something
that I, and undoubtedly many others, very much appreciate about Postfix
- and miss in many many other software products.
--
Jesper Dybdal, Denmark.
http://www.dybdal.dk (in Danish).
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Jesper Dybdal-2
In reply to this post by mouss-2
On Mon, 29 Dec 2008 22:30:53 +0100, mouss <[hidden email]> wrote:

>Jesper Dybdal a écrit :
>> Since mailing list software using XVERP needs to recognize the address
>> from the envelope return path as being equal to the subscribed address,
>
>Really? AFAIK, most list managers use the From: header.

The point in VERP is that the list subscriber that bounces can be
recognized by the address that the bounce is sent to.  This is a much
safer way to identify the subscriber than any attempt to parse the
bounce message to determine which address actually bounced.
--
Jesper Dybdal, Denmark.
http://www.dybdal.dk (in Danish).
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

mouss-2
Jesper Dybdal a écrit :

> On Mon, 29 Dec 2008 22:30:53 +0100, mouss <[hidden email]> wrote:
>
>> Jesper Dybdal a écrit :
>>> Since mailing list software using XVERP needs to recognize the address
>>> from the envelope return path as being equal to the subscribed address,
>> Really? AFAIK, most list managers use the From: header.
>
> The point in VERP is that the list subscriber that bounces can be
> recognized by the address that the bounce is sent to.  This is a much
> safer way to identify the subscriber than any attempt to parse the
> bounce message to determine which address actually bounced.

I misunderstood. I thought you were talking about checking that a post
is from a member (before resending).


Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Jesper Dybdal-2
In reply to this post by Jesper Dybdal-2
On Tue, 30 Dec 2008 01:10:16 +0100, I wrote:

>Since my first mail, I have tried an experiment where the rewriting of
>the sender address is done by a .forward file instead of by
>virtual_regexp; in that case, VERP actually uses the recipient address
>before it has been changed by .forward, as I would like it to do.

That should of course be "rewriting of the *recipient* address", not
"sender address".
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Wietse Venema
Jesper Dybdal:
> On Tue, 30 Dec 2008 01:10:16 +0100, I wrote:
>
> >Since my first mail, I have tried an experiment where the rewriting of
> >the sender address is done by a .forward file instead of by
> >virtual_regexp; in that case, VERP actually uses the recipient address
> >before it has been changed by .forward, as I would like it to do.
>
> That should of course be "rewriting of the *recipient* address", not
> "sender address".

FYI, Postfix implements VERP embedding in the queue manager, not
in the individual delivery agents (one exception had to be added
later, to handle the case where local(8) replaces the envelope
sender upon delivery to an alias that has an owner-alias).

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Jesper Dybdal-2
In reply to this post by Jesper Dybdal-2
On Mon, 29 Dec 2008 21:54:52 +0100, I wrote:

>... I was surprised to see that when the recipient address
>provided by Mailman is rewritten by Postfix' virtual_regexp, then the
>recipient address that Postfix encodes in the envelope return path is
>the rewritten address, rather than the original subscriber address that
>Mailman knows.

I have just realized that there is another way to look at this, which
may be a better argument for the semantics I would like:

The problem occurs only because the sending server and the receiving
server is the same; the recipient address is in a domain handled by the
same postfix instance that Mailman uses to submit mail.  If there were
two independent postfix instances, this would not happen.

In such a case, it seems to me that the result ought to be the same as
if processing clearly related to the sending side, such as VERP address
generation, happened before processing clearly clearly related to the
receiving side, such as recipient address rewriting in virtual_maps.

I.e., VERP belongs to "sending processing" and its result should
therefore not depend on virtual_maps rewriting, which are part of the
"receiving processing" and thus belongs logically "later"; it comes into
effect in the same postfix instance only because the subscriber happens
to be a local user.

(But as I wrote earlier, I can live with the current semantics, and this
will - probably - be my last attempt to convince you that the order
ought to be different.)
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Wietse Venema
Jesper Dybdal:

> On Mon, 29 Dec 2008 21:54:52 +0100, I wrote:
>
> >... I was surprised to see that when the recipient address
> >provided by Mailman is rewritten by Postfix' virtual_regexp, then the
> >recipient address that Postfix encodes in the envelope return path is
> >the rewritten address, rather than the original subscriber address that
> >Mailman knows.
>
> I have just realized that there is another way to look at this, which
> may be a better argument for the semantics I would like:
>
> The problem occurs only because the sending server and the receiving
> server is the same; the recipient address is in a domain handled by the
> same postfix instance that Mailman uses to submit mail.  If there were
> two independent postfix instances, this would not happen.
>
> In such a case, it seems to me that the result ought to be the same as
> if processing clearly related to the sending side, such as VERP address
> generation, happened before processing clearly clearly related to the
> receiving side, such as recipient address rewriting in virtual_maps.
>
> I.e., VERP belongs to "sending processing" and its result should
> therefore not depend on virtual_maps rewriting, which are part of the
> "receiving processing" and thus belongs logically "later"; it comes into
> effect in the same postfix instance only because the subscriber happens
> to be a local user.
>
> (But as I wrote earlier, I can live with the current semantics, and this
> will - probably - be my last attempt to convince you that the order
> ought to be different.)

You don't need to convince me of anything.  Where possible Postfix
is based on general principles that solve a larger class of problems,
instead of a bunch of special-purpose hacks that tend to get in
the way when I want to add other features in the same area.

In your case, the general principle was not obvious: VERP is a form
of input processing. Fortunately, Postfix has original recipient
information at hand. Unfortunately, the information is not guaranteed
to be in the canonical user@domain form. However, in the special
case of VERP this is OK. The consumer of VERP bounces really wants
to see the same string that it gave to the MTA.

        Wietse

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: VERP uses the recipient name after virtual_regexp rewriting

Jesper Dybdal-2
On Fri, 2 Jan 2009 15:25:14 -0500 (EST), [hidden email] (Wietse
Venema) wrote:

>Fortunately, Postfix has original recipient
>information at hand. Unfortunately, the information is not guaranteed
>to be in the canonical user@domain form. However, in the special
>case of VERP this is OK.

I'm glad to hear that sufficient information is available at the
relevant point in Postfix' processing.

>The consumer of VERP bounces really wants
>to see the same string that it gave to the MTA.

We agree completely then.  Thanks for your response (and for Postfix in
general - it is excellent software).