Technical question to Postfix

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Technical question to Postfix

Christian Rößner-2
Hi,

I have thought a while, if I can ask that here, but as it just is a technical question, I hope you understand that I am just interested to understand something and not to make another feature request. So with respect to your time:

As an example, we have smtpd_proxy_filter. From my understanding that receives a mail and hands it over to a real time content filter that should be designed to return the mail afterwards back into postfix. I guess that is, because of cleanup, trivial-rewrite and qmgr and stuff.

As more and more system are not the final destination in terms of moving a received mail directly into a Maildir folder or calling procmail/maildrop, they use LMTP to connect to Cyrus or Dovecot (simplified spoken). So the mail is received and cleanup/trivial-rewrite, qmgr are doing their jobs. After that LMTP is called and Postfix tries to "relay" it over to i.e. Dovecot, which might have sieve active or quota. In both cases it could be a problem, if Dovecot rejects the mail (not temp failures).

I hope my understanding was right so far.

Would it technically possible to have a smtpd_to_lmtp_proxy option (or however it could be called), that would receive on smtpd and open a connection to its LMTP server, doing cleanup and Co. in memory and wait for the result of the LMTP server? If the LMTP process gets 250 OK, postfix would give that back to the client. Else closing the session with error status code received from the LMTP server.

RFC2033:


3.  Introduction and Overview

   This queuing requirement is beneficial in the situation for which
   SMTP was originally designed: store-and-forward relay of mail between
   networked hosts.  In some limited situations, it is desirable to have
   a server which does not manage a queue, instead relying on the client
   to perform queue management.  As an example, consider a hypothetical
   host with a mail system designed as follows:


Positive aspects:

- sieve reject being no problem
- over quota bounce
- speed

If I understood LMTP correctly, we would have a problem with recipients that produce a failure. How would the SMTP session behave? I don't know, how Postfix is doing this at the moment.

Furthermore using some kind of smtpd_to_lmtp_proxy would mean not to use qmgr, as we would not queue the mail on persistent storage. Is that right?

I am far away from being an postfix expert, so this is just trivial thinking about something that might be extremely complex to accomplish. And I am always willing to learn and to understand :)

Thanks for reading. And thanks in advance for an answer.

Kind regards

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Reinaldo Gil Lima de Carvalho
You can use a policy daemon to check quota on cyrus.

[]'s

Reinaldo de Carvalho

Em 04/11/2012, às 14:16, Christian Rößner <[hidden email]> escreveu:

> Hi,
>
> I have thought a while, if I can ask that here, but as it just is a technical question, I hope you understand that I am just interested to understand something and not to make another feature request. So with respect to your time:
>
> As an example, we have smtpd_proxy_filter. From my understanding that receives a mail and hands it over to a real time content filter that should be designed to return the mail afterwards back into postfix. I guess that is, because of cleanup, trivial-rewrite and qmgr and stuff.
>
> As more and more system are not the final destination in terms of moving a received mail directly into a Maildir folder or calling procmail/maildrop, they use LMTP to connect to Cyrus or Dovecot (simplified spoken). So the mail is received and cleanup/trivial-rewrite, qmgr are doing their jobs. After that LMTP is called and Postfix tries to "relay" it over to i.e. Dovecot, which might have sieve active or quota. In both cases it could be a problem, if Dovecot rejects the mail (not temp failures).
>
> I hope my understanding was right so far.
>
> Would it technically possible to have a smtpd_to_lmtp_proxy option (or however it could be called), that would receive on smtpd and open a connection to its LMTP server, doing cleanup and Co. in memory and wait for the result of the LMTP server? If the LMTP process gets 250 OK, postfix would give that back to the client. Else closing the session with error status code received from the LMTP server.
>
> RFC2033:
>
> …
> 3.  Introduction and Overview
> …
>   This queuing requirement is beneficial in the situation for which
>   SMTP was originally designed: store-and-forward relay of mail between
>   networked hosts.  In some limited situations, it is desirable to have
>   a server which does not manage a queue, instead relying on the client
>   to perform queue management.  As an example, consider a hypothetical
>   host with a mail system designed as follows:
> …
>
> Positive aspects:
>
> - sieve reject being no problem
> - over quota bounce
> - speed
>
> If I understood LMTP correctly, we would have a problem with recipients that produce a failure. How would the SMTP session behave? I don't know, how Postfix is doing this at the moment.
>
> Furthermore using some kind of smtpd_to_lmtp_proxy would mean not to use qmgr, as we would not queue the mail on persistent storage. Is that right?
>
> I am far away from being an postfix expert, so this is just trivial thinking about something that might be extremely complex to accomplish. And I am always willing to learn and to understand :)
>
> Thanks for reading. And thanks in advance for an answer.
>
> Kind regards
>
> -Christian Rößner
>
> --
> [*] sys4 AG
>
> http://sys4.de, +49 (89) 30 90 46 64
> Franziskanerstraße 15, 81669 München
>
> Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
> Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
> Aufsichtsratsvorsitzender: Joerg Heidrich
>
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Reindl Harald-2
In reply to this post by Christian Rößner-2


Am 04.11.2012 18:16, schrieb Christian Rößner:
> Would it technically possible to have a smtpd_to_lmtp_proxy option (or however it could be called), that would receive on smtpd and open a connection to its LMTP server, doing cleanup and Co. in memory and wait for the result of the LMTP server? If the LMTP process gets 250 OK, postfix would give that back to the client. Else closing the session with error status code received from the LMTP server.

quota and such things can be done with a policyd, below a example for dbmail
___________________________

generally wait for a 250 OK from LMTP would be dumb

 * think about performance
 * think about temporary LMTP problems
 * consider how make a difference hard/soft
 * you do NOT want a hard-bounce obly because LMTP hangs

some months ago i did as example a major-upgrade on dbmail
i stopped imap/pop3 and closed submission port
but we received new messages due the whole migration
after that "postqueue -f" delivered all of them to the inboxes
___________________________

https://github.com/jnorell/dbmail-postfix-policyd
smtpd_recipient_restrictions = check_policy_service unix:/var/spool/postfix/dbmail-postfix-policyd/socket


signature.asc (267 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Viktor Dukhovni
In reply to this post by Christian Rößner-2
On Sun, Nov 04, 2012 at 06:16:26PM +0100, Christian R??ner wrote:

> Would it technically possible to have a smtpd_to_lmtp_proxy option
> (or however it could be called), that would receive on smtpd and
> open a connection to its LMTP server, doing cleanup and Co. in
> memory and wait for the result of the LMTP server?

No. The two protocols have incompatible transaction models. LMTP
can accept a subset of the recipients of a message after "DATA",
while SMTP cannot. Some of the recipients can rejected and the
rest accepted. Or some can be temp-failed.

Therefore the SMTP-to-LMTP proxy has to be willing to:

  - Send bounces for rejected recipients: this requires a queue

  - Retry tempfailed recipients: this requires a queue

One could image a proxy that only does this as necessary, while
avoiding queues for mail for which all deliveries give the same
result for all recipients, but the complexity of this is not
appealing.

This also creates more extreme load spikes on the IMAP delivery
system which is more expensive than SMTP queue processing. With
Postfix between the sender and the IMAP server, the load is smoothed
out by the maximum concurrency of the LMTP transport, the Postfix
queue acts as a buffer between bursty clients and the IMAP server.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Wietse Venema
In reply to this post by Christian Rößner-2
Christian R??ner:
> Would it technically possible to have a smtpd_to_lmtp_proxy option
> (or however it could be called), that would receive on smtpd and
> open a connection to its LMTP server, doing cleanup and Co. in
> memory and wait for the result of the LMTP server? If the LMTP

This does not scale. The problem is that you find out if a recipient
is over quota AFTER Postfix has already allocated the maximum amount
of mail server resources.

The scalable way is to make the quota check BEFORE allocating more
mail server resources. As has been suggested many times over the
past 13 years, this means using an SMTPD access map or policy daemon
to block mail for over-quota recipients.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2
In reply to this post by Reindl Harald-2
Hi,

>> Would it technically possible to have a smtpd_to_lmtp_proxy option (or however it could be called), that would receive on smtpd and open a connection to its LMTP server, doing cleanup and Co. in memory and wait for the result of the LMTP server? If the LMTP process gets 250 OK, postfix would give that back to the client. Else closing the session with error status code received from the LMTP server.
>
> quota and such things can be done with a policyd, below a example for dbmail
> ___________________________
>
> generally wait for a 250 OK from LMTP would be dumb

But you have to wait for that with smtpd_proxy_filter as well ;-)

>
> * think about performance

Improvmnet. No queuing on MTA side

> * think about temporary LMTP problems

450 Service currently not available. Same behavior than with Milter problems.

> * consider how make a difference hard/soft

Can you give mor details on that?

> * you do NOT want a hard-bounce obly because LMTP hangs

Why? If I can not hand over mail to the LMTP server, the remote MTA would still queue the mail and retry later.

> some months ago i did as example a major-upgrade on dbmail
> i stopped imap/pop3 and closed submission port
> but we received new messages due the whole migration
> after that "postqueue -f" delivered all of them to the inboxes

See above.

Kind regards

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2
In reply to this post by Wietse Venema
Hi Wietse,

>> Would it technically possible to have a smtpd_to_lmtp_proxy option
>> (or however it could be called), that would receive on smtpd and
>> open a connection to its LMTP server, doing cleanup and Co. in
>> memory and wait for the result of the LMTP server? If the LMTP
>
> This does not scale. The problem is that you find out if a recipient
> is over quota AFTER Postfix has already allocated the maximum amount
> of mail server resources.
>
> The scalable way is to make the quota check BEFORE allocating more
> mail server resources. As has been suggested many times over the
> past 13 years, this means using an SMTPD access map or policy daemon
> to block mail for over-quota recipients.


Ok, I can understand that. Thank you very much for this feedback.

Kind regards

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Wietse Venema
In reply to this post by Wietse Venema
Wietse Venema:

> Christian R??ner:
> > Would it technically possible to have a smtpd_to_lmtp_proxy option
> > (or however it could be called), that would receive on smtpd and
> > open a connection to its LMTP server, doing cleanup and Co. in
> > memory and wait for the result of the LMTP server? If the LMTP
>
> This does not scale. The problem is that you find out if a recipient
> is over quota AFTER Postfix has already allocated the maximum amount
> of mail server resources.
>
> The scalable way is to make the quota check BEFORE allocating more
> mail server resources. As has been suggested many times over the
> past 13 years, this means using an SMTPD access map or policy daemon
> to block mail for over-quota recipients.

Or use "reject_unverified_recipient", which uses a cache
of previous decisions so it won't hammer the mailbox server.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Reindl Harald-2
In reply to this post by Christian Rößner-2


Am 04.11.2012 19:05, schrieb Christian Rößner:

> Hi,
>
>>> Would it technically possible to have a smtpd_to_lmtp_proxy option (or however it could be called), that would receive on smtpd and open a connection to its LMTP server, doing cleanup and Co. in memory and wait for the result of the LMTP server? If the LMTP process gets 250 OK, postfix would give that back to the client. Else closing the session with error status code received from the LMTP server.
>>
>> quota and such things can be done with a policyd, below a example for dbmail
>> ___________________________
>>
>> generally wait for a 250 OK from LMTP would be dumb
>
> But you have to wait for that with smtpd_proxy_filter as well ;-)
>
>>
>> * think about performance
>
> Improvmnet. No queuing on MTA side
this is NOT a improvement
this degrades performance DRAMATICALLY

don't get me wrong but the queuing was developed by
smarter people you understand mail-flow and to
think "No queuing" is a improvement shows only
a lot of missing knowledge

usually people not undertsand how e-mail works coming
up with such solutions not realizinh that they only make
a huge damage to their mail-system and weeks later they
come back here and whine for help

>> * think about temporary LMTP problems
>
> 450 Service currently not available. Same behavior than with Milter problems.

oh so you do NOT want always wait for 250 OK

>> * consider how make a difference hard/soft
> Can you give mor details on that?

hard: do not try again do deliver this message
soft: temporary error

i have seen way to often people trying what you like resulting in
hard-bounces by the first delivery where you can see the origin
"450 Service currently not available" of the last hop or the
opposite where a poor configured mailsystem anserws with 450
where you can see their last hop's reply is a "user does not exist"

>> * you do NOT want a hard-bounce obly because LMTP hangs
> Why? If I can not hand over mail to the LMTP server, the remote MTA would
> still queue the mail and retry later

because if i know the RCPT is valid there is NO reason
to reject in cause of temprary internal errors, if
this happens longer the remote MTA would have a large delay
while the postfix  queue would deliver immeditially after
the problem is fixed

>> some months ago i did as example a major-upgrade on dbmail
>> i stopped imap/pop3 and closed submission port
>> but we received new messages due the whole migration
>> after that "postqueue -f" delivered all of them to the inboxes
>
> See above

NO, not "see above"

it is the wanted behavior not to reject messages if i know
their RCPT is valid and i have a queue which is EXACTLY
for such situations


signature.asc (267 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2
In reply to this post by Reinaldo Gil Lima de Carvalho
Hi,

> You can use a policy daemon to check quota on cyrus.
>
> []'s


I have written a python policy service that can check for "over quota". It is a basic version that runs with Dovecot here. It has the following limitations:

- Dovecot must have been configured with a master user
- It requires imaps port 993

It connects to the Dovecot server over IMAP protocol and logs in as user and gets quota information. On that decision it does DUNNO or REJECT.

I think about using memcached with some TTL (for example 10 minutes), but not sure, if this is required.

If someone wants to have a copy of that, please contact me off list.

Thanks

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2
In reply to this post by Wietse Venema
Hi,

> Or use "reject_unverified_recipient", which uses a cache
> of previous decisions so it won't hammer the mailbox server.


thank you very much. Tested it and the solution is so simple and at the same time so powerful. I wonder that there are still many people not thinking about this solution and pointing to policy-services.

Kind regards

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Viktor Dukhovni
On Mon, Nov 05, 2012 at 02:11:56PM +0100, Christian R??ner wrote:

> > Or use "reject_unverified_recipient", which uses a cache
> > of previous decisions so it won't hammer the mailbox server.
>
>
> thank you very much. Tested it and the solution is so simple and
> at the same time so powerful. I wonder that there are still many
> people not thinking about this solution and pointing to policy-services.

Quick question: With Dovecot (or other LMTP servers) does this
detect over-quota conditions? The Postfix verification probe does
not send any message data, just "MAIL FROM", "RCPT TO" and then
"RSET" + "QUIT" (or perhaps just "QUIT").

So if the LMTP server does not reject "RCPT TO" for over-quota
recipients, the probe will only confirm recipient existence. Of
course an LMTP server should ideally detect over-quota before
message transfer begins, so perhaps Dovecot and other implementations
do that.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Wietse Venema
Viktor Dukhovni:

> On Mon, Nov 05, 2012 at 02:11:56PM +0100, Christian R??ner wrote:
>
> > > Or use "reject_unverified_recipient", which uses a cache
> > > of previous decisions so it won't hammer the mailbox server.
> >
> >
> > thank you very much. Tested it and the solution is so simple and
> > at the same time so powerful. I wonder that there are still many
> > people not thinking about this solution and pointing to policy-services.
>
> Quick question: With Dovecot (or other LMTP servers) does this
> detect over-quota conditions? The Postfix verification probe does
> not send any message data, just "MAIL FROM", "RCPT TO" and then
> "RSET" + "QUIT" (or perhaps just "QUIT").

Postfix sends the message size in the MAIL FROM command.  If the
recipient is already over quota, then the LMTP server can reject
the message regardless of size. Otherwise, the LMTP server can
decide if the message would put the recipient over quota.

If this is indeed how Dovecot works, then quota checks via
reject_unverified_recipient would be a good alternative, compared
to policy servers or access maps.

        Wietse

> So if the LMTP server does not reject "RCPT TO" for over-quota
> recipients, the probe will only confirm recipient existence. Of
> course an LMTP server should ideally detect over-quota before
> message transfer begins, so perhaps Dovecot and other implementations
> do that.
>
> --
> Viktor.
>
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2
Hi,

>> Quick question: With Dovecot (or other LMTP servers) does this
>> detect over-quota conditions? The Postfix verification probe does
>> not send any message data, just "MAIL FROM", "RCPT TO" and then
>> "RSET" + "QUIT" (or perhaps just "QUIT").
>
> Postfix sends the message size in the MAIL FROM command.  If the
> recipient is already over quota, then the LMTP server can reject
> the message regardless of size. Otherwise, the LMTP server can
> decide if the message would put the recipient over quota.
>
> If this is indeed how Dovecot works, then quota checks via
> reject_unverified_recipient would be a good alternative, compared
> to policy servers or access maps.
>
> Wietse
>
>> So if the LMTP server does not reject "RCPT TO" for over-quota
>> recipients, the probe will only confirm recipient existence. Of
>> course an LMTP server should ideally detect over-quota before
>> message transfer begins, so perhaps Dovecot and other implementations
>> do that.

I also will test, if "sieve reject" is working the same way. If so, I can enable this flag and give users a chance to reject unwanted mails in session.

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2

Am 06.11.2012 um 08:31 schrieb Christian Rößner <[hidden email]>:

> I also will test, if "sieve reject" is working the same way. If so, I can enable this flag and give users a chance to reject unwanted mails in session.

"reject" creates a new mail and sends it out.

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Nikolaos Milas
In reply to this post by Wietse Venema
On 4/11/2012 8:17 μμ, Wietse Venema wrote:

> Or use "reject_unverified_recipient", which uses a cache
> of previous decisions so it won't hammer the mailbox server.

A clarification: Does the cache of "reject_unverified_recipient"
decisions include the result of "relay_recipient_maps" lookups?

This might be esp. useful in case the "relay_recipient_maps" setting is
blank, in which case the final destination server(s) is (are) queried
for the recipient (and should reply with a 550 if the recipient does not
exist).

Thanks,
Nick
Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Christian Rößner-2
>> Or use "reject_unverified_recipient", which uses a cache
>> of previous decisions so it won't hammer the mailbox server.
>
> A clarification: Does the cache of "reject_unverified_recipient" decisions include the result of "relay_recipient_maps" lookups?
>
> This might be esp. useful in case the "relay_recipient_maps" setting is blank, in which case the final destination server(s) is (are) queried for the recipient (and should reply with a 550 if the recipient does not exist).

I did some tests yesterday. Removing relay_reciepient_maps and virtual_alias_maps completely. It works, but I have the feeling it takes a little bit longer than asking LDAP over proxymap. Furthermore I want the possibility of email forwarding, so I re-added both options. But in general that works.

-Christian Rößner

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Reply | Threaded
Open this post in threaded view
|

Re: Technical question to Postfix

Wietse Venema
In reply to this post by Nikolaos Milas
Nikolaos Milas:
> On 4/11/2012 8:17 ??, Wietse Venema wrote:
>
> > Or use "reject_unverified_recipient", which uses a cache
> > of previous decisions so it won't hammer the mailbox server.
>
> A clarification: Does the cache of "reject_unverified_recipient"
> decisions include the result of "relay_recipient_maps" lookups?

As documented, reject_unverified_recipient caches whether or not
recipient address verification was successful. It caches the RESULT.
It DOES NOT cache HOW Postfix came to that conclusion.

It is as fast or faster then LDAP, once the result is in cache.

        Wietse