Quantcast

Any best practices for stacking filters?

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Any best practices for stacking filters?

Quanah Gibson-Mount-3
Right now, we can have up to 4 different processing filters in our
configuration, based on what features are enabled.

In general, we always have:

1) OpenDKIM for signing

Then we almost always have

2) Amavis

Then we sometimes have

3) A Zimbra written Milter service

And rarely

4) A journaling milter


For OpenDKIM, I currently have it set up as a content_filter, which works
well as long as Amavis is not also enabled.

smtp      inet  n       -       n       -       -       smtpd
    -o content_filter=scan:[127.0.0.1]:10029

If Amavis is enabled, it is called as originating vs foreign via:
postconf smtpd_sender_restrictions
smtpd_sender_restrictions = check_sender_access
regexp:/opt/zimbra/postfix/conf/tag_as_originating.re, permit_mynetworks,
permit_sasl_authenticated, permit_tls_clientcerts, check_sender_access
regexp:/opt/zimbra/postfix/conf/tag_as_foreign.re

zimbra@zre-ldap002:~/postfix/conf$ cat tag_as_originating.re
/^/  FILTER smtp-amavis:[127.0.0.1]:10026

zimbra@zre-ldap002:~/postfix/conf$ cat tag_as_foreign.re
/^/  FILTER smtp-amavis:[127.0.0.1]:10024

This of course overrides the content_filter line, so OpenDKIM never fires
there.  I fixed that by adding -o smtpd_milters=inet:localhost:8465 to the
port 10025 re-injection smtpd for Amavis.  However, this causes OpenDKIM to
run after Amavis executes, when I want it to run prior to Amavis.

The milter server is handled by setting smtpd_milters via postconf, which I
haven't tested yet, but I'm pretty certain will override the setting to
Amavis.

The journaling server is handled via smtpd_proxy_filter to the smtp process:
        -o smtpd_proxy_filter=127.0.0.1:10027

So is there an easy/best way to get things to all work & play together at
once?  I believe if I move OpenDKIM to be a smtpd_proxy_filter, that will
allow me to force it to execute before Amavis, but then what do I do for
the journaling server?  Basically, trying to figure out the best way to
force things to flow in the sequence I require which is:

Journaling (if enabled)
OpenDKIM
Amavis (if enabled)
Milter server (if enabled)

Any insights appreciated!

Thanks,
Quanah

--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:

> Right now, we can have up to 4 different processing filters in our
> configuration, based on what features are enabled.
>
> In general, we always have:
>
> 1) OpenDKIM for signing
>
> Then we almost always have
>
> 2) Amavis
>
> Then we sometimes have
>
> 3) A Zimbra written Milter service
>
> And rarely
>
> 4) A journaling milter

I can't share experiences with configurations of similar complexity,
just want to make a reminder that Milters don't work (well) before
a proxy filter. In particular the Milter will not receive header,
body, or end-of-body events, nor will it be able to send requests
to modify the envelope or message content.  Those features are
implemented in the cleanup daemon as it writes the message to queue
file, but of course there are no cleanup daemon or queue file
before a proxy filter.

network -> smtpd -> proxy filter -> smtpd -> cleanup -> queue file
             |
        buffer file

It would not require a lot of code to send header, body, or end-of-body
events to a Milter before a proxy filter. However, requests to
modify the envelope or message content would require significant
coding, and without those editing features, Milter support would
still be crippled.

One approach is to reuse the buffer file that is created with
"smtpd_proxy_options = speed_adjust", and to make the queue file
editing code work in that environment.

It's much easier to tell people not to use Milters before a proxy
filter...

    Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

DTNX Postmaster
In reply to this post by Quanah Gibson-Mount-3
On Oct 18, 2012, at 00:56, Quanah Gibson-Mount wrote:

> Right now, we can have up to 4 different processing filters in our configuration, based on what features are enabled.
>
> In general, we always have:
>
> 1) OpenDKIM for signing
>
> Then we almost always have
>
> 2) Amavis
>
> Then we sometimes have
>
> 3) A Zimbra written Milter service
>
> And rarely
>
> 4) A journaling milter
>
>
> For OpenDKIM, I currently have it set up as a content_filter, which works well as long as Amavis is not also enabled.
>
> smtp      inet  n       -       n       -       -       smtpd
>   -o content_filter=scan:[127.0.0.1]:10029
>
> If Amavis is enabled, it is called as originating vs foreign via:
> postconf smtpd_sender_restrictions
> smtpd_sender_restrictions = check_sender_access regexp:/opt/zimbra/postfix/conf/tag_as_originating.re, permit_mynetworks, permit_sasl_authenticated, permit_tls_clientcerts, check_sender_access regexp:/opt/zimbra/postfix/conf/tag_as_foreign.re
>
> zimbra@zre-ldap002:~/postfix/conf$ cat tag_as_originating.re
> /^/  FILTER smtp-amavis:[127.0.0.1]:10026
>
> zimbra@zre-ldap002:~/postfix/conf$ cat tag_as_foreign.re
> /^/  FILTER smtp-amavis:[127.0.0.1]:10024
>
> This of course overrides the content_filter line, so OpenDKIM never fires there.  I fixed that by adding -o smtpd_milters=inet:localhost:8465 to the port 10025 re-injection smtpd for Amavis.  However, this causes OpenDKIM to run after Amavis executes, when I want it to run prior to Amavis.

What if you added '-o smtpd_milters=inet:localhost:8465' to the smtp line mentioned above, to call OpenDKIM, instead of the content_filter? Would that be overridden too by your FILTER action for Amavis?

If it's not, then it may be as simple as listing your journaling and OpenDKIM milters there, and adding the milter server to the Amavis reinjection smtpd, if Amavis is enabled, or as a third argument to 'smtpd_milters' if Amavis is disabled, but your milter server enabled?

My understanding of how milters interact is limited, however, as I have no practical experience with more than one. I could very well be misunderstanding the complexities of a workflow like this.

HTH,
Jona

--

> The milter server is handled by setting smtpd_milters via postconf, which I haven't tested yet, but I'm pretty certain will override the setting to Amavis.
>
> The journaling server is handled via smtpd_proxy_filter to the smtp process:
>       -o smtpd_proxy_filter=127.0.0.1:10027
>
> So is there an easy/best way to get things to all work & play together at once?  I believe if I move OpenDKIM to be a smtpd_proxy_filter, that will allow me to force it to execute before Amavis, but then what do I do for the journaling server?  Basically, trying to figure out the best way to force things to flow in the sequence I require which is:
>
> Journaling (if enabled)
> OpenDKIM
> Amavis (if enabled)
> Milter server (if enabled)
>
> Any insights appreciated!
>
> Thanks,
> Quanah
>
> --
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
In reply to this post by Wietse Venema
--On Wednesday, October 17, 2012 7:52 PM -0400 Wietse Venema
<[hidden email]> wrote:


> It's much easier to tell people not to use Milters before a proxy
> filter...

If you use the milter after the proxy server, which is what I'm currently
doing, then I result in the following problem:

If Amavis is called before OpenDKIM via the filter trigger, then Amavis
does DKIM verification on the message before it is actually signed by
OpenDKIM.  So the message gets delivered without having the signing
verified.  This only happens for email between users on the server itself.
Since the filter regex overrides content_filter, I'm not sure how to force
OpenDKIM to execute for signing prior to Amavis executing verification. :/

I.e., I need the OpenDKIM milter to be processed before the proxy filter so
that the email is correctly signed before it is passed to the proxy filter.
Then Amavis can correctly verify the signature prior to delivery.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:
> <[hidden email]> wrote:
> > It's much easier to tell people not to use Milters before a proxy
> > filter...
>
> If you use the milter after the proxy server, which is what I'm currently
> doing, then I result in the following problem:

You just confirmed the limitation that I explained at length, so I
won't repeat that diatribe.

One suggestion I can make is to avoid mixing mail streams from
outside with mail streams from inside, before your mail is signed.

For example,

- Use before-queue filters for mail from outside so that you can
  reject mail before it hits the queue.

- Use after-queue filters for mail from inside. Then, your mail
  from "inside" is not affected by the limitation. You can sign it
  with dkim-milter and the like.

I suspect that you could feed both mail streams into the same Amavis
content filter.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 3:33 PM -0400 Wietse Venema
<[hidden email]> wrote:

> One suggestion I can make is to avoid mixing mail streams from
> outside with mail streams from inside, before your mail is signed.
>
> For example,
>
> - Use before-queue filters for mail from outside so that you can
>   reject mail before it hits the queue.
>
> - Use after-queue filters for mail from inside. Then, your mail
>   from "inside" is not affected by the limitation. You can sign it
>   with dkim-milter and the like.

Hi Wieste,

As I noted in my original mail, I already use the filters to separate out
the streams:

smtpd_sender_restrictions = check_sender_access
regexp:/opt/zimbra/postfix/conf/tag_as_originating.re, permit_mynetworks,
permit_sasl_authenticated, permit_tls_clientcerts, check_sender_access
regexp:/opt/zimbra/postfix/conf/tag_as_foreign.re

zimbra@zre-ldap002:~/postfix/conf$ cat tag_as_originating.re
/^/  FILTER smtp-amavis:[127.0.0.1]:10026

zimbra@zre-ldap002:~/postfix/conf$ cat tag_as_foreign.re
/^/  FILTER smtp-amavis:[127.0.0.1]:10024


So I believe I am already, as you said, diverting the mail into different
streams.  Both of which go to Amavis.  I.e., originating mail gets directed
to amavis on port 10026.  Foreign mail goes to amavis on port 10024.  Which
gets me into the entire problem I'm having now.  Or am I misunderstanding
what you said?

Mail gets re-injected from Amavis to Postfix on port 10025.  Then it is
signed.  The problem is, at that point, Amavis is already done with the
mail.  So again, I think I'm doing what you suggest, but I can't figure out
how to get it to sign the mail via OpenDKIM prior to Amavis processing.

Here's my master.cf again as well:

smtp      inet  n       -       n       -       -       smtpd
    -o content_filter=scan:[127.0.0.1]:10029
465    inet  n       -       n       -       -       smtpd
    -o smtpd_tls_wrappermode=yes
    -o smtpd_sasl_auth_enable=yes
    -o content_filter=scan:[127.0.0.1]:10029
submission inet n      -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_sasl_auth_enable=yes
    -o smtpd_client_restrictions=permit_sasl_authenticated,reject
    -o smtpd_tls_security_level=may
scan      unix  -       -       n       -       10      smtp
    -o smtp_send_xforward_command=yes
    -o disable_mime_output_conversion=yes
    -o smtp_generic_maps=
pickup    fifo  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -   -   n   -   1   scache
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
old-cyrus unix  -       n       n       -       -       pipe
  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
cyrus     unix  -       n       n       -       -       pipe
  user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail
($recipient)
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop
$recipient
smtp-amavis unix -      -       n       -       10  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
127.0.0.1:10025 inet n  -       n       -       -  smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o virtual_mailbox_maps=
    -o virtual_alias_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_milters=inet:localhost:8465
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks_style=host
    -o mynetworks=127.0.0.0/8,[::1]/128
    -o strict_rfc821_envelopes=yes
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o
receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings
    -o local_header_rewrite_clients=
[127.0.0.1]:10029 inet n - n - - smtpd
    -o smtpd_milters=inet:localhost:8465


--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 1:03 PM -0700 Quanah Gibson-Mount
<[hidden email]> wrote:

>
> Hi Wieste,

Wietse even.  Sorry. ;)


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
In reply to this post by Quanah Gibson-Mount-3
Quanah Gibson-Mount:

> --On Monday, October 22, 2012 3:33 PM -0400 Wietse Venema
> <[hidden email]> wrote:
>
> > One suggestion I can make is to avoid mixing mail streams from
> > outside with mail streams from inside, before your mail is signed.
> >
> > For example,
> >
> > - Use before-queue filters for mail from outside so that you can
> >   reject mail before it hits the queue.
> >
> > - Use after-queue filters for mail from inside. Then, your mail
> >   from "inside" is not affected by the limitation. You can sign it
> >   with dkim-milter and the like.
>
> As I noted in my original mail, I already use the filters to separate out
> the streams:

My example CAN sign mail with dkim-milter before it hits the Amavis
filter.

Your example CANNOT sign mail with dkim-milter before it hits the
Amavis filter.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 4:24 PM -0400 Wietse Venema
<[hidden email]> wrote:

> My example CAN sign mail with dkim-milter before it hits the Amavis
> filter.
>
> Your example CANNOT sign mail with dkim-milter before it hits the
> Amavis filter.

I believe what you are saying is that I should adjust my originating filter
to go to another postfix agent, rather than amavis.  That postfix agent
triggers signing, and then passes the mail on to amavis on port 10026.
Correct?

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:

> --On Monday, October 22, 2012 4:24 PM -0400 Wietse Venema
> <[hidden email]> wrote:
>
> > My example CAN sign mail with dkim-milter before it hits the Amavis
> > filter.
> >
> > Your example CANNOT sign mail with dkim-milter before it hits the
> > Amavis filter.
>
> I believe what you are saying is that I should adjust my originating filter
> to go to another postfix agent, rather than amavis.  That postfix agent
> triggers signing, and then passes the mail on to amavis on port 10026.
> Correct?

1) Use the before-queue filter for mail from outside:

                                external clients -> smtpd -> Amavis ...

2) Use the after-queue filter for mail from inside:

    internal clients -> smtpd -> cleanup -> queue -> smtp -> Amavis ...

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 5:09 PM -0400 Wietse Venema
<[hidden email]> wrote:

> 1) Use the before-queue filter for mail from outside:
>
> external clients -> smtpd -> Amavis ...
>
> 2) Use the after-queue filter for mail from inside:
>
>     internal clients -> smtpd -> cleanup -> queue -> smtp -> Amavis ...
>
> Wietse

I'm going to assume you mean something like this then:

smtp      inet  n       -       n       -       -       smtpd
    -o smtpd_proxy_filter=[127.0.0.1]:10029
    -o smtpd_client_connection_count_limit=10
    -o smtpd_proxy_options=speed_adjust



I already tried this, and it is not an acceptable solution, because postfix
will not accept mail if OpenDKIM is not running.  I need Postfix to accept
and queue the email in that scenario, rather than reject it.

Oct 22 14:54:35 zqa-398 postfix/smtpd[2854]: connect from
zqa-398.eng.vmware.com[10.137.245.143]
Oct 22 14:54:35 zqa-398 postfix/smtpd[2854]: warning: access table
regexp:/opt/zimbra/postfix/conf/tag_as_originating.re: with
smtpd_proxy_filter specified, action FILTER is unavailable
Oct 22 14:54:35 zqa-398 postfix/smtpd[2854]: NOQUEUE:
client=zqa-398.eng.vmware.com[10.137.245.143]
Oct 22 14:54:35 zqa-398 postfix/smtpd[2857]: connect from
localhost[127.0.0.1]
Oct 22 14:54:35 zqa-398 postfix/smtpd[2857]: warning: connect to Milter
service inet:localhost:8465: Connection refused
Oct 22 14:54:35 zqa-398 postfix/smtpd[2857]: NOQUEUE: milter-reject:
CONNECT from localhost[127.0.0.1]: 451 4.7.1 Service unavailable - try
again later; proto=SMTP
Oct 22 14:54:35 zqa-398 postfix/smtpd[2857]: NOQUEUE: milter-reject: EHLO
from localhost[127.0.0.1]: 451 4.7.1 Service unavailable - try again later;
proto=SMTP helo=<zqa-398.eng.vmware.com>
Oct 22 14:54:35 zqa-398 postfix/smtpd[2857]: NOQUEUE: milter-reject: MAIL
from localhost[127.0.0.1]: 451 4.7.1 Service unavailable - try again later;
from=<[hidden email]> proto=ESMTP
helo=<zqa-398.eng.vmware.com>
Oct 22 14:54:35 zqa-398 postfix/smtpd[2854]: warning: proxy
[127.0.0.1]:10029 rejected "MAIL FROM:<[hidden email]>": "451
4.7.1 Service unavailable - try again later"
Oct 22 14:54:35 zqa-398 postfix/smtpd[2854]: proxy-reject: END-OF-MESSAGE:
451 4.7.1 Service unavailable - try again later;
from=<[hidden email]> to=<[hidden email]>
proto=ESMTP helo=<zqa-398.eng.vmware.com>
Oct 22 14:54:35 zqa-398 postfix/smtpd[2857]: lost connection after MAIL
from localhost[127.0.0.1]


--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:

> --On Monday, October 22, 2012 5:09 PM -0400 Wietse Venema
> <[hidden email]> wrote:
>
> > 1) Use the before-queue filter for mail from outside:
> >
> > external clients -> smtpd -> Amavis ...
> >
> > 2) Use the after-queue filter for mail from inside:
> >
> >     internal clients -> smtpd -> cleanup -> queue -> smtp -> Amavis ...
> >
> > Wietse
>
> I already tried this, and it is not an acceptable solution, because postfix
> will not accept mail if OpenDKIM is not running.  I need Postfix to accept
> and queue the email in that scenario, rather than reject it.

RTFM http://www.postfix.org/postconf.5.html#milter_default_action

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 6:17 PM -0400 Wietse Venema
<[hidden email]> wrote:

> Quanah Gibson-Mount:
>> --On Monday, October 22, 2012 5:09 PM -0400 Wietse Venema
>> <[hidden email]> wrote:
>>
>> > 1) Use the before-queue filter for mail from outside:
>> >
>> > external clients -> smtpd -> Amavis ...
>> >
>> > 2) Use the after-queue filter for mail from inside:
>> >
>> >     internal clients -> smtpd -> cleanup -> queue -> smtp -> Amavis ...
>> >
>> > Wietse
>>
>> I already tried this, and it is not an acceptable solution, because
>> postfix  will not accept mail if OpenDKIM is not running.  I need
>> Postfix to accept  and queue the email in that scenario, rather than
>> reject it.
>
> RTFM http://www.postfix.org/postconf.5.html#milter_default_action

I have read that before.  None of the actions it allows are desirable.

Changing the action to quarantine requires manual intervention on the admin
side to ever get this to deliver.

"accept" is not acceptable, because it gets delivered instead of queued.

--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:

> --On Monday, October 22, 2012 6:17 PM -0400 Wietse Venema
> <[hidden email]> wrote:
>
> > Quanah Gibson-Mount:
> >> --On Monday, October 22, 2012 5:09 PM -0400 Wietse Venema
> >> <[hidden email]> wrote:
> >>
> >> > 1) Use the before-queue filter for mail from outside:
> >> >
> >> > external clients -> smtpd -> Amavis ...
> >> >
> >> > 2) Use the after-queue filter for mail from inside:
> >> >
> >> >     internal clients -> smtpd -> cleanup -> queue -> smtp -> Amavis ...
> >> >
> >> > Wietse
> >>
> >> I already tried this, and it is not an acceptable solution, because
> >> postfix  will not accept mail if OpenDKIM is not running.  I need
> >> Postfix to accept  and queue the email in that scenario, rather than
> >> reject it.
> >
> > RTFM http://www.postfix.org/postconf.5.html#milter_default_action
>
> I have read that before.  None of the actions it allows are desirable.
>
> Changing the action to quarantine requires manual intervention on the admin
> side to ever get this to deliver.

You had a problem with not being able to sign mail with a Milter
before it enters your content filter.

I kindly provided an example that allows you to do that. It even
works with the same content filter.

Now you reject the solution. Not because it would fail to sign mail
as promised. Not because it wouldn't work with the filter as promised.

There is, and there will not be, a queue between the Postfix SMTP
server protocol engine and the Postfix Milter client protocol engine,
where email messages wait until a broken Milter server comes back.

Not in Postfix, not in Sendmail, not in other MTAs.  The Milter
protocol is designed for before-queue agents, so that they can
inspect the SMTP command stream as it happens.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 7:09 PM -0400 Wietse Venema
<[hidden email]> wrote:

>
> You had a problem with not being able to sign mail with a Milter
> before it enters your content filter.
>
> I kindly provided an example that allows you to do that. It even
> works with the same content filter.
>
> Now you reject the solution. Not because it would fail to sign mail
> as promised. Not because it wouldn't work with the filter as promised.
>
> There is, and there will not be, a queue between the Postfix SMTP
> server protocol engine and the Postfix Milter client protocol engine,
> where email messages wait until a broken Milter server comes back.
>
> Not in Postfix, not in Sendmail, not in other MTAs.  The Milter
> protocol is designed for before-queue agents, so that they can
> inspect the SMTP command stream as it happens.

Ok.  So basically it is impossible to do what I want to do, thanks.  Let me
go back to the OpenDKIM folks then. ;)

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 4:23 PM -0700 Quanah Gibson-Mount
<[hidden email]> wrote:


>> There is, and there will not be, a queue between the Postfix SMTP
>> server protocol engine and the Postfix Milter client protocol engine,
>> where email messages wait until a broken Milter server comes back.

By the way, as long as Amavis isn't involved, I can get Postfix to queue
mail if the milter is down just fine, by setting things up as a content
filter that fronts the milter.

I.e., this configuration *does* queue email being sent to the milter:

smtp      inet  n       -       n       -       -       smtpd
        -o content_filter=scan:[127.0.0.1]:10029
scan      unix  -       -       n       -       10      smtp
        -o smtp_send_xforward_command=yes
        -o disable_mime_output_conversion=yes
        -o smtp_generic_maps=
[127.0.0.1]:10029 inet n - n - - smtpd
        -o smtpd_milters=inet:localhost:8465


With this setup, if I stop OpenDKIM, the mail queues until OpenDKIM is
restarted, even though OpenDKIM is being called as a milter.

--Quanah



--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:

> --On Monday, October 22, 2012 4:23 PM -0700 Quanah Gibson-Mount
> <[hidden email]> wrote:
>
>
> >> There is, and there will not be, a queue between the Postfix SMTP
> >> server protocol engine and the Postfix Milter client protocol engine,
> >> where email messages wait until a broken Milter server comes back.
>
> By the way, as long as Amavis isn't involved, I can get Postfix to queue
> mail if the milter is down just fine, by setting things up as a content
> filter that fronts the milter.

There is, however, no milter_default_action ON THE SMTP SERVER SIDE
that accepts mail and keeps it queued until the milter comes back.
That's what you objected to, and that's what will never exist.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 7:39 PM -0400 Wietse Venema
<[hidden email]> wrote:

> There is, however, no milter_default_action ON THE SMTP SERVER SIDE
> that accepts mail and keeps it queued until the milter comes back.
> That's what you objected to, and that's what will never exist.

All I want is to be able to send an email, have it processed and signed by
OpenDKIM, and then handed off to Amavis.

It seems to me if this can work without Amavis in the picture, it should be
possible to do it with Amavis in the picture too.

I understand the milter has to be fronted as a filter.  I understand I need
the ability to route mail differently through amavis depending on whether
or not it is originating or foreign email.  What I'm missing is how to get
all this to play together the way I want it to.

I.e., it is fine with me if "milter" is not how the SMTP server "sees"
things, just as it doesn't see it that way using the content filter.

So, is setting it up this way truly impossible as well, or is there some
way to stack filters (not milters), which was my original question.


--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Wietse Venema
Quanah Gibson-Mount:

> --On Monday, October 22, 2012 7:39 PM -0400 Wietse Venema
> <[hidden email]> wrote:
>
> > There is, however, no milter_default_action ON THE SMTP SERVER SIDE
> > that accepts mail and keeps it queued until the milter comes back.
> > That's what you objected to, and that's what will never exist.
>
> All I want is to be able to send an email, have it processed and signed by
> OpenDKIM, and then handed off to Amavis.
>
> It seems to me if this can work without Amavis in the picture, it should be
> possible to do it with Amavis in the picture too.
>
> I understand the milter has to be fronted as a filter.  I understand I need
> the ability to route mail differently through amavis depending on whether
> or not it is originating or foreign email.  What I'm missing is how to get
> all this to play together the way I want it to.
>
> I.e., it is fine with me if "milter" is not how the SMTP server "sees"
> things, just as it doesn't see it that way using the content filter.
>
> So, is setting it up this way truly impossible as well, or is there some
> way to stack filters (not milters), which was my original question.

Use a before-queue filter for mail from outside.

    Internet -> smtpd -> Amavis ...

Use a Postfix queue BEFORE and AFTER the signing Milter for mail
from inside.

    ... -> queue -> smtp -> smtpd -> cleanup -> queue -> smtp -> Amavis ...
                                |    |
                                signing
                                milter

The part with "smtp -> smtpd" is a "null filter" where the two
programs talk directly to each other.

Instead of the above you could simply use Amavis's DKIM support to
sign the messages.

No doubt there will be some other objection, and never a thankyou.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Any best practices for stacking filters?

Quanah Gibson-Mount-3
--On Monday, October 22, 2012 8:32 PM -0400 Wietse Venema
<[hidden email]> wrote:


> No doubt there will be some other objection, and never a thankyou.

Actually, I greatly appreciate your time and help with this.  So thank you,
very much for your patience as well.

As for using Amavis to sign via DKIM, that is waiting on Amavisd to be
scalable, a feature request I've had open with the Amavis author for
several years, and which he is working on.  If that gets implemented then
yes, I can switch to Amavis entirely, and my life will be much simpler.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Loading...