Greylisting: soft fail if grey listing server down

[Marc G. Fournier-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I have grey listing setup using:

smtpd_recipient_restrictions =
        reject_unauth_pipelining
        reject_non_fqdn_recipient
        permit_mynetworks
        permit_sasl_authenticated
        reject_unauth_destination
        reject_unknown_sender_domain
        reject_unlisted_recipient
        check_policy_service inet:glh.hub.org:2501
        permit

If, for some reason, the greylisting server is down (ie. maintenance), my logs
start to show:

warning: problem talking to server glh.hub.org:2501: Operation timed out

What happens then?  Does the remote end see that as a hard failure, or a soft
one?  Is there a setting (like unknown_local_recipient_reject_code = 450) that
I can set to make it soft, so that email isn't lost at times like this?

Thx

- --
Marc G. Fournier        Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . [hidden email]                              MSN . [hidden email]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.8 (FreeBSD)

iEYEARECAAYFAkglHXoACgkQ4QvfyHIvDvMR2wCeObWu6rozzXl8sO5THB9CFyX3
IE8An2afD85NnTgtHWXRIFSZbOGSo4G5
=uYhL
-----END PGP SIGNATURE-----

[Noel Jones-2]

Marc G. Fournier wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> I have grey listing setup using:
>
> smtpd_recipient_restrictions =
>         reject_unauth_pipelining
>         reject_non_fqdn_recipient
>         permit_mynetworks
>         permit_sasl_authenticated
>         reject_unauth_destination
>         reject_unknown_sender_domain
>         reject_unlisted_recipient
>         check_policy_service inet:glh.hub.org:2501
>         permit
>
> If, for some reason, the greylisting server is down (ie. maintenance), my logs
> start to show:
>
> warning: problem talking to server glh.hub.org:2501: Operation timed out
>
> What happens then?  Does the remote end see that as a hard failure, or a soft
> one?  Is there a setting (like unknown_local_recipient_reject_code = 450) that
> I can set to make it soft, so that email isn't lost at times like this?
>

Surely you can find the answer a few lines later on in your
log...

(hint - postfix always does the safe thing by default)

There is no setting to adjust postfix behavior when a policy
service is unavailable.

--
Noel Jones

[Wietse Venema]

Marc G. Fournier:
-- Start of PGP signed section.

>
> I have grey listing setup using:
>
> smtpd_recipient_restrictions =
>         reject_unauth_pipelining
>         reject_non_fqdn_recipient
>         permit_mynetworks
>         permit_sasl_authenticated
>         reject_unauth_destination
>         reject_unknown_sender_domain
>         reject_unlisted_recipient
>         check_policy_service inet:glh.hub.org:2501
>         permit
>
> If, for some reason, the greylisting server is down (ie. maintenance), my logs
> start to show:
>
> warning: problem talking to server glh.hub.org:2501: Operation timed out
>
> What happens then?  Does the remote end see that as a hard failure, or a soft
> one?  Is there a setting (like unknown_local_recipient_reject_code = 450) that
> I can set to make it soft, so that email isn't lost at times like this?

Oh ye of small faith. Postfix always replies with 4xx if it doesn't
get a definite reply.

        Wietse

[Geert Hendrickx]

On Sat, May 10, 2008 at 09:18:47AM -0400, Wietse Venema wrote:
> Marc G. Fournier:
> > What happens then?  Does the remote end see that as a hard failure, or a
> > soft one?  Is there a setting (like unknown_local_recipient_reject_code =
> > 450) that I can set to make it soft, so that email isn't lost at times
> > like this?
>
> Oh ye of small faith. Postfix always replies with 4xx if it doesn't get a
> definite reply.

For milters there is milter_default_action = accept.  Can this be implemented
for policy daemons as well?

        Geert

[Wietse Venema]

Geert Hendrickx:

> On Sat, May 10, 2008 at 09:18:47AM -0400, Wietse Venema wrote:
> > Marc G. Fournier:
> > > What happens then?  Does the remote end see that as a hard failure, or a
> > > soft one?  Is there a setting (like unknown_local_recipient_reject_code =
> > > 450) that I can set to make it soft, so that email isn't lost at times
> > > like this?
> >
> > Oh ye of small faith. Postfix always replies with 4xx if it doesn't get a
> > definite reply.
>
> For milters there is milter_default_action = accept.

This is because Sendmail provides it, not because it is a good idea.

>  Can this be implemented for policy daemons as well?

I don't like dropping the shields down because some service dies.

It causes people to lose confidence in Postfix, because it results in
unpredictable behavior. There is enough bad software on the Internet.

        Wietse

[Aaron Wolfe]

On Sat, May 10, 2008 at 8:09 PM, Wietse Venema <[hidden email]> wrote:

Geert Hendrickx:

> On Sat, May 10, 2008 at 09:18:47AM -0400, Wietse Venema wrote:
> > Marc G. Fournier:
> > > What happens then?  Does the remote end see that as a hard failure, or a
> > > soft one?  Is there a setting (like unknown_local_recipient_reject_code =
> > > 450) that I can set to make it soft, so that email isn't lost at times
> > > like this?
> >
> > Oh ye of small faith. Postfix always replies with 4xx if it doesn't get a
> > definite reply.
>
> For milters there is milter_default_action = accept.

This is because Sendmail provides it, not because it is a good idea.

>  Can this be implemented for policy daemons as well?

I don't like dropping the shields down because some service dies.

It causes people to lose confidence in Postfix, because it results in
unpredictable behavior. There is enough bad software on the Internet.

Would it be possible (and make sense) to have an option to specify somehow that a particular policy filter is non essential and it's failure should be considered equivalent to  "DUNNO" or similar? 

I have had more than one situation where a daemon failed and the resulting mail delays were unfortunate and really unnecessary as the mail system (besides the policy filter) was working fine.  In our case, dropping this filter from our UCE fighting checks would not have resulted in "shields down" or more spam, just extra work for the more resource intensive checks later on in the chain. 

-Aaron

[Wietse Venema]

Aaron Wolfe:

> > > For milters there is milter_default_action = accept.
> >
> > This is because Sendmail provides it, not because it is a good idea.
> >
> > >  Can this be implemented for policy daemons as well?
> >
> > I don't like dropping the shields down because some service dies.
> >
> > It causes people to lose confidence in Postfix, because it results in
> > unpredictable behavior. There is enough bad software on the Internet.
>
> Would it be possible (and make sense) to have an option to specify somehow
> that a particular policy filter is non essential and it's failure should be
> considered equivalent to  "DUNNO" or similar?

Anyone who provides a critical service should periodically run a
tool that performs sanity checks, and that gives an errant server
a whack over its head when it locks up.

I would appreciate it if people would stopp asking for "drop the
shields down" features, because they will be misused. Instead of
adding crap to Postfix, invest the effort into making services
more reliable.

        Wietse

[Aaron Wolfe]

On Sat, May 10, 2008 at 8:45 PM, Wietse Venema <[hidden email]> wrote:

Aaron Wolfe:

> > > For milters there is milter_default_action = accept.
> >
> > This is because Sendmail provides it, not because it is a good idea.
> >
> > >  Can this be implemented for policy daemons as well?
> >
> > I don't like dropping the shields down because some service dies.
> >
> > It causes people to lose confidence in Postfix, because it results in
> > unpredictable behavior. There is enough bad software on the Internet.
>
> Would it be possible (and make sense) to have an option to specify somehow
> that a particular policy filter is non essential and it's failure should be
> considered equivalent to  "DUNNO" or similar?

Anyone who provides a critical service should periodically run a
tool that performs sanity checks, and that gives an errant server
a whack over its head when it locks up.

I would appreciate it if people would stopp asking for "drop the
shields down" features, because they will be misused. Instead of
adding crap to Postfix, invest the effort into making services
more reliable.

I would appreciate it if you could explain what "drop the shields down" means, and how making postfix more tolerant of policy filter failure would lead to misuse?
Maybe folks would be less likely to ask for this feature if we understood better why it is bad for us?

-Aaron

[Wietse Venema]

Aaron Wolfe:

> > Anyone who provides a critical service should periodically run a
> > tool that performs sanity checks, and that gives an errant server
> > a whack over its head when it locks up.
> >
> > I would appreciate it if people would stopp asking for "drop the
> > shields down" features, because they will be misused. Instead of
> > adding crap to Postfix, invest the effort into making services
> > more reliable.
> >
>
> I would appreciate it if you could explain what "drop the shields
> down" means, and how making postfix more tolerant of policy filter
> failure would lead to misuse? Maybe folks would be less likely to
> ask for this feature if we understood better why it is bad for
> us?

Anyone who provides a critical service should periodically run a
tool that performs sanity checks, and that gives an errant server
a whack over its head when it locks up.

I would appreciate it if people would stop asking for features
to ignore errors and to let mail slip through.

        Wietse

[Terry Allen]

>
>Anyone who provides a critical service should periodically run a
>tool that performs sanity checks, and that gives an errant server
>a whack over its head when it locks up.
>
>I would appreciate it if people would stopp asking for "drop the
>shields down" features, because they will be misused. Instead of
>adding crap to Postfix, invest the effort into making services
>more reliable.
>
> Wietse

Hi again,
        I would have to agree with Wietse - in the years my systems
have been running Postfix, I have never experienced a mail system
failure that has been the fault of Postfix itself - it has always
either been my screw-ups, file corruption, or a failure of some
additional piece of software I have running alongside. If other
software manufacturers spent the time to make their software to the
same standards as Postfix, these issues would not crop up.
--

        Bye for now, Terry Allen
        ___________________________________________________________________
hEARd

Postal Address:
        hEARd, 26B Glenning Rd, Glenning Valley, NSW 2261, Australia
Internet -
        WWW: http://heard.com.au http://itavservices.com
EMAIL: [hidden email]
Phone: Australia - 02 4388 1400 / International - + 61 2 43881400
Mobile: Australia - 04 28881400 / International - 61 4 28881400
-----------------------------------------------
Non profit promotion for new music - since 1994
-----------------------------------------------

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Terry Allen wrote:
| Hi again,
|     I would have to agree with Wietse - in the years my systems have

Me too. Wietse is right. If some component dies or fails for any reason, I want to know
immediatelly. Via sanity checks, preemptive procedures and/or users complaining. Whatever happens first.

If the system's components are healthy, that'd never happen anyway :)

- --
Arturo "Buanzo" Busleiman
Reliable inter-continental Mail Relay Service - Ask me!
Independent Security Consultant - SANS - OISSG
http://www.buanzo.com.ar/pro/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIJlSBAlpOsGhXcE0RCmINAJ4/w6KP4+U4jzhyCH2INVXPG61DQQCePrqy
93yDlpyl6jCf3nbHdJprUBo=
=zIwZ
-----END PGP SIGNATURE-----

[Geert Hendrickx]

On Sat, May 10, 2008 at 11:05:53PM -0300, Arturo 'Buanzo' Busleiman wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Terry Allen wrote:
> | Hi again,
> |     I would have to agree with Wietse - in the years my systems have
>
> Me too. Wietse is right. If some component dies or fails for any reason, I
> want to know
> immediatelly. Via sanity checks, preemptive procedures and/or users
> complaining. Whatever happens first.
>
> If the system's components are healthy, that'd never happen anyway :)

Why do people always insist to link "unavailable" to "crappy software" ?

This could just as well be about a network problem, or a greylisting daemon
being upgraded, or whatever.  For example in the case of greylisting via a
policy daemon, a failure is anything but critical and e-mail could just as
well be accepted instead of deferred.  Any greylisting implementation has a
similar "bypass" option for when its database is unavailable.

        Geert

[Geert Hendrickx]

On Sat, May 10, 2008 at 08:09:35PM -0400, Wietse Venema wrote:
> I don't like dropping the shields down because some service dies.

That depends entirely on the purpose of the greylisting daemon or milter
you're talking about.  In case of a virus filter, the default in case of a
failure should never be accept, in case of a spam filter, it could be either
accept or defer depending on the site's policy, in case of greylisting it
could just as well be accept, etc.  

> It causes people to lose confidence in Postfix, because it results in
> unpredictable behavior. There is enough bad software on the Internet.

Unpredictable behaviour?  Just log a line "accepted because FOO was
unavailable" and anyone can look up the reason of a specific accept.

        Geert

[Wietse Venema]

Geert Hendrickx:

> On Sat, May 10, 2008 at 11:05:53PM -0300, Arturo 'Buanzo' Busleiman wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > Terry Allen wrote:
> > | Hi again,
> > |     I would have to agree with Wietse - in the years my systems have
> >
> > Me too. Wietse is right. If some component dies or fails for any reason, I
> > want to know
> > immediatelly. Via sanity checks, preemptive procedures and/or users
> > complaining. Whatever happens first.
> >
> > If the system's components are healthy, that'd never happen anyway :)
>
>
> Why do people always insist to link "unavailable" to "crappy software" ?

Oh, we could have said "because of poor system administration practices"
but we are too friendly here.

> This could just as well be about a network problem, or a greylisting daemon
> being upgraded, or whatever.  For example in the case of greylisting via a
> policy daemon, a failure is anything but critical and e-mail could just as
> well be accepted instead of deferred.  Any greylisting implementation has a
> similar "bypass" option for when its database is unavailable.

Sorry, knocking out a server on a live machine is bad practice.
This could easily be avoided by starting the new greylist server
on a different port, updating the server port in main.cf, and
letting Postfix transition to the new server without disruption.
Even if this required editing master.cf, "postfix reload" still
would not disrupt the mail flow.

With Postfix's modular design with mostly short-lived processes,
such on-the-fly updates without disruption should be a natural part
of sysadmin processes.

        Wietse

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Wietse Venema wrote:
| With Postfix's modular design with mostly short-lived processes,
| such on-the-fly updates without disruption should be a natural part
| of sysadmin processes.

And that practice is part of the "healthy" and "preemptive" (probably proactive, not preemptive,
excuse my english) thing I mentioned.

- --
Arturo "Buanzo" Busleiman
Reliable inter-continental Mail Relay Service - Ask me!
Independent Security Consultant - SANS - OISSG
http://www.buanzo.com.ar/pro/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIJvgbAlpOsGhXcE0RCvEvAJ9minv48cuuyJiRDYDq1zaqxIWVfwCdHo6I
LOqtKRpOmWoozW4ol4WMXpY=
=gGnj
-----END PGP SIGNATURE-----

[mouss-2]

Geert Hendrickx wrote:
>
> Why do people always insist to link "unavailable" to "crappy software" ?
>  

because it is? now, stop talking theory. give us examples of real
failures of good policy servers.
> This could just as well be about a network problem,

How many times do you have a network problem? if it happens often, then
fix it.
>  or a greylisting daemon
> being upgraded, or whatever.

how many times do you upgrade your daemons?

>  For example in the case of greylisting via a
> policy daemon, a failure is anything but critical and e-mail could just as
> well be accepted instead of deferred.

you are wrong. this is not what I want on my system. I believ that only
suspicious transactions should be greylstied. and in such a case, a
defer is the right fallback.

if you want to implement a default return code, then it is trivial to
write a proxy that does so.

>  Any greylisting implementation has a
> similar "bypass" option for when its database is unavailable.
>  

This is irrelevant.

[Cami-2]

Wietse Venema wrote:

> Geert Hendrickx:
>> For milters there is milter_default_action = accept.
>
> This is because Sendmail provides it, not because it is a good idea.
>
>>  Can this be implemented for policy daemons as well?
>
> I don't like dropping the shields down because some service dies.
>
> It causes people to lose confidence in Postfix, because it results in
> unpredictable behavior. There is enough bad software on the Internet.

While i agree with you, isn't it fairly inconsistent to allow
milters to have that functionality but not policy daemons?

Cami

[Wietse Venema]

Cami Sardinha:

> Wietse Venema wrote:
> > Geert Hendrickx:
> >> For milters there is milter_default_action = accept.
> >
> > This is because Sendmail provides it, not because it is a good idea.
> >
> >>  Can this be implemented for policy daemons as well?
> >
> > I don't like dropping the shields down because some service dies.
> >
> > It causes people to lose confidence in Postfix, because it results in
> > unpredictable behavior. There is enough bad software on the Internet.
>
> While i agree with you, isn't it fairly inconsistent to allow
> milters to have that functionality but not policy daemons?

This is because Sendmail provides it, not because it is a good idea.

        Wietse

[Geert Hendrickx]

On Sun, May 11, 2008 at 09:29:34AM -0400, Wietse Venema wrote:
> Sorry, knocking out a server on a live machine is bad practice.  This could
> easily be avoided by starting the new greylist server on a different port,
> updating the server port in main.cf, and letting Postfix transition to the
> new server without disruption.  Even if this required editing master.cf,
> "postfix reload" still would not disrupt the mail flow.


That's how we just upgraded our 8 MX servers from policyd v1 to v2 last week,
precisely to avoid impact on the mail flow during the process.  Yet I think
it would be beneficial to be able to mark certain policy daemons as "optional"
and allow for outages.  But oh well. :-)

        Geert

[Aaron Wolfe]

On Sun, May 11, 2008 at 4:29 PM, mouss <[hidden email]> wrote:

Geert Hendrickx wrote:

Why do people always insist to link "unavailable" to "crappy software" ?
 

because it is? now, stop talking theory. give us examples of real failures of good policy servers.

I can give an example.  We wanted to evaluate greylisting here, and after looking at the available options decided sqlgrey would fit our needs.  We installed it on a non production box, learned how it worked,  did some testing, etc.  Things looked good, so we added it to our live environment.  Things worked well for several days, might have been a week. 

Then one day, *very* early in the morning, I woke to my cell going crazy with emails from nagios and voicemails from frantic sales people.  No mail was coming in, oh the humanity, horror, horror, etc etc.

sqlgrey had a bug (since fixed) where it would occasionally die during log rotation.  Restart and all is well.  It managed to die on both of my inbound servers at the same time: jackpot winner.

Later that day I wrote a script to check and restart it as neccesary.  It still bothered me that I was deferring a handful of messages every few days while the script did it's business, but soon a patch was available (although that script still runs, just in case). 

I did have monitoring in place when the failure occurred, but since the bug was unknown to me, I had nothing automatic to restart the service.  If I could have just indicated to postfix that I really don't care if that policy server works or not somehow, that would have been a much better morning for me and lots of the people I support.  It wasn't a huge thing, obviously everything eventually came in, but it did kind of suck for a while.  Finding some new errors in the logs and sorting out the failure on my own time would have been vastly preferable to dealing with angry salesholes in the early morning.

Maybe I am a bad administrator?  I am completely open to that idea, but if you have criticism please express it in a way that will help me learn better ways to do this. 

The idea of writing a proxy/wrapper to ensure a DUNNO result had occurred to me before, but that feels like a hack.  Maybe the only way though.  Would this be a "Bad Idea" to do?

What is the best way to run a service that you don't care to depend upon?  If everything is always critical, how do you test new things?  I would like to be as 'safe' as possible.  For some things, I suppose you can just test it on only one of your servers and be fairly safe, but with spam fighting stuff like greylisting, that doesn't really work.

-Aaron