Tailored filter

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Seb
Reply | Threaded
Open this post in threaded view
|

Tailored filter

Seb

Hello,


I run a small publishing company and for the sake of easing communication
between authors (who work in teams) I have provided each of them with a
local alias. Typically, mail sent to <firstname>.<lastname>@<mydomain> is
redirected to <firstname>.<lastname>@gmail.com, the usual email address of
the author.

I've been using this for 15+ years and it's been great. Unfortunately,
I'm losing the war against spam. In spite of careful configuration of
Postfix, the use of Postgrey and hand-drawn blacklists, too much spam
passes through. What my server regards as legitimate email (but is
sometimes spam) gets resent to sites such as GMail which, in turn, tend
to flag all email from my domain as spam, even legitimate emails. And this
is starting to jeopardize my communication with the rest of the world.

In this very special setting, I have a solution that preserves
functionality: all email adressed to me (through various aliases) should
come to my box with only the usual filters enabled, but an email for an
author (@<mydomain>) should be allowed to pass through only if it is sent
from the usual email address of another author. I have a complete list of
all these valid sender email addresses (most authors use several). For
instance, if authors Alice and Bob have email addresses [hidden email]
and [hidden email], Alice can write to bob@<mydomain> from her GMail
account. But emails sent to bob@<mydomain> from an address unknown in my
database should be silently dropped.

I have turned to Postfix' documentation to find the best way to do this. I
thought that among the seemingly endless options for main.cf one would
match, but I could not find it. Perhaps I didn't understand enough to
identify the right one. The difficulty seems to be that the rule for
authors must not apply to me, because I need to be contacted by people
whose email address I don't already know.

I also looked Postfix' documentation about milters. These really look like
what I need to put in place, if a simpler solution cannot be found. I can
write a nice program in Perl. But hooking a Perl script into Postfix as a
milter seems quite specific and the documentation is rather evasive on
this matter.

Which way of configuring Postfix would you recommend for this job?


Thanks for you help !

Sébastien.
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Dominic Raferd
On 19 October 2017 at 10:48, Seb <[hidden email]> wrote:

​... 
Typically, mail sent to <firstname>.<lastname>@<mydomain> is redirected to <firstname>.<lastname>@gmail.com, the usual email address of the author.

I've been using this for 15+ years and it's been great. Unfortunately, I'm losing the war against spam. In spite of careful configuration of Postfix, the use of Postgrey and hand-drawn blacklists, too much spam passes through. What my server regards as legitimate email (but is sometimes spam) gets resent to sites such as GMail which, in turn, tend to flag all email from my domain as spam, even legitimate emails. And this is starting to jeopardize my communication with the rest of the world.
​..

I relay successfully to Gmail with some small domains. My setup in outline is:
  • emails from unauthenticated non-local senders are rejected unless to a known approved recipient
  • many external rbls (requires local DNS server [bind] with forwarding - but with forwarding disabled for rbl zones)
  • python-policyd-spf (adds header for review by opendmarc)
  • opendkim (tests 'incoming' emails and adds header for review by opendmarc, signs 'outgoing' emails)
  • opendmarc - with enforcement for 'incoming' emails
  • reject_unknown_reverse_client_hostname
  • amavisd-new - uses Spamassassin (with use_bayes 0), ClamAV (with Sanesecurity), razor, pyzor
  • bespoke filtering by ip, sender name, client/host name, helo name, headers
  • fail2ban (tweaked 'dovecot' and bespoke 'postfix-failedauth' jails)
  • relay-enforcer (bespoke, requires fail2ban and short-term local backup of emails)
  • permanent ip banning via ufw for repeat fail2ban or unresolved hostname offenders (bespoke)
​Some of this is bespoke but much of it is easily replicated. I recently gave up using postgrey because the delayed delivery drove users mad.

I realise this doesn't answer your question, but it may suggest a different way forward.
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Noel Jones-2
In reply to this post by Seb
On 10/19/2017 4:48 AM, Seb wrote:

>
> Hello,
>
>
> I run a small publishing company and for the sake of easing
> communication between authors (who work in teams) I have provided
> each of them with a local alias. Typically, mail sent to
> <firstname>.<lastname>@<mydomain> is redirected to
> <firstname>.<lastname>@gmail.com, the usual email address of the
> author.
>


One of the casualties in the war on spam is mail forwarders.

The built-in postfix way to control the sender/recipient pairs is
restriction classes. This works well for a small number of
combinations, but quickly gets unmanageable.
http://www.postfix.org/RESTRICTION_CLASS_README.html

You can use a postfix policy service such as postfwd to create a
list of allowed senders for some particular recipient.  This isn't
difficult, but will require manual intervention anytime a change is
needed.  Postfwd may be kinda old, but is still widely used.
http://postfwd.org/

Another alternative is to use one of the existing perl (or java)
milters such as milter-regex and add your sender/recipient checks
there.  Some ideas here, but just about any milter should work:
http://www.postfix.org/addon.html

Or just blow the whole thing up.  Either host the mail locally and
stop forwarding, or migrate your domain to gsuite so it's all inside
their system.



  -- Noel Jones
Seb
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Seb

Hello,


Thanks a lot Noel for this bird's-eye view of possible solutions. The most
promising tool for my setting seems to be Postfwd, which I'll now explore.


Sébastien.


> One of the casualties in the war on spam is mail forwarders.
>
> The built-in postfix way to control the sender/recipient pairs is
> restriction classes. This works well for a small number of
> combinations, but quickly gets unmanageable.
> http://www.postfix.org/RESTRICTION_CLASS_README.html
>
> You can use a postfix policy service such as postfwd to create a
> list of allowed senders for some particular recipient.  This isn't
> difficult, but will require manual intervention anytime a change is
> needed.  Postfwd may be kinda old, but is still widely used.
> http://postfwd.org/
>
> Another alternative is to use one of the existing perl (or java)
> milters such as milter-regex and add your sender/recipient checks
> there.  Some ideas here, but just about any milter should work:
> http://www.postfix.org/addon.html
>
> Or just blow the whole thing up.  Either host the mail locally and
> stop forwarding, or migrate your domain to gsuite so it's all inside
> their system.
>
>
>
>  -- Noel Jones
>
Seb
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Seb
In reply to this post by Noel Jones-2

Hello,


>> I run a small publishing company and for the sake of easing
>> communication between authors (who work in teams) I have provided each
>> of them with a local alias. Typically, mail sent to
>> <firstname>.<lastname>@<mydomain> is redirected to
>> <firstname>.<lastname>@gmail.com, the usual email address of the
>> author.
> You can use a postfix policy service such as postfwd to create a list of
> allowed senders for some particular recipient.  This isn't difficult,
> but will require manual intervention anytime a change is needed.
> Postfwd may be kinda old, but is still widely used. http://postfwd.org/
(Reminder: my aim is to allow emails for our site's users only if the
emails come from a certain dynamic list of addresses. It's a tool to
Filter Unauthorized Communications with Keyholes, or in short, let's call
it f.u.c.k.)

I looked at the three solutions hinted at by Noel Jones. Postfwd was
closest to my needs but it seemed more straightforward to use Postfix'
SMTP Access Policy Delegation, which is mentioned in Postfwd's own
documentation. To this end I closely followed the instructions provided by
the documentation:
  http://www.postfix.org/SMTPD_POLICY_README.html#client_config

In master.cf I added the lines
  # service type  private unpriv  chroot  wakeup  maxproc command + args
  policy    unix  -       n       n       -       0       spawn
    user=nobody argv=/home/seb/sandra/bin/fuck
(I tried writing this on 1 line or on 2 lines.)

In main.cf I extended smtpd_recipient_restrictions to:
  smtpd_recipient_restrictions =
     reject_invalid_helo_hostname,
     [...]
     check_policy_service inet:127.0.0.1:10023,
     check_policy_service unix:/home/seb/sandra/bin/fuck,
     permit
and, as the doc instructed, I also added:
  policy_time_limit = 3600

As for the Perl script that would decide whether an email should go
through or not, for testing purposes I simply wrote:
  #!/usr/bin/perl
  print "action=dunno\n\n";

I then did a "chmod a+x" on /home/seb/sandra/bin/fuck and a "postfix
reload"; my postfix version is 2.11 (Debian 8).

This setup is as close to the documentation as I can make it. Yet I have
missed something because /var/log/mail.log says:
Nov  7 13:51:17 ns3358511 postfix/smtpd[14177]: warning: connect to
  /home/seb/sandra/bin/fuck: No such file or directory
Nov  7 13:51:17 ns3358511 postfix/smtpd[14177]: warning: problem talking
  to server /home/seb/sandra/bin/fuck: No such file or directory

although the file really exists:
  ~>ls -l /home/seb/sandra/bin/fuck
-rwxr-xr-x 1 seb seb 2880 Nov  7 16:31 /home/seb/sandra/bin/fuck

The problem could very well be something simple or so self-evident that it
was not deemed necessary to write it in the documentation.

I toyed with this as much as I dared on a live system, and still have no
clue what the message in mail.log really means.

Any help figuring this out would be very much appreciated, thank you!


Kind regards,
Sébastien.
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Noel Jones-2
On 11/7/2017 9:40 AM, Seb wrote:

>
> Hello,
>
>
>>> I run a small publishing company and for the sake of easing
>>> communication between authors (who work in teams) I have provided
>>> each of them with a local alias. Typically, mail sent to
>>> <firstname>.<lastname>@<mydomain> is redirected to
>>> <firstname>.<lastname>@gmail.com, the usual email address of the
>>> author.
>> You can use a postfix policy service such as postfwd to create a
>> list of allowed senders for some particular recipient.  This isn't
>> difficult, but will require manual intervention anytime a change
>> is needed. Postfwd may be kinda old, but is still widely used.
>> http://postfwd.org/
>
> (Reminder: my aim is to allow emails for our site's users only if
> the emails come from a certain dynamic list of addresses. It's a
> tool to Filter Unauthorized Communications with Keyholes, or in
> short, let's call it f.u.c.k.)
>
> I looked at the three solutions hinted at by Noel Jones. Postfwd was
> closest to my needs but it seemed more straightforward to use
> Postfix' SMTP Access Policy Delegation, which is mentioned in
> Postfwd's own documentation. To this end I closely followed the
> instructions provided by the documentation:
>     http://www.postfix.org/SMTPD_POLICY_README.html#client_config
>
> In master.cf I added the lines
>     # service type  private unpriv  chroot  wakeup  maxproc command
> + args
>     policy    unix  -       n       n       -       0       spawn
>        user=nobody argv=/home/seb/sandra/bin/fuck
> (I tried writing this on 1 line or on 2 lines.)
>
> In main.cf I extended smtpd_recipient_restrictions to:
>     smtpd_recipient_restrictions =
>         reject_invalid_helo_hostname,
>         [...]
>         check_policy_service inet:127.0.0.1:10023,
>         check_policy_service unix:/home/seb/sandra/bin/fuck,
>         permit
> and, as the doc instructed, I also added:
>     policy_time_limit = 3600
>
> As for the Perl script that would decide whether an email should go
> through or not, for testing purposes I simply wrote:
>     #!/usr/bin/perl
>     print "action=dunno\n\n";
>
> I then did a "chmod a+x" on /home/seb/sandra/bin/fuck and a "postfix
> reload"; my postfix version is 2.11 (Debian 8).
>
> This setup is as close to the documentation as I can make it. Yet I
> have missed something because /var/log/mail.log says:
> Nov  7 13:51:17 ns3358511 postfix/smtpd[14177]: warning: connect to
>     /home/seb/sandra/bin/fuck: No such file or directory
> Nov  7 13:51:17 ns3358511 postfix/smtpd[14177]: warning: problem
> talking
>     to server /home/seb/sandra/bin/fuck: No such file or directory
>
> although the file really exists:
>  ~>ls -l /home/seb/sandra/bin/fuck
> -rwxr-xr-x 1 seb seb 2880 Nov  7 16:31 /home/seb/sandra/bin/fuck
>
> The problem could very well be something simple or so self-evident
> that it was not deemed necessary to write it in the documentation.
>
> I toyed with this as much as I dared on a live system, and still
> have no clue what the message in mail.log really means.
>
> Any help figuring this out would be very much appreciated, thank you!
>
>
> Kind regards,
> Sébastien.


My first guess is
http://www.postfix.org/DEBUG_README.html#no_chroot



  -- Noel Jones
Seb
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Seb

Hello,


> My first guess is
> http://www.postfix.org/DEBUG_README.html#no_chroot

So I went through /etc/postfix/master.cf and changed each line to make
sure that nothing was left in a chroot:

#egrep '^[a-z]' master.cf
smtp      inet  n       -       n       -       -       smtpd
pickup    fifo  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
maildrop  unix  -       n       n       -       -       pipe
policy    unix  -       n       n       -       0       spawn
uucp      unix  -       n       n       -       -       pipe
ifmail    unix  -       n       n       -       -       pipe
bsmtp     unix  -       n       n       -       -       pipe
scalemail-backend unix  -       n       n       -       2       pipe
mailman   unix  -       n       n       -       -       pipe
smtp-amavis unix    -   -       n       -       2     smtp

and alas, the message remains in mail.log . It was a good call though.
Any other idea?


Sébastien.


Noel Jones (Tue, 7 Nov 2017):

> On 11/7/2017 9:40 AM, Seb wrote:
>>
>> Hello,
>>
>>
>>>> I run a small publishing company and for the sake of easing
>>>> communication between authors (who work in teams) I have provided
>>>> each of them with a local alias. Typically, mail sent to
>>>> <firstname>.<lastname>@<mydomain> is redirected to
>>>> <firstname>.<lastname>@gmail.com, the usual email address of the
>>>> author.
>>> You can use a postfix policy service such as postfwd to create a
>>> list of allowed senders for some particular recipient.  This isn't
>>> difficult, but will require manual intervention anytime a change
>>> is needed. Postfwd may be kinda old, but is still widely used.
>>> http://postfwd.org/
>>
>> (Reminder: my aim is to allow emails for our site's users only if
>> the emails come from a certain dynamic list of addresses. It's a
>> tool to Filter Unauthorized Communications with Keyholes, or in
>> short, let's call it f.u.c.k.)
>>
>> I looked at the three solutions hinted at by Noel Jones. Postfwd was
>> closest to my needs but it seemed more straightforward to use
>> Postfix' SMTP Access Policy Delegation, which is mentioned in
>> Postfwd's own documentation. To this end I closely followed the
>> instructions provided by the documentation:
>>     http://www.postfix.org/SMTPD_POLICY_README.html#client_config
>>
>> In master.cf I added the lines
>>     # service type  private unpriv  chroot  wakeup  maxproc command
>> + args
>>     policy    unix  -       n       n       -       0       spawn
>>        user=nobody argv=/home/seb/sandra/bin/fuck
>> (I tried writing this on 1 line or on 2 lines.)
>>
>> In main.cf I extended smtpd_recipient_restrictions to:
>>     smtpd_recipient_restrictions =
>>         reject_invalid_helo_hostname,
>>         [...]
>>         check_policy_service inet:127.0.0.1:10023,
>>         check_policy_service unix:/home/seb/sandra/bin/fuck,
>>         permit
>> and, as the doc instructed, I also added:
>>     policy_time_limit = 3600
>>
>> As for the Perl script that would decide whether an email should go
>> through or not, for testing purposes I simply wrote:
>>     #!/usr/bin/perl
>>     print "action=dunno\n\n";
>>
>> I then did a "chmod a+x" on /home/seb/sandra/bin/fuck and a "postfix
>> reload"; my postfix version is 2.11 (Debian 8).
>>
>> This setup is as close to the documentation as I can make it. Yet I
>> have missed something because /var/log/mail.log says:
>> Nov  7 13:51:17 ns3358511 postfix/smtpd[14177]: warning: connect to
>>     /home/seb/sandra/bin/fuck: No such file or directory
>> Nov  7 13:51:17 ns3358511 postfix/smtpd[14177]: warning: problem
>> talking
>>     to server /home/seb/sandra/bin/fuck: No such file or directory
>>
>> although the file really exists:
>>  ~>ls -l /home/seb/sandra/bin/fuck
>> -rwxr-xr-x 1 seb seb 2880 Nov  7 16:31 /home/seb/sandra/bin/fuck
>>
>> The problem could very well be something simple or so self-evident
>> that it was not deemed necessary to write it in the documentation.
>>
>> I toyed with this as much as I dared on a live system, and still
>> have no clue what the message in mail.log really means.
>>
>> Any help figuring this out would be very much appreciated, thank you!
>>
>>
>> Kind regards,
>> Sébastien.
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Noel Jones-2
On 11/7/2017 10:18 AM, Seb wrote:
>
> Hello,
>
>
>> My first guess is
>> http://www.postfix.org/DEBUG_README.html#no_chroot
>

Sorry, wrong guess.  Your test program is not a policy server.  You
need something that listens on a unix or tcp socket. An incomplete
list of available policy servers can be found here:
http://www.postfix.org/addon.html#policy



  -- Noel Jones
Seb
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Seb

Hello,


>>> My first guess is
>>> http://www.postfix.org/DEBUG_README.html#no_chroot
> Sorry, wrong guess.  Your test program is not a policy server.  You need
> something that listens on a unix or tcp socket. An incomplete list of
> available policy servers can be found here:
> http://www.postfix.org/addon.html#policy

I see. So the example given in the online documentation can't work, and
the more complete, apparently self-sufficient example greylist.pl given in
postfix' source code can't work either. That's disturbing :-) But okay, I
had a look at Postgrey's source code and it does indeed communicate
through a socket, a feature that is definitely missing from what I tried.
I'll give it another go in a few weeks.

Thanks!


Sébastien.
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Noel Jones-2
On 11/7/2017 10:55 AM, Seb wrote:

>
> Hello,
>
>
>>>> My first guess is
>>>> http://www.postfix.org/DEBUG_README.html#no_chroot
>> Sorry, wrong guess.  Your test program is not a policy server. 
>> You need something that listens on a unix or tcp socket. An
>> incomplete list of available policy servers can be found here:
>> http://www.postfix.org/addon.html#policy
>
> I see. So the example given in the online documentation can't work,
> and the more complete, apparently self-sufficient example
> greylist.pl given in postfix' source code can't work either. That's
> disturbing :-) But okay, I had a look at Postgrey's source code and
> it does indeed communicate through a socket, a feature that is
> definitely missing from what I tried. I'll give it another go in a
> few weeks.
>
> Thanks!
>
>
> Sébastien.

The examples work just fine.  More likely my explanation is flawed.
Refer to the documentation and examples, which are correct.



  -- Noel Jones
Seb
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Seb

Hello,


>>>>> My first guess is http://www.postfix.org/DEBUG_README.html#no_chroot
>>> Sorry, wrong guess.  Your test program is not a policy server. 
>>> http://www.postfix.org/addon.html#policy
>> I see. So the example given in the online documentation can't work, and
>> the more complete, apparently self-sufficient example greylist.pl given
>> in postfix' source code can't work either. That's disturbing :-) But
>> okay, I had a look at Postgrey's source code and it does indeed
>> communicate through a socket, a feature that is definitely missing from
>> what I tried. I'll give it another go in a few weeks.
> The examples work just fine.  More likely my explanation is flawed.
> Refer to the documentation and examples, which are correct.
Well I just followed that path: reading carefully the documentation and
reproducing as closely as possible the examples given within, to no avail.
The documentation does mention sockets, even though its examples don't use
them. So either way you are right :-)


Sébastien.

Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Noel Jones-2
On 11/7/2017 1:53 PM, Seb wrote:
> Well I just followed that path: reading carefully the documentation
> and reproducing as closely as possible the examples given within, to
> no avail. The documentation does mention sockets, even though its
> examples don't use them. So either way you are right :-)


Just because I don't see your implementation error doesn't mean
there isn't one.

I can confirm that the example greylist.pl program and sample
configuration work perfectly.

Sometimes system security features such as SELinux or AppArmor can
interfere with program access.  If you have such features, turn them
off for testing.

and of course I'm right.  ;)


  -- Noel Jones
Seb
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Seb

Hello,


> Just because I don't see your implementation error doesn't mean there
> isn't one.

Fair point.

> I can confirm that the example greylist.pl program and sample
> configuration work perfectly.

This is very comforting. Would you mind telling me more? Did you try it
yourself, just following the instructions in
  http://www.postfix.org/SMTPD_POLICY_README.html#client_config
?

> Sometimes system security features such as SELinux or AppArmor can
> interfere with program access.  If you have such features, turn them off
> for testing.

I do not use these, but following this line of thought I remembered that
for configuring DKIM I added to main.cf the following line:
  smtpd_milters = inet:localhost:12301
Could it be that I need to add here the information about greylist.pl?

On a related note, is it possible in smtpd_recipient_restrictions to use
the following two lines?
     check_policy_service inet:127.0.0.1:10023,
     check_policy_service unix:/home/seb/sandra/bin/fuck,
I mean, would the second one complement the first one (this is what I
hope) or would it interfer with it because they share the same beginning?


Kind regards,
Sébastien.
Reply | Threaded
Open this post in threaded view
|

Re: Tailored filter

Noel Jones-2
On 11/7/2017 2:36 PM, Seb wrote:
>> I can confirm that the example greylist.pl program and sample
>> configuration work perfectly.
>
> This is very comforting. Would you mind telling me more? Did you try
> it yourself, just following the instructions in
>     http://www.postfix.org/SMTPD_POLICY_README.html#client_config

I followed the instructions and "it works for me"TM

>
>> Sometimes system security features such as SELinux or AppArmor can
>> interfere with program access.  If you have such features, turn
>> them off for testing.
>
> I do not use these, but following this line of thought I remembered
> that for configuring DKIM I added to main.cf the following line:
>     smtpd_milters = inet:localhost:12301
> Could it be that I need to add here the information about greylist.pl?

Milters are a separate feature and controlled by their own
configuration.

The example config for greylist.pl shows all the necessary steps for
adding a policy service.

> On a related note, is it possible in smtpd_recipient_restrictions to
> use the following two lines?
>     check_policy_service inet:127.0.0.1:10023,
>     check_policy_service unix:/home/seb/sandra/bin/fuck,
> I mean, would the second one complement the first one (this is what
> I hope) or would it interfer with it because they share the same
> beginning?

It's valid to have as many check_policy_service lines as needed, and
they are executed in the order you specify.  Execution stops when a
policy service returns OK or REJECT.  Other actions available are
listed in the access(5) man page.




  -- Noel Jones