Re: Porn spam killer PCRE

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Nikolaos Milas
On 20/8/2016 7:56 μμ, Sean Greenslade wrote:

> ...
> - Enable & configure per-user bayesan filtering
> - Increase allowed storage space for bayesan databases
> - Update a particular perl package to make SPF work (CentOS / RHEL
>    specific bug)
> ...
> - Re-weight a bunch of internal rules, in particular the bayes, SPF,
>    and mailspike rules
>
> Another thing I did was enable the spam report to be added to all
> messages, that way I could more easily debug why spam that was getting
> past the filter didn't trigger it.
>
> ...I slowly ramped up rule weights and waited to see what would happen. ...

Thank you Sean for your advice. I would appreciate if you provide more
technical details on how to try the above and/or indicate documentation
links with associated guide(s).

Your experience might help avoid re-inventing the wheel while trying to
make users life easier in our really congested daily routine.

Thanks in advance,
Nick
Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Sean Greenslade
On Tue, Aug 23, 2016 at 11:27:39PM +0300, Nikolaos Milas wrote:

> On 20/8/2016 7:56 μμ, Sean Greenslade wrote:
>
> > ...
> > - Enable & configure per-user bayesan filtering
> > - Increase allowed storage space for bayesan databases
> > - Update a particular perl package to make SPF work (CentOS / RHEL
> >    specific bug)
> > ...
> > - Re-weight a bunch of internal rules, in particular the bayes, SPF,
> >    and mailspike rules
> >
> > Another thing I did was enable the spam report to be added to all
> > messages, that way I could more easily debug why spam that was getting
> > past the filter didn't trigger it.
> >
> > ...I slowly ramped up rule weights and waited to see what would happen. ...
>
> Thank you Sean for your advice. I would appreciate if you provide more
> technical details on how to try the above and/or indicate documentation
> links with associated guide(s).
>
> Your experience might help avoid re-inventing the wheel while trying to make
> users life easier in our really congested daily routine.

As I said, it's going to need to be tailored to your particular use
case, so my config files probably won't be useful as-is to you. Fighting
spam is a constantly changing battle, and it really depends on what
types of spam you're currently suffering from.  I'll provide some
snippets so that you can see what I'm doing. Any filenames will be the
CentOS variants or my own personal systems.

The main spamassassin conf (/etc/mail/spamassassin/local.cf) has all of
my spamassassin modifications.

I've added these settings to help analyze SA's decisions:
> report_safe 0 # Don't encapsulate spam, just tag it.
> add_header all Report _REPORT_ # Provide spam report in all mails.

My bayesian filter settings:
> # Bayesian filtering yeah!
> bayes_path /var/mail/bayes_db/bayes
> bayes_file_mode 0775
> bayes_expiry_max_db_size 300000
> # 150000 = ~4 MiB

The main tweak is the size increase. See this page for bayes help:
https://wiki.apache.org/spamassassin/BayesFaq

One of my custom rules, this one for .docm files:
> loadplugin Mail::SpamAssassin::Plugin::MIMEHeader
> mimeheader S_DOCM_ATTACHED Content-Type =~ /docm|ms-word\.document\.macroEnabled/i
> describe   S_DOCM_ATTACHED email contains a docm file attachment
> score      S_DOCM_ATTACHED 4.5

Something to note is that none of my rules ever have enough weight on
their own to trigger the filter. But don't worry, most spam will trigger
many, many rules. I routinely get messages that have a spam score of
60+ points, with a trigger threshold of 5.2 points.

My custom rule for the sales@* spam:
> header   S_SALESFORCE            To =~ /sales\@/i
> describe S_SALESFORCE            The sales team has been complaining about all the spam.
> score    S_SALESFORCE            2.0

Note that this rule has a lower weight, because I'm less certain that
this alone is indicative of spam.

And finally, a few of my custom rule weights. These came from analysing
incoming spam that was missed and figuring out which rules triggered on
it most reliably:
> score LOTS_OF_MONEY 0.9
> score FREEMAIL_FORGED_FROMDOMAIN 0.5
> score WEIRD_PORT 1.5
> score TO_IN_SUBJ 1.0

And these I added to make sure they work, since SA has some weird rule
weights that disable certain checks if bayesian filtering is on:

> score RCVD_IN_MSPIKE_BL 1.0
> score RCVD_IN_MSPIKE_L5 2.5
> score RCVD_IN_MSPIKE_L4 2.0
> score RCVD_IN_MSPIKE_L3 1.5
> score RCVD_IN_MSPIKE_L2 1.0
> score RCVD_IN_MSPIKE_H5 -1.0
> score RCVD_IN_MSPIKE_H4 -0.8
> score RCVD_IN_MSPIKE_H3 -0.6
> score RCVD_IN_MSPIKE_H2 -0.4
> score RCVD_IN_MSPIKE_WL -0.5
> score SPF_NONE 0.001
> score SPF_HELO_NONE 0.001
> score SPF_PASS -0.1
> score SPF_HELO_PASS -0.1
> score SPF_FAIL 3.0
> score SPF_HELO_FAIL 3.0
> score SPF_NEUTRAL 0.001
> score SPF_HELO_NEUTRAL 0.001
> score SPF_SOFTFAIL 1.0
> score SPF_HELO_SOFTFAIL 1.0


To tie spamassassin into postfix, I use these settings in
/etc/postfix/master.cf:
> smtp      inet  n       -       n       -       -       smtpd
>   -o content_filter=spamfilter
>   -o smtpd_tls_security_level=may
>   -o smtpd_sasl_auth_enable=no
> # ...
> spamfilter
>           unix  -       n       n       -       -       pipe
>    flags=Rq user=mailfilter argv=/opt/postfix_spamcheck.sh -oi -f ${sender} ${recipient}

And /opt/postfix_spamcheck.sh:

> #!/bin/bash
> # Simple filter to plug SpamAssassin into the Postfix MTA
> # File locations:
> # (CHANGE AS REQUIRED TO SUIT YOUR SERVER)
> SENDMAIL=/usr/sbin/sendmail
> SPAMASSASSIN=/usr/bin/spamc
>
> #logger <<<"Spam filter piping to SpamAssassin, then to: $SENDMAIL $@"
> ${SPAMASSASSIN} | ${SENDMAIL} "$@"
>
> exit $?


Then I have some scripts that read mails out of a specific maildir
(isspam) and runs sa-learn on them to allow users to train the bayesian
filter from thir mail clients. I won't post those since they are heavily
tied into my system for sorting and processing emails, which is quite
unusual and full of python.

Hope this is helpful,

--Sean

Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Nikolaos Milas
On 23/8/2016 11:58 μμ, Sean Greenslade wrote:

> Hope this is helpful,

Thanks Sean for your time and eagerness to help. I appreciate it.

I am planning to try your suggestions.

I am using CentOS 6. Can you please let me know about the perl package
SPF-related bug you mentioned?

I have installed:

    yum --enablerepo=epel install perl-Mail-SPF perl-Sys-Hostname-Long

Would this be OK, or I should add/change something?

Finally, I note that in my setup I am loading/running spamassassin from
within amavis and not directly in postfix.

Thanks again,
Nick

Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Sean Greenslade
On Thu, Aug 25, 2016 at 05:28:35PM +0300, Nikolaos Milas wrote:

> On 23/8/2016 11:58 μμ, Sean Greenslade wrote:
>
> > Hope this is helpful,
>
> Thanks Sean for your time and eagerness to help. I appreciate it.
>
> I am planning to try your suggestions.
>
> I am using CentOS 6. Can you please let me know about the perl package
> SPF-related bug you mentioned?

The bug is a CentOS 7-specific bug as far as I know. Here is the bug
page if you're interested (see comment 29):
https://bugzilla.redhat.com/show_bug.cgi?id=1200167

I would also suggest considering upgrading to CentOS 7, as 6 is going to
end full updates next year.

> I have installed:
>
>    yum --enablerepo=epel install perl-Mail-SPF perl-Sys-Hostname-Long
>
> Would this be OK, or I should add/change something?

Make sure the SPF rules have weights set, then check the spam report for
a message. If SPF is working, you should see at least one of the SPF
rules trigger for every message.

> Finally, I note that in my setup I am loading/running spamassassin from
> within amavis and not directly in postfix.

I'm not familiar with amavis, but I doubt that would be a problem.
SpamAssassin runs as the spamd daemon, which loads its configuration
directly from the /etc files.

--Sean

Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Nikolaos Milas
On 25/8/2016 7:46 μμ, Sean Greenslade wrote:

> Make sure the SPF rules have weights set, then check the spam report for
> a message. If SPF is working, you should see at least one of the SPF
> rules trigger for every message.

Thank you Sean for all your help.

I am focusing on amavisd-new / spamassassin for my efforts.

I have started a thread in the amavisd-new users mailing list, since
bayesian filtering is off-topic here.

However, before ending this thread, I would like to ask: all these rules
for which you have defined scores:

    score LOTS_OF_MONEY 0.9
    score FREEMAIL_FORGED_FROMDOMAIN 0.5
    score WEIRD_PORT 1.5
    score TO_IN_SUBJ 1.0

    score RCVD_IN_MSPIKE_BL 1.0
    score RCVD_IN_MSPIKE_L5 2.5
    score RCVD_IN_MSPIKE_L4 2.0
    ...

...are custom rules you have set up? If so, what do they look like?(You
may want to reply off-list, if you deem it is more approrpriate.)

The same question goes for the SPF rules.

I am asking because I see:

    # spamassassin -D --lint
    ...
    Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_H4
    Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_WL
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule FREEMAIL_FORGED_FROMDOMAIN
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_L5
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_H3
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule PP_TOO_MUCH_UNICODE02
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule HEADER_HOST_IN_WHITELIST
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule URI_HOST_IN_BLACKLIST
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_H2
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule HEADER_FROM_DIFFERENT_DOMAINS
    Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
    non-existent rule URI_HOST_IN_WHITELIST
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule HELO_LH_HOME
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule HEADER_HOST_IN_BLACKLIST
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_ZBI
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_L2
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule HK_NAME_MR_MRS
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_BL
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule PP_TOO_MUCH_UNICODE05
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_L4
    Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_H5
    Aug 27 17:13:56.863 [10384] dbg: config: warning: score set for
    non-existent rule URIBL_SBL_A
    Aug 27 17:13:56.863 [10384] dbg: config: warning: score set for
    non-existent rule PP_MIME_FAKE_ASCII_TEXT
    Aug 27 17:13:56.863 [10384] dbg: config: warning: score set for
    non-existent rule RCVD_IN_MSPIKE_L3
    ...

Also note that re. SPF tests, I see:

    # spamassassin -D --lint
    ...
    Aug 27 17:13:58.228 [10384] warn: rules: failed to run SPF_NONE
    test, skipping:
    Aug 27 17:13:58.229 [10384] warn:  (Can't locate object method
    "check_for_spf_none" via package "Mail:
    [...]:SpamAssassin::PerMsgStatus" at (eval 1241) line 343.
    Aug 27 17:13:58.229 [10384] warn: )
    ...
    Aug 27 17:13:58.231 [10384] warn: rules: failed to run SPF_HELO_NONE
    test, skipping:
    Aug 27 17:13:58.231 [10384] warn:  (Can't locate object method
    "check_for_spf_helo_none" via package "Mail:
    [...]:SpamAssassin::PerMsgStatus" at (eval 1241) line 959.
    Aug 27 17:13:58.231 [10384] warn: )
    ...

What may be a problem here?

Nick

Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Sean Greenslade
On Sat, Aug 27, 2016 at 05:27:18PM +0300, Nikolaos Milas wrote:

> Thank you Sean for all your help.
>
> I am focusing on amavisd-new / spamassassin for my efforts.
>
> I have started a thread in the amavisd-new users mailing list, since
> bayesian filtering is off-topic here.
>
> However, before ending this thread, I would like to ask: all these rules for
> which you have defined scores:
>
>    score LOTS_OF_MONEY 0.9
>    score FREEMAIL_FORGED_FROMDOMAIN 0.5
>    score WEIRD_PORT 1.5
>    score TO_IN_SUBJ 1.0
>
>    score RCVD_IN_MSPIKE_BL 1.0
>    score RCVD_IN_MSPIKE_L5 2.5
>    score RCVD_IN_MSPIKE_L4 2.0
>    ...
>
> ...are custom rules you have set up? If so, what do they look like?(You may
> want to reply off-list, if you deem it is more approrpriate.)

These are not custom rules. You may need to look into the amavisd
documentation to see if they do anything unusual with SA, but those
rules are part of the normal SA install.

> The same question goes for the SPF rules.
>
> I am asking because I see:
>
>    # spamassassin -D --lint
>    ...
>    Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
>    non-existent rule RCVD_IN_MSPIKE_H4
>    Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
>    non-existent rule RCVD_IN_MSPIKE_WL
> <SNIP>
> Also note that re. SPF tests, I see:
>
>    # spamassassin -D --lint
>    ...
>    Aug 27 17:13:58.228 [10384] warn: rules: failed to run SPF_NONE
>    test, skipping:
>    Aug 27 17:13:58.229 [10384] warn:  (Can't locate object method
>    "check_for_spf_none" via package "Mail:
> <SNIP>
>
> What may be a problem here?

Since you're not using SA in the same way I am, I can't really say. Note
that SA has a rules update mechanism that may need to be triggered to
get the full rule set.

--Sean

Reply | Threaded
Open this post in threaded view
|

Re: Porn spam killer PCRE

Benny Pedersen-2
In reply to this post by Nikolaos Milas
On 2016-08-27 16:27, Nikolaos Milas wrote:

> What may be a problem here?

you miss update of SPF plugin to a never version

you may need to if version > xxxx around rules there

so spf none only is used on versions that support it

upstream bug here