new anti spam feature: "zombielist", help wanted

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

new anti spam feature: "zombielist", help wanted

Michael Monnerie-4
Dear spam haters,

we've developed an idea and made a quick "proof of concept" hack and
tested it during the last months, and it works very successfully. The
idea was to decrease the load on the mailgate running spamassassin +
postfix, by automatically creating a "zombielist" of IPs that are known
to be bad. These IPs are then blacklisted in postfix, this file is used
before any checks of RBLs, URIBLs, etc.
Bad in our case are
- no reverse lookup for that IP at all
- "lost connection after" messages
- RBL listing of that IP (in order to prevent future lookups)
- "day old bread": sender domains that are younger than 5 days (which
kills a lot of those http://domainnamekiting.com/ domains spams.

This simple table alone helps us reject about 30% of all spam before any
spamassassin or DNS lookups are needed. It currently lists about
250.000 single IPs for us.

The current implementation also has some downsides:
- we used the python script "blockhosts" which is horribly slow. It
reads the maillog and greps the strings from there
- the script does not run as a daemon but from cron, which means a
several minute delay from the first access of a zombie until it is
listed
- because of that delay, we set the time until deletion of an IP very
high, which makes good hosts that once had no reverse IP lookup but now
have not go off the list very quickly, meaning manual interaction
sometimes
- it's only possible to list or not list. A bias depending on which
error you got would be good in order to make a difference between "is
listed on RBL" and "junkdomain <5d old".

We've had lots of ideas for a new daemon, but we're no hackers and have
nobody to do it.

Roughly it should be able to do this:
- run as a permanent daemon
- be high speed in reading mail logs and updating records
- use an SQL db (postgreSQL at least) for managing records
- finer control over how many errors of which type are allowed
- manual whitelist IPs (e.g. known ISPs)
- a simple web interface to manage IPs
- GNU GPL licence

Anybody willing to help? I crosspost to [hidden email]
also, but please keep discussions on the postfix list if possible. It's
not really about postfix, but even less about spamassassin, it's just
another tool to get better filtering. If we start development, we'll
make a sf.net project anyway.

mfg zmi
--
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: new anti spam feature: "zombielist", help wanted

Henrik K
On Mon, May 26, 2008 at 03:54:07PM +0200, Michael Monnerie wrote:

> Dear spam haters,
>
> we've developed an idea and made a quick "proof of concept" hack and
> tested it during the last months, and it works very successfully. The
> idea was to decrease the load on the mailgate running spamassassin +
> postfix, by automatically creating a "zombielist" of IPs that are known
> to be bad. These IPs are then blacklisted in postfix, this file is used
> before any checks of RBLs, URIBLs, etc.
> Bad in our case are
> - no reverse lookup for that IP at all

Good luck. I'd reject heaps of legimate mail with that.

> - "lost connection after" messages
> - RBL listing of that IP (in order to prevent future lookups)
> - "day old bread": sender domains that are younger than 5 days (which
> kills a lot of those http://domainnamekiting.com/ domains spams.
>
> This simple table alone helps us reject about 30% of all spam before any
> spamassassin or DNS lookups are needed. It currently lists about
> 250.000 single IPs for us.
>
> The current implementation also has some downsides:
> - we used the python script "blockhosts" which is horribly slow. It
> reads the maillog and greps the strings from there
> - the script does not run as a daemon but from cron, which means a
> several minute delay from the first access of a zombie until it is
> listed

I just use a simple perl-script that blocks hosts with iptables. It uses
File::Tail to monitor log, so blocking is done almost without delays. I
don't see much point in using postfix to block things, since it takes more
resources.

http://hege.li/contrib/smtp_block.pl

Reply | Threaded
Open this post in threaded view
|

Re: new anti spam feature: "zombielist", help wanted

Michael Monnerie-4
In reply to this post by Michael Monnerie-4
On Dienstag, 27. Mai 2008 Henrik K wrote:
> What do you do when you get a single spam from hotmail/gmail etc? You
> are going to defer a host that sends you mostly legimate mails?

No, it's not even about spam currently. It's plainly one of the three
options I mentioned in my first e-mail (see also below) than can get
you on the list.
Also, there's a counter, and only after several errors a host get's
listed.
With the new daemon, it should be possible to count for a sender IP the
number of hams/spams and then you could say "if a sender sends 50% spam
during the last 24h, block it".

> You should modify it atleast to take account of how much ham has been
> sent.

That is currently not implemented as we do not look at this at all.

> I would continue this on amavis mailing list, since it's a bit
> offtopic.

It's also not an amavis thingy, but a good idea I will drop the message
there also. It best fits on postfix-users I guess, but we will see.

[From another post:]
> what's the goal here exactly?

* reduce the CPU and network load:
- you don't need to ask RBLs too often
- if the sender server does not have a rDNS, you block it for a while
- sender's domains younger than 5 days are blocked
- and with the new daemon I want to be able to have finer control, so
you can also block senders who constantly connect and then "lost
connection after ..." (we've had this once from lot's of hosts, looked
like a dDoS). The current implementation only gives one possible
timeout value for delisting, while the new should have a timeout per
rule (eg. delist IP after 6h if listed because of RBL, but 1d if listed
because of no reverse IP).

[From another post:]
>> - no reverse lookup for that IP at all
> Good luck. I'd reject heaps of legimate mail with that.

We've started implementing that early 2007. By then, we've had some
issues, but they are constantly reducing. We're in Austria/Europe, and
one big advantage is that GMX is also rejecting in case of missing
rDNS: http://faq.gmx.de/optionen/email/antispam/4.html
If nobody does start to enforce stricter rules, the SPAM problem won't
go away. But anyway, it's of course configurable if you want to reject
on a missing rDNS or not.
One solution could be to make a "transition phase", in that you greylist
hosts without rDNS or reject e-mails for an hour with a message that
they should create a rDNS entry and then allow for an hour... but not
doing anything because that could possibly upset someone with a
not-so-clean setup is not on our mind.
Currently, when we have a valid customer w/o rDNS then we whitelist then
for 14 days, and send an info e-mail to them and their postmaster@,
with the explanation what to do. Works quite well, very seldom we hear
again after 14 days that they want to be whitelisted again - and only
one I had somebody 3x whitelisted ;-)

> I just use a simple perl-script that blocks hosts with iptables. It
> uses File::Tail to monitor log, so blocking is done almost without
> delays. I don't see much point in using postfix to block things,
> since it takes more resources.
> http://hege.li/contrib/smtp_block.pl

That way you cannot even see when a host tried to send an e-mail, which
makes looking for missing e-mails impossible. If you have a false
positive, you cannot even look it up in mail logs. That's absolutely a
no-go for us. Either we accept an e-mail or we reject it - on SMTP
level. It might be a good idea for hosts that behave very badly though
(say, both listed on RBLs and no rDNS), but the little overhead
involved in rejecting at SMTP level is no real ressource pain.

But it's a good start for a daemon :-)

mfg zmi
--
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: new anti spam feature: "zombielist", help wanted

Henrik K
On Tue, May 27, 2008 at 10:30:56AM +0200, Michael Monnerie wrote:

>
> > I just use a simple perl-script that blocks hosts with iptables. It
> > uses File::Tail to monitor log, so blocking is done almost without
> > delays. I don't see much point in using postfix to block things,
> > since it takes more resources.
> > http://hege.li/contrib/smtp_block.pl
>
> That way you cannot even see when a host tried to send an e-mail, which
> makes looking for missing e-mails impossible. If you have a false
> positive, you cannot even look it up in mail logs. That's absolutely a
> no-go for us. Either we accept an e-mail or we reject it - on SMTP

Obviously the first try will be logged. I only add blocks because of RBL
listing or other rules that would reject. The mail would be lost anyway. You
could even reduce the block to 10 minutes to ease some hammering.

Also try to reply properly. Instead of stuffing everything to one mail, you
even replied some of my text that wasn't directed at you to 3 lists.