Logging sender recipient pairs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Logging sender recipient pairs

Chris Turan-2
Hi All,

I'm attempting to come up with a better solution for detecting email
customers who attempt to send email campaigns using my mail servers.

I'd like to find a way to have postfix log the sender and recipient
addresses into a flat file, as well as the message id and timestamp.

The idea is to count the number of envelope recipients to determine
who's sending to lots of people.  If someone goes over 500 per day, flag
them as suspicious and alert me.

Postfix already logs part of this in syslog but the recipient list is
truncated or split up between multiple syslog messages.  Its not easily
usable directly from syslog in its current form.

Anyone do anything like this yet?  Have any suggestions or alternative
ways of doing this?

-Chris
Reply | Threaded
Open this post in threaded view
|

Re: Logging sender recipient pairs

Barney Desmond
2009/7/9 Chris Turan <[hidden email]>:

> The idea is to count the number of envelope recipients to determine who's
> sending to lots of people.  If someone goes over 500 per day, flag them as
> suspicious and alert me.
>
> Postfix already logs part of this in syslog but the recipient list is
> truncated or split up between multiple syslog messages.  Its not easily
> usable directly from syslog in its current form.
>
> Anyone do anything like this yet?  Have any suggestions or alternative ways
> of doing this?

I haven't done this myself, but I hear policy servers are quite
popular for this sort of thing (the usual question is how to setup
sending quotas for users, so this would be a slight modification).
Reply | Threaded
Open this post in threaded view
|

Re: Logging sender recipient pairs

Sahil Tandon
In reply to this post by Chris Turan-2
On Wed, 08 Jul 2009, Chris Turan wrote:

> The idea is to count the number of envelope recipients to determine  
> who's sending to lots of people.  If someone goes over 500 per day, flag  
> them as suspicious and alert me.

It might be better to define a "someone" as an IP rather than an envelope
sender, which is easily spoofed.  But your implementation requirements may
not allow this.

> Postfix already logs part of this in syslog but the recipient list is  
> truncated or split up between multiple syslog messages.  Its not easily  
> usable directly from syslog in its current form.

You might be able to use the fact that qmgr(8) logs the original recipient
count.  Example:

postfix/qmgr[54662]: 98EF25C51: from=<[hidden email]>, size=717, nrcpt=5

Take care to avoid double counting in situations where mail to some
recipients is temporarily deferred.  In that case, Postfix periodically
retries and similar qmgr(8) log entries will contain that same recipient
count with that same queue ID.  That is probably one of many caveats when
parsing the logs to aggregate recipients per sender over any time period.

> Anyone do anything like this yet?  Have any suggestions or alternative  
> ways of doing this?

You could parse the logs and implement your own solution or maybe use
an existing policy service that already has this functionality.

--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Logging sender recipient pairs

Magnus Bäck
On Thursday, July 09, 2009 at 03:44 CEST,
     Sahil Tandon <[hidden email]> wrote:

[...]

> You might be able to use the fact that qmgr(8) logs the original recipient
> count.  Example:
>
> postfix/qmgr[54662]: 98EF25C51: from=<[hidden email]>, size=717, nrcpt=5
>
> Take care to avoid double counting in situations where mail to some
> recipients is temporarily deferred.  In that case, Postfix periodically
> retries and similar qmgr(8) log entries will contain that same recipient
> count with that same queue ID.  That is probably one of many caveats when
> parsing the logs to aggregate recipients per sender over any time period.

Doesn't the nrcpt atttribute contain the number of *remaining*
recipients? In that case the number will be steadily decreasing.

But regardless qmgr(8) will log this line multiple times when
deferrals occur, so care must be taken not to count the recipients
multiple times.

[...]

--
Magnus Bäck
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Logging sender recipient pairs

Victor Duchovni
On Thu, Jul 09, 2009 at 06:23:09AM +0200, Magnus Bäck wrote:

> On Thursday, July 09, 2009 at 03:44 CEST,
>      Sahil Tandon <[hidden email]> wrote:
>
> [...]
>
> > You might be able to use the fact that qmgr(8) logs the original recipient
> > count.  Example:
> >
> > postfix/qmgr[54662]: 98EF25C51: from=<[hidden email]>, size=717, nrcpt=5
> >
> > Take care to avoid double counting in situations where mail to some
> > recipients is temporarily deferred.  In that case, Postfix periodically
> > retries and similar qmgr(8) log entries will contain that same recipient
> > count with that same queue ID.  That is probably one of many caveats when
> > parsing the logs to aggregate recipients per sender over any time period.
>
> Doesn't the nrcpt atttribute contain the number of *remaining*
> recipients? In that case the number will be steadily decreasing.

No, the recipient count does not change, it is read from the "size"
record at the top of the queue file.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|

Re: Logging sender recipient pairs

brian moore-9
In reply to this post by Barney Desmond
On Thu, 9 Jul 2009 09:25:40 +1000
Barney Desmond <[hidden email]> wrote:

> I haven't done this myself, but I hear policy servers are quite
> popular for this sort of thing (the usual question is how to setup
> sending quotas for users, so this would be a slight modification).

Yes, postfixpolicyd can do this.

The real trick is, as you point out, the quotas for local users.

We had to install this here since our customers have some strange
desire to send their username and password off to whoever asks
for it despite being told multiple times that no one needs their
password....  rate limiting them works, but does break some people
who have (legit) mailing lists run off some strange homebrew
mechanism.  (Small businesses typically, the largest is the local
arts movie theater with a monthly mailing of upcoming movies.)

See http://www.policyd.org/ for details.

Easy enough to install, tricky to tweak to be useful.


Reply | Threaded
Open this post in threaded view
|

Re: Logging sender recipient pairs

Sahil Tandon
On Thu, 09 Jul 2009, brian moore wrote:

> > I haven't done this myself, but I hear policy servers are quite
> > popular for this sort of thing (the usual question is how to setup
> > sending quotas for users, so this would be a slight modification).
>
> Yes, postfixpolicyd can do this.
>
> The real trick is, as you point out, the quotas for local users.

This can also be done via postfwd with a small patch I've submitted to the
developer.  If anyone is interested, contact me off-list (as we're venturing
off-topic!) or wait for Jan to hopefully include it in the next release. :)

--
Sahil Tandon <[hidden email]>