monitoring outgoing emails

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
Thank you for answer. I am going to use your command - without any typos :P - and wrap it by some bash script which will check the "Hits" value and send email with report. I hope I will do it. :)

2018-03-30 17:52 GMT+02:00 chaouche yacine <[hidden email]>:
Absolutely. Amavis comes with a default score of 5.0. Any e-mail which has a 5.0 score or higher is considered spam. You might have false positives though, for example if the user's ISP addresses are blacklisted, which might be the case dependning on the country and ISP.

Yassine.

On Friday, March 30, 2018, 10:44:27 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Yassine, appreciate your answer. I will check further in it but do you think that spam score could help with estimate which mail from which account is or not spam?

2018-03-30 9:27 GMT+02:00 chaouche yacine <[hidden email]>:
Here are some ideas :

1/ Create a directory somewhere in /var/, for example mailstats
2/ The directory will contain one file per sender
3/ Your bash script will parse the mail log file in real time (tail -f) then tee each matching line to the corresponding mailstats/user file, for example if the line is matching [hidden email] it will go to mailstats/bob. That way you will have, for each user, the number of outgoing emails.


Another script will simply wc -l each mailstats user file, that will give you the number of sent mails. You can use fail2ban for this task instead of writing you own script. Fail2ban can be configured to scan logfiles looking for a particular line. It will count the matching lines and if it reaches the (configurable) maximum count in a certain (configurable) amount of time, it will do whatever action you have configured, for example sending you an e-mail.

The mailstats file will need some maintenance, otherwise they will grow infinitely and possibly slow down you scripts. You can use logrotate to archive your mailstats files and create new ones automatically for you after either a specific amount of time or after a specific mail size.

It's not trivial, but it should work.


Yassine.


On Friday, March 30, 2018, 7:16:33 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Some emails has "Hits" value even, for example 2,5. What is (if it's possible to say) good value? I am going to create script in bash  which send me an email when from particular email account will outbound for example 300 emails per day. Kind of warning. But I am not sure I could use spam score to it. What do you think guys about it?

2018-03-29 17:58 GMT+02:00 chaouche yacine <[hidden email]>:

It is, that's the spam score. It helps to visualise if a particular mailbox is bombarded with spam (can happen with lots and lots of e-mails from qq.com, I have that domain banned in postfix itself).

Yassine.
On Thursday, March 29, 2018, 3:21:16 PM GMT+1, Alex JOST <[hidden email]> wrote:


Am 29.03.2018 um 15:30 schrieb Poliman - Serwis:

> This one works well. One question based on one from generated lines:
> Mar 26 11:47:41  ORIGINATING LOCAL [127.0.0.1]:38920 <[hidden email]>
> -> <[hidden email]>,<[hidden email]>, Hits: 0.742
>
> Mar 26 11:47:41 --> this is date and hour when mail from
> [hidden email] was sent to [hidden email] and
> [hidden email], am I right?
> What are "Hits: 0.742" ?


Looks like amavisd scoring.

--
Alex JOST




--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
In reply to this post by chaouche yacine
Could you tell me I could add e-mails together from mail.log which are in line with "from=" part? Hmm I hope I say clear. I need count emails from particular mailbox. Can I base on "from="? For example:
Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>, size=4000, nrcpt=1 (queue active)

2018-03-30 17:52 GMT+02:00 chaouche yacine <[hidden email]>:
Absolutely. Amavis comes with a default score of 5.0. Any e-mail which has a 5.0 score or higher is considered spam. You might have false positives though, for example if the user's ISP addresses are blacklisted, which might be the case dependning on the country and ISP.

Yassine.

On Friday, March 30, 2018, 10:44:27 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Yassine, appreciate your answer. I will check further in it but do you think that spam score could help with estimate which mail from which account is or not spam?

2018-03-30 9:27 GMT+02:00 chaouche yacine <[hidden email]>:
Here are some ideas :

1/ Create a directory somewhere in /var/, for example mailstats
2/ The directory will contain one file per sender
3/ Your bash script will parse the mail log file in real time (tail -f) then tee each matching line to the corresponding mailstats/user file, for example if the line is matching [hidden email] it will go to mailstats/bob. That way you will have, for each user, the number of outgoing emails.


Another script will simply wc -l each mailstats user file, that will give you the number of sent mails. You can use fail2ban for this task instead of writing you own script. Fail2ban can be configured to scan logfiles looking for a particular line. It will count the matching lines and if it reaches the (configurable) maximum count in a certain (configurable) amount of time, it will do whatever action you have configured, for example sending you an e-mail.

The mailstats file will need some maintenance, otherwise they will grow infinitely and possibly slow down you scripts. You can use logrotate to archive your mailstats files and create new ones automatically for you after either a specific amount of time or after a specific mail size.

It's not trivial, but it should work.


Yassine.


On Friday, March 30, 2018, 7:16:33 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Some emails has "Hits" value even, for example 2,5. What is (if it's possible to say) good value? I am going to create script in bash  which send me an email when from particular email account will outbound for example 300 emails per day. Kind of warning. But I am not sure I could use spam score to it. What do you think guys about it?

2018-03-29 17:58 GMT+02:00 chaouche yacine <[hidden email]>:

It is, that's the spam score. It helps to visualise if a particular mailbox is bombarded with spam (can happen with lots and lots of e-mails from qq.com, I have that domain banned in postfix itself).

Yassine.
On Thursday, March 29, 2018, 3:21:16 PM GMT+1, Alex JOST <[hidden email]> wrote:


Am 29.03.2018 um 15:30 schrieb Poliman - Serwis:

> This one works well. One question based on one from generated lines:
> Mar 26 11:47:41  ORIGINATING LOCAL [127.0.0.1]:38920 <[hidden email]>
> -> <[hidden email]>,<[hidden email]>, Hits: 0.742
>
> Mar 26 11:47:41 --> this is date and hour when mail from
> [hidden email] was sent to [hidden email] and
> [hidden email], am I right?
> What are "Hits: 0.742" ?


Looks like amavisd scoring.

--
Alex JOST




--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
Hmm, probably I can't base on this, because when I send one email I have in log three lines with "from=" and value <[hidden email]>.
1st line --> Apr  4 09:32:41 s1 postfix/submission/smtpd[5622]: NOQUEUE: filter: RCPT from host-X.Y.Z.W.static.com[X.Y.Z.W]: < [hidden email] >: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=< [hidden email] > to=<[hidden email]> proto=ESMTP helo=<[192.168.101.112]>
2nd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: 74F9980483: from=< [hidden email]>, size=4359, nrcpt=1 (queue active)
3rd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: E180480484: from=< [hidden email]>, size=4931, nrcpt=1 (queue active)


2018-04-04 7:53 GMT+02:00 Poliman - Serwis <[hidden email]>:
Could you tell me I could add e-mails together from mail.log which are in line with "from=" part? Hmm I hope I say clear. I need count emails from particular mailbox. Can I base on "from="? For example:
Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>, size=4000, nrcpt=1 (queue active)

2018-03-30 17:52 GMT+02:00 chaouche yacine <[hidden email]>:
Absolutely. Amavis comes with a default score of 5.0. Any e-mail which has a 5.0 score or higher is considered spam. You might have false positives though, for example if the user's ISP addresses are blacklisted, which might be the case dependning on the country and ISP.

Yassine.

On Friday, March 30, 2018, 10:44:27 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Yassine, appreciate your answer. I will check further in it but do you think that spam score could help with estimate which mail from which account is or not spam?

2018-03-30 9:27 GMT+02:00 chaouche yacine <[hidden email]>:
Here are some ideas :

1/ Create a directory somewhere in /var/, for example mailstats
2/ The directory will contain one file per sender
3/ Your bash script will parse the mail log file in real time (tail -f) then tee each matching line to the corresponding mailstats/user file, for example if the line is matching [hidden email] it will go to mailstats/bob. That way you will have, for each user, the number of outgoing emails.


Another script will simply wc -l each mailstats user file, that will give you the number of sent mails. You can use fail2ban for this task instead of writing you own script. Fail2ban can be configured to scan logfiles looking for a particular line. It will count the matching lines and if it reaches the (configurable) maximum count in a certain (configurable) amount of time, it will do whatever action you have configured, for example sending you an e-mail.

The mailstats file will need some maintenance, otherwise they will grow infinitely and possibly slow down you scripts. You can use logrotate to archive your mailstats files and create new ones automatically for you after either a specific amount of time or after a specific mail size.

It's not trivial, but it should work.


Yassine.


On Friday, March 30, 2018, 7:16:33 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Some emails has "Hits" value even, for example 2,5. What is (if it's possible to say) good value? I am going to create script in bash  which send me an email when from particular email account will outbound for example 300 emails per day. Kind of warning. But I am not sure I could use spam score to it. What do you think guys about it?

2018-03-29 17:58 GMT+02:00 chaouche yacine <[hidden email]>:

It is, that's the spam score. It helps to visualise if a particular mailbox is bombarded with spam (can happen with lots and lots of e-mails from qq.com, I have that domain banned in postfix itself).

Yassine.
On Thursday, March 29, 2018, 3:21:16 PM GMT+1, Alex JOST <[hidden email]> wrote:


Am 29.03.2018 um 15:30 schrieb Poliman - Serwis:

> This one works well. One question based on one from generated lines:
> Mar 26 11:47:41  ORIGINATING LOCAL [127.0.0.1]:38920 <[hidden email]>
> -> <[hidden email]>,<[hidden email]>, Hits: 0.742
>
> Mar 26 11:47:41 --> this is date and hour when mail from
> [hidden email] was sent to [hidden email] and
> [hidden email], am I right?
> What are "Hits: 0.742" ?


Looks like amavisd scoring.

--
Alex JOST




--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
Or maybe I could base on this value but divided by 3.

2018-04-04 9:43 GMT+02:00 Poliman - Serwis <[hidden email]>:
Hmm, probably I can't base on this, because when I send one email I have in log three lines with "from=" and value <[hidden email]>.
1st line --> Apr  4 09:32:41 s1 postfix/submission/smtpd[5622]: NOQUEUE: filter: RCPT from host-X.Y.Z.W.static.com[X.Y.Z.W]: < [hidden email] >: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=< [hidden email] > to=<[hidden email]> proto=ESMTP helo=<[192.168.101.112]>
2nd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: 74F9980483: from=< [hidden email]>, size=4359, nrcpt=1 (queue active)
3rd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: E180480484: from=< [hidden email]>, size=4931, nrcpt=1 (queue active)


2018-04-04 7:53 GMT+02:00 Poliman - Serwis <[hidden email]>:
Could you tell me I could add e-mails together from mail.log which are in line with "from=" part? Hmm I hope I say clear. I need count emails from particular mailbox. Can I base on "from="? For example:
Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>, size=4000, nrcpt=1 (queue active)

2018-03-30 17:52 GMT+02:00 chaouche yacine <[hidden email]>:
Absolutely. Amavis comes with a default score of 5.0. Any e-mail which has a 5.0 score or higher is considered spam. You might have false positives though, for example if the user's ISP addresses are blacklisted, which might be the case dependning on the country and ISP.

Yassine.

On Friday, March 30, 2018, 10:44:27 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Yassine, appreciate your answer. I will check further in it but do you think that spam score could help with estimate which mail from which account is or not spam?

2018-03-30 9:27 GMT+02:00 chaouche yacine <[hidden email]>:
Here are some ideas :

1/ Create a directory somewhere in /var/, for example mailstats
2/ The directory will contain one file per sender
3/ Your bash script will parse the mail log file in real time (tail -f) then tee each matching line to the corresponding mailstats/user file, for example if the line is matching [hidden email] it will go to mailstats/bob. That way you will have, for each user, the number of outgoing emails.


Another script will simply wc -l each mailstats user file, that will give you the number of sent mails. You can use fail2ban for this task instead of writing you own script. Fail2ban can be configured to scan logfiles looking for a particular line. It will count the matching lines and if it reaches the (configurable) maximum count in a certain (configurable) amount of time, it will do whatever action you have configured, for example sending you an e-mail.

The mailstats file will need some maintenance, otherwise they will grow infinitely and possibly slow down you scripts. You can use logrotate to archive your mailstats files and create new ones automatically for you after either a specific amount of time or after a specific mail size.

It's not trivial, but it should work.


Yassine.


On Friday, March 30, 2018, 7:16:33 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Some emails has "Hits" value even, for example 2,5. What is (if it's possible to say) good value? I am going to create script in bash  which send me an email when from particular email account will outbound for example 300 emails per day. Kind of warning. But I am not sure I could use spam score to it. What do you think guys about it?

2018-03-29 17:58 GMT+02:00 chaouche yacine <[hidden email]>:

It is, that's the spam score. It helps to visualise if a particular mailbox is bombarded with spam (can happen with lots and lots of e-mails from qq.com, I have that domain banned in postfix itself).

Yassine.
On Thursday, March 29, 2018, 3:21:16 PM GMT+1, Alex JOST <[hidden email]> wrote:


Am 29.03.2018 um 15:30 schrieb Poliman - Serwis:

> This one works well. One question based on one from generated lines:
> Mar 26 11:47:41  ORIGINATING LOCAL [127.0.0.1]:38920 <[hidden email]>
> -> <[hidden email]>,<[hidden email]>, Hits: 0.742
>
> Mar 26 11:47:41 --> this is date and hour when mail from
> [hidden email] was sent to [hidden email] and
> [hidden email], am I right?
> What are "Hits: 0.742" ?


Looks like amavisd scoring.

--
Alex JOST




--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

chaouche yacine
The log line from avmavis already has the sender a single time, regardless of the number of recipients.

Also, if you grep on from, keep in mind that the email first goes from outside to postfix (1st from), the from postfix to amavis (second from), then from amavis back to postfix (third from).



Yassine.


On Wednesday, April 4, 2018, 8:49:43 AM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Or maybe I could base on this value but divided by 3.

2018-04-04 9:43 GMT+02:00 Poliman - Serwis <[hidden email]>:
Hmm, probably I can't base on this, because when I send one email I have in log three lines with "from=" and value <[hidden email]>.
1st line --> Apr  4 09:32:41 s1 postfix/submission/smtpd[5622] : NOQUEUE: filter: RCPT from host-X.Y.Z.W.static.com[X.Y.Z. W]: < [hidden email] >: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=< [hidden email] > to=<[hidden email]> proto=ESMTP helo=<[192.168.101.112]>
2nd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: 74F9980483: from=< [hidden email]>, size=4359, nrcpt=1 (queue active)
3rd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: E180480484: from=< [hidden email]>, size=4931, nrcpt=1 (queue active)


2018-04-04 7:53 GMT+02:00 Poliman - Serwis <[hidden email]>:
Could you tell me I could add e-mails together from mail.log which are in line with "from=" part? Hmm I hope I say clear. I need count emails from particular mailbox. Can I base on "from="? For example:
Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>, size=4000, nrcpt=1 (queue active)

2018-03-30 17:52 GMT+02:00 chaouche yacine <[hidden email]>:
Absolutely. Amavis comes with a default score of 5.0. Any e-mail which has a 5.0 score or higher is considered spam. You might have false positives though, for example if the user's ISP addresses are blacklisted, which might be the case dependning on the country and ISP.

Yassine.

On Friday, March 30, 2018, 10:44:27 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Yassine, appreciate your answer. I will check further in it but do you think that spam score could help with estimate which mail from which account is or not spam?

2018-03-30 9:27 GMT+02:00 chaouche yacine <[hidden email]>:
Here are some ideas :

1/ Create a directory somewhere in /var/, for example mailstats
2/ The directory will contain one file per sender
3/ Your bash script will parse the mail log file in real time (tail -f) then tee each matching line to the corresponding mailstats/user file, for example if the line is matching [hidden email] it will go to mailstats/bob. That way you will have, for each user, the number of outgoing emails.


Another script will simply wc -l each mailstats user file, that will give you the number of sent mails. You can use fail2ban for this task instead of writing you own script. Fail2ban can be configured to scan logfiles looking for a particular line. It will count the matching lines and if it reaches the (configurable) maximum count in a certain (configurable) amount of time, it will do whatever action you have configured, for example sending you an e-mail.

The mailstats file will need some maintenance, otherwise they will grow infinitely and possibly slow down you scripts. You can use logrotate to archive your mailstats files and create new ones automatically for you after either a specific amount of time or after a specific mail size.

It's not trivial, but it should work.


Yassine.


On Friday, March 30, 2018, 7:16:33 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Some emails has "Hits" value even, for example 2,5. What is (if it's possible to say) good value? I am going to create script in bash  which send me an email when from particular email account will outbound for example 300 emails per day. Kind of warning. But I am not sure I could use spam score to it. What do you think guys about it?

2018-03-29 17:58 GMT+02:00 chaouche yacine <[hidden email]>:

It is, that's the spam score. It helps to visualise if a particular mailbox is bombarded with spam (can happen with lots and lots of e-mails from qq.com, I have that domain banned in postfix itself).

Yassine.
On Thursday, March 29, 2018, 3:21:16 PM GMT+1, Alex JOST <[hidden email]> wrote:


Am 29.03.2018 um 15:30 schrieb Poliman - Serwis:

> This one works well. One question based on one from generated lines:
> Mar 26 11:47:41  ORIGINATING LOCAL [127.0.0.1]:38920 <[hidden email]>
> -> <[hidden email]>,<[hidden email]>, Hits: 0.742
>
> Mar 26 11:47:41 --> this is date and hour when mail from
> [hidden email] was sent to [hidden email] and
> [hidden email], am I right?
> What are "Hits: 0.742" ?


Looks like amavisd scoring.

--
Alex JOST




--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Wietse Venema
In reply to this post by Poliman - Serwis
Poliman - Serwis:
> Could you tell me I could add e-mails together from mail.log which are in
> line with "from=" part? Hmm I hope I say clear. I need count emails from
> particular mailbox. Can I base on "from="? For example:
> Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>,
> size=4000, nrcpt=1 (queue active)

The script auxiliary/collate/collate.pl (in the Postfix source-code
distribution) combines records from multiple Postfix daemons into
one transaction (mainly, a group of logfile records with the same
queue ID).

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
In reply to this post by chaouche yacine
I am not sure I understood well. There are three "from=", and you said which one repond to which behavior, so I think I could base on "from=" from log file but I should divide by three number of emails send by specific user. Am I right?

2018-04-04 11:11 GMT+02:00 chaouche yacine <[hidden email]>:
The log line from avmavis already has the sender a single time, regardless of the number of recipients.

Also, if you grep on from, keep in mind that the email first goes from outside to postfix (1st from), the from postfix to amavis (second from), then from amavis back to postfix (third from).



Yassine.


On Wednesday, April 4, 2018, 8:49:43 AM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Or maybe I could base on this value but divided by 3.

2018-04-04 9:43 GMT+02:00 Poliman - Serwis <[hidden email]>:
Hmm, probably I can't base on this, because when I send one email I have in log three lines with "from=" and value <[hidden email]>.
1st line --> Apr  4 09:32:41 s1 postfix/submission/smtpd[5622] : NOQUEUE: filter: RCPT from host-X.Y.Z.W.static.com[X.Y.Z. W]: < [hidden email] >: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=< [hidden email] > to=<[hidden email]> proto=ESMTP helo=<[192.168.101.112]>
2nd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: 74F9980483: from=< [hidden email]>, size=4359, nrcpt=1 (queue active)
3rd line --> Apr  4 09:32:41 s1 postfix/qmgr[4801]: E180480484: from=< [hidden email]>, size=4931, nrcpt=1 (queue active)


2018-04-04 7:53 GMT+02:00 Poliman - Serwis <[hidden email]>:
Could you tell me I could add e-mails together from mail.log which are in line with "from=" part? Hmm I hope I say clear. I need count emails from particular mailbox. Can I base on "from="? For example:
Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>, size=4000, nrcpt=1 (queue active)

2018-03-30 17:52 GMT+02:00 chaouche yacine <[hidden email]>:
Absolutely. Amavis comes with a default score of 5.0. Any e-mail which has a 5.0 score or higher is considered spam. You might have false positives though, for example if the user's ISP addresses are blacklisted, which might be the case dependning on the country and ISP.

Yassine.

On Friday, March 30, 2018, 10:44:27 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Yassine, appreciate your answer. I will check further in it but do you think that spam score could help with estimate which mail from which account is or not spam?

2018-03-30 9:27 GMT+02:00 chaouche yacine <[hidden email]>:
Here are some ideas :

1/ Create a directory somewhere in /var/, for example mailstats
2/ The directory will contain one file per sender
3/ Your bash script will parse the mail log file in real time (tail -f) then tee each matching line to the corresponding mailstats/user file, for example if the line is matching [hidden email] it will go to mailstats/bob. That way you will have, for each user, the number of outgoing emails.


Another script will simply wc -l each mailstats user file, that will give you the number of sent mails. You can use fail2ban for this task instead of writing you own script. Fail2ban can be configured to scan logfiles looking for a particular line. It will count the matching lines and if it reaches the (configurable) maximum count in a certain (configurable) amount of time, it will do whatever action you have configured, for example sending you an e-mail.

The mailstats file will need some maintenance, otherwise they will grow infinitely and possibly slow down you scripts. You can use logrotate to archive your mailstats files and create new ones automatically for you after either a specific amount of time or after a specific mail size.

It's not trivial, but it should work.


Yassine.


On Friday, March 30, 2018, 7:16:33 AM GMT+2, Poliman - Serwis <[hidden email]> wrote:


Some emails has "Hits" value even, for example 2,5. What is (if it's possible to say) good value? I am going to create script in bash  which send me an email when from particular email account will outbound for example 300 emails per day. Kind of warning. But I am not sure I could use spam score to it. What do you think guys about it?

2018-03-29 17:58 GMT+02:00 chaouche yacine <[hidden email]>:

It is, that's the spam score. It helps to visualise if a particular mailbox is bombarded with spam (can happen with lots and lots of e-mails from qq.com, I have that domain banned in postfix itself).

Yassine.
On Thursday, March 29, 2018, 3:21:16 PM GMT+1, Alex JOST <[hidden email]> wrote:


Am 29.03.2018 um 15:30 schrieb Poliman - Serwis:

> This one works well. One question based on one from generated lines:
> Mar 26 11:47:41  ORIGINATING LOCAL [127.0.0.1]:38920 <[hidden email]>
> -> <[hidden email]>,<[hidden email]>, Hits: 0.742
>
> Mar 26 11:47:41 --> this is date and hour when mail from
> [hidden email] was sent to [hidden email] and
> [hidden email], am I right?
> What are "Hits: 0.742" ?


Looks like amavisd scoring.

--
Alex JOST




--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
In reply to this post by Wietse Venema
Unfortunately I use Postfix from Ubuntu repos.

2018-04-04 13:08 GMT+02:00 Wietse Venema <[hidden email]>:
Poliman - Serwis:
> Could you tell me I could add e-mails together from mail.log which are in
> line with "from=" part? Hmm I hope I say clear. I need count emails from
> particular mailbox. Can I base on "from="? For example:
> Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<[hidden email]>,
> size=4000, nrcpt=1 (queue active)

The script auxiliary/collate/collate.pl (in the Postfix source-code
distribution) combines records from multiple Postfix daemons into
one transaction (mainly, a group of logfile records with the same
queue ID).

        Wietse



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Scott Kitterman-4
On Thursday, April 05, 2018 07:34:44 AM Poliman - Serwis wrote:
> Unfortunately I use Postfix from Ubuntu repos.

apt-get source postfix
cd postfix-[version] (depends your Ubuntu release)
cd auxiliary/collate
ls

and you'll see both collate.pl and the associated README.

Scott K

> 2018-04-04 13:08 GMT+02:00 Wietse Venema <[hidden email]>:
> > Poliman - Serwis:
> > > Could you tell me I could add e-mails together from mail.log which are
> > > in
> > > line with "from=" part? Hmm I hope I say clear. I need count emails from
> > > particular mailbox. Can I base on "from="? For example:
> > > Apr  3 11:49:48 s1 postfix/qmgr[722]: 3B8C313BE2D: from=<
> >
> > [hidden email]>,
> >
> > > size=4000, nrcpt=1 (queue active)
> >
> > The script auxiliary/collate/collate.pl (in the Postfix source-code
> > distribution) combines records from multiple Postfix daemons into
> > one transaction (mainly, a group of logfile records with the same
> > queue ID).
> >
> >         Wietse

Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Viktor Dukhovni


> On Apr 5, 2018, at 1:39 AM, Scott Kitterman <[hidden email]> wrote:
>
> On Thursday, April 05, 2018 07:34:44 AM Poliman - Serwis wrote:
>> Unfortunately I use Postfix from Ubuntu repos.
>
> apt-get source postfix
> cd postfix-[version] (depends your Ubuntu release)
> cd auxiliary/collate
> ls
>
> and you'll see both collate.pl and the associated README.

Alternatively:

  https://github.com/vdukhovni/postfix/tree/master/postfix/auxiliary/collate

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

2018-04-05 7:57 GMT+02:00 Viktor Dukhovni <[hidden email]>:


> On Apr 5, 2018, at 1:39 AM, Scott Kitterman <[hidden email]> wrote:
>
> On Thursday, April 05, 2018 07:34:44 AM Poliman - Serwis wrote:
>> Unfortunately I use Postfix from Ubuntu repos.
>
> apt-get source postfix
> cd postfix-[version] (depends your Ubuntu release)
> cd auxiliary/collate
> ls
>
> and you'll see both collate.pl and the associated README.

Alternatively:

  https://github.com/vdukhovni/postfix/tree/master/postfix/auxiliary/collate

--
        Viktor.




--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Viktor Dukhovni


> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.

--
        Viktor.

Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

chaouche yacine
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
Yacine, do you say about collate.pl script or "from=" part from log file? I suppose that abotu script. If collate.pl could group by some id, it would be nice, because I would have only one line from log dependent from particular email sent.

2018-04-05 12:31 GMT+02:00 chaouche yacine <[hidden email]>:
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

chaouche yacine

I was talking about collate.pl
On Thursday, April 5, 2018, 12:04:45 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Yacine, do you say about collate.pl script or "from=" part from log file? I suppose that abotu script. If collate.pl could group by some id, it would be nice, because I would have only one line from log dependent from particular email sent.

2018-04-05 12:31 GMT+02:00 chaouche yacine <[hidden email]>:
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

chaouche yacine
You didn't say what's wrong the line grepping on amavis ? it should give you what you want : one line by sender.


On Thursday, April 5, 2018, 1:51:28 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


I used this script and after comparison result generated by collate.pl and mail.log file I think that sending one email gives few lines (generated by collate.pl) which one of them include sender email address, in my case it looks like in "from=<[hidden email]>" and one include line "from=<root>". And this behavior appears that many times as many emails I will send. To be honest I am looking some pattern I could base.

2018-04-05 14:30 GMT+02:00 chaouche yacine <[hidden email]>:

I was talking about collate.pl
On Thursday, April 5, 2018, 12:04:45 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Yacine, do you say about collate.pl script or "from=" part from log file? I suppose that abotu script. If collate.pl could group by some id, it would be nice, because I would have only one line from log dependent from particular email sent.

2018-04-05 12:31 GMT+02:00 chaouche yacine <[hidden email]>:
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
I wasn't able to find text "amavis" in log file. I tried production server and finally I see it and I know what you suggest me. It looks like:
Apr  5 15:11:56 s1 amavis[26789]: (26789-13) Passed CLEAN {RelayedOutbound}, LOCAL [127.0.0.1] <[hidden email]> -> <[hidden email]>

Is it the line about which you said?

2018-04-05 14:53 GMT+02:00 chaouche yacine <[hidden email]>:
You didn't say what's wrong the line grepping on amavis ? it should give you what you want : one line by sender.


On Thursday, April 5, 2018, 1:51:28 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


I used this script and after comparison result generated by collate.pl and mail.log file I think that sending one email gives few lines (generated by collate.pl) which one of them include sender email address, in my case it looks like in "from=<[hidden email]>" and one include line "from=<root>". And this behavior appears that many times as many emails I will send. To be honest I am looking some pattern I could base.

2018-04-05 14:30 GMT+02:00 chaouche yacine <[hidden email]>:

I was talking about collate.pl
On Thursday, April 5, 2018, 12:04:45 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Yacine, do you say about collate.pl script or "from=" part from log file? I suppose that abotu script. If collate.pl could group by some id, it would be nice, because I would have only one line from log dependent from particular email sent.

2018-04-05 12:31 GMT+02:00 chaouche yacine <[hidden email]>:
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

chaouche yacine

Yes, more specifically you should grep on 'Relay' to avoid other amavis lines

root@messagerie[10.10.10.19] ~ # grep amavis /var/log/mail.log | grep -v Relay | head
Apr  1 06:59:29 messagerie-prep amavis[25741]: starting. /usr/sbin/amavisd-new at myhost.mydomain.tld amavisd-new-2.10.1 (20141025), Unicode aware, LC_ALL="C", LANG="en_US.UTF-8"
Apr  1 06:59:29 messagerie-prep amavis[25748]: Net::Server: Group Not Defined.  Defaulting to EGID '116 116'
Apr  1 06:59:29 messagerie-prep amavis[25748]: Net::Server: User Not Defined.  Defaulting to EUID '109'
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Amavis::Conf        2.404
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Archive::Zip        1.39
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module BerkeleyDB          0.54
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Compress::Raw::Zlib 2.065
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Compress::Zlib      2.064
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Crypt::OpenSSL::RSA 0.28
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module DB_File             1.831
root@messagerie[10.10.10.19] ~ #





The only problem is when you have a single mail sent to many recipients, then the log line could be split in two, so you wouldn't have all the recipients in just one line

Apr  5 14:49:26 messagerie-prep amavis[15005]: (15005-12) Passed CLEAN {RelayedInternal}, LOCAL [127.0.0.1]:55954 <[hidden email]> -> <[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<[hidden email]>,<xxx@myd...
Apr  5 14:49:26 messagerie-prep amavis[15005]: (15005-12) ...omain.tld>,<[hidden email]>, Queue-ID: 946FC640066, Message-ID: <[hidden email]>, mail_id: SdFWN26NSt8A, Hits: 0.516, size: 1783, queued_as: D7B1C640068, 299 ms










On Thursday, April 5, 2018, 2:17:23 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


I wasn't able to find text "amavis" in log file. I tried production server and finally I see it and I know what you suggest me. It looks like:
Apr  5 15:11:56 s1 amavis[26789]: (26789-13) Passed CLEAN {RelayedOutbound}, LOCAL [127.0.0.1] <[hidden email]> -> <[hidden email]>

Is it the line about which you said?

2018-04-05 14:53 GMT+02:00 chaouche yacine <[hidden email]>:
You didn't say what's wrong the line grepping on amavis ? it should give you what you want : one line by sender.


On Thursday, April 5, 2018, 1:51:28 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


I used this script and after comparison result generated by collate.pl and mail.log file I think that sending one email gives few lines (generated by collate.pl) which one of them include sender email address, in my case it looks like in "from=<[hidden email]>" and one include line "from=<root>". And this behavior appears that many times as many emails I will send. To be honest I am looking some pattern I could base.

2018-04-05 14:30 GMT+02:00 chaouche yacine <[hidden email]>:

I was talking about collate.pl
On Thursday, April 5, 2018, 12:04:45 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Yacine, do you say about collate.pl script or "from=" part from log file? I suppose that abotu script. If collate.pl could group by some id, it would be nice, because I would have only one line from log dependent from particular email sent.

2018-04-05 12:31 GMT+02:00 chaouche yacine <[hidden email]>:
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
Reply | Threaded
Open this post in threaded view
|

Re: monitoring outgoing emails

Poliman - Serwis
Thank you. I have to get all these message and try to build script which send me an email with specific number of emails send from particular email account.

2018-04-05 16:00 GMT+02:00 chaouche yacine <[hidden email]>:

Yes, more specifically you should grep on 'Relay' to avoid other amavis lines

root@messagerie[10.10.10.19] ~ # grep amavis /var/log/mail.log | grep -v Relay | head
Apr  1 06:59:29 messagerie-prep amavis[25741]: starting. /usr/sbin/amavisd-new at myhost.mydomain.tld amavisd-new-2.10.1 (20141025), Unicode aware, LC_ALL="C", LANG="en_US.UTF-8"
Apr  1 06:59:29 messagerie-prep amavis[25748]: Net::Server: Group Not Defined.  Defaulting to EGID '116 116'
Apr  1 06:59:29 messagerie-prep amavis[25748]: Net::Server: User Not Defined.  Defaulting to EUID '109'
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Amavis::Conf        2.404
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Archive::Zip        1.39
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module BerkeleyDB          0.54
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Compress::Raw::Zlib 2.065
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Compress::Zlib      2.064
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module Crypt::OpenSSL::RSA 0.28
Apr  1 06:59:29 messagerie-prep amavis[25748]: Module DB_File             1.831
root@messagerie[10.10.10.19] ~ #





The only problem is when you have a single mail sent to many recipients, then the log line could be split in two, so you wouldn't have all the recipients in just one line

Apr  5 14:49:26 messagerie-prep amavis[15005]: (15005-12) Passed CLEAN {RelayedInternal}, LOCAL [127.0.0.1]:55954 <[hidden email]> -> <[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@mydomain.tld>,<[hidden email]>,<xxx@mydomain.tld>,<xxx@myd...
Apr  5 14:49:26 messagerie-prep amavis[15005]: (15005-12) ...omain.tld>,<xxx@mydomain.tld>, Queue-ID: 946FC640066, Message-ID: <[hidden email]>, mail_id: SdFWN26NSt8A, Hits: 0.516, size: 1783, queued_as: D7B1C640068, 299 ms










On Thursday, April 5, 2018, 2:17:23 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


I wasn't able to find text "amavis" in log file. I tried production server and finally I see it and I know what you suggest me. It looks like:
Apr  5 15:11:56 s1 amavis[26789]: (26789-13) Passed CLEAN {RelayedOutbound}, LOCAL [127.0.0.1] <[hidden email]> -> <[hidden email]>

Is it the line about which you said?

2018-04-05 14:53 GMT+02:00 chaouche yacine <[hidden email]>:
You didn't say what's wrong the line grepping on amavis ? it should give you what you want : one line by sender.


On Thursday, April 5, 2018, 1:51:28 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


I used this script and after comparison result generated by collate.pl and mail.log file I think that sending one email gives few lines (generated by collate.pl) which one of them include sender email address, in my case it looks like in "from=<[hidden email]>" and one include line "from=<root>". And this behavior appears that many times as many emails I will send. To be honest I am looking some pattern I could base.

2018-04-05 14:30 GMT+02:00 chaouche yacine <[hidden email]>:

I was talking about collate.pl
On Thursday, April 5, 2018, 12:04:45 PM GMT+1, Poliman - Serwis <[hidden email]> wrote:


Yacine, do you say about collate.pl script or "from=" part from log file? I suppose that abotu script. If collate.pl could group by some id, it would be nice, because I would have only one line from log dependent from particular email sent.

2018-04-05 12:31 GMT+02:00 chaouche yacine <[hidden email]>:
No it won't, it will simply group qids together so that you can trace individual e-mails, instead of having intermingled log lines from different e-mails.






On Thursday, April 5, 2018, 7:10:11 AM GMT+1, Viktor Dukhovni <[hidden email]> wrote:




> On Apr 5, 2018, at 2:07 AM, Poliman - Serwis <[hidden email]> wrote:
>
> Using collate.pl script I won't have to count "from=" from mail log, this script merge it, am I right?

Try it and see what you get.  You may need to make some adjustments to the regular expressions
depending on how your syslog formats the output, especially the date.


--
    Viktor.



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha



--
Pozdrawiam / Best Regards
Piotr Bracha
12