Realtime log reporting when postfix delivers mails

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Realtime log reporting when postfix delivers mails

Ramprasad-5
I have a postfix server sending out mails and we are creating reports by
parsing the maillogs using a couple of perl cron scripts
(linux machine with mysql )

Now the requirement is of realtime reporting.
I tried using rsyslog with a mysql table. But the performance is far too
bad. Rsyslog seems to have some memory leak and it brings down the machine.

I guess realtime logging should be a very common requirement. What is
the best way for this


Thanks
Ram





Reply | Threaded
Open this post in threaded view
|

Re: Realtime log reporting when postfix delivers mails

Stan Hoeppner
On 3/15/2013 1:59 AM, Ram wrote:

> I have a postfix server sending out mails and we are creating reports by
> parsing the maillogs using a couple of perl cron scripts
> (linux machine with mysql )
>
> Now the requirement is of realtime reporting.
>
> I tried using rsyslog with a mysql table. But the performance is far too
> bad. Rsyslog seems to have some memory leak and it brings down the machine.
>
> I guess realtime logging should be a very common requirement.

Actually it's not.

> What is
> the best way for this

Postfix logs to the UNIX syslog facility.  What you do with it from
there is outside the scope of Postfix.

What you should probably be looking for is a generic log watching daemon
that can capture appends on the fly.

--
Stan

Reply | Threaded
Open this post in threaded view
|

Re: Realtime log reporting when postfix delivers mails

Robert Schetterer-2
In reply to this post by Ramprasad-5
Am 15.03.2013 07:59, schrieb Ram:

> I have a postfix server sending out mails and we are creating reports by
> parsing the maillogs using a couple of perl cron scripts
> (linux machine with mysql )
>
> Now the requirement is of realtime reporting.
> I tried using rsyslog with a mysql table. But the performance is far too
> bad. Rsyslog seems to have some memory leak and it brings down the machine.
>
> I guess realtime logging should be a very common requirement. What is
> the best way for this
>
>
> Thanks
> Ram
>
>
>
>
>

real time reporting is easy done with over ssh and tail -f /var/log/mail.log

without ssh you may use webmin over https, then create some unprivileged
webmin user , do local login in the webmin gui, give the user enough
permission to read tail -f /var/log/mail.log

you may also use webmins inbuild read syslog log stuff,
or write some own scripts for grep/tail etc

there are some other solutions with http guis
but as i dont like them after testing, but it would be nice to here some
more

xymon has limited stuff to read log files, but it may enough to create
alarms to special events in mail log, guess nagios etc have equal

you may also try some other syslog compatible servers with guis
for real time search and display







Best Regards
MfG Robert Schetterer

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich
Reply | Threaded
Open this post in threaded view
|

Re: Realtime log reporting when postfix delivers mails

Abhijeet Rastogi
In reply to this post by Ramprasad-5
Have a look into logstash project. I have just started using it for mail logs and it's awesome.

On Fri, Mar 15, 2013 at 12:29 PM, Ram <[hidden email]> wrote:
I have a postfix server sending out mails and we are creating reports by parsing the maillogs using a couple of perl cron scripts
(linux machine with mysql )

Now the requirement is of realtime reporting.
I tried using rsyslog with a mysql table. But the performance is far too bad. Rsyslog seems to have some memory leak and it brings down the machine.

I guess realtime logging should be a very common requirement. What is the best way for this


Thanks
Ram








--
Regards,
Abhijeet Rastogi (shadyabhi)
https://plus.google.com/107316377741966576356/
Reply | Threaded
Open this post in threaded view
|

Re: Realtime log reporting when postfix delivers mails

Reinaldo Gil Lima de Carvalho
In reply to this post by Ramprasad-5
We need a structured log to avoid parsing. I talk with Wietse in the year 2011 at FISL conference (Porto Alegre/Brasil).

The second problem is load this data to a database. Rsyslog put the data in a single column, and use full text search is inevitable.

While don't have a better solution, I wrote a daemon to parse and insert the data on a database. I will search this code and put on github on next week.

[]'s

Reinaldo Gil Lima de Carvalho

Em 15/03/2013, às 03:59, Ram <[hidden email]> escreveu:

> I have a postfix server sending out mails and we are creating reports by parsing the maillogs using a couple of perl cron scripts
> (linux machine with mysql )
>
> Now the requirement is of realtime reporting.
> I tried using rsyslog with a mysql table. But the performance is far too bad. Rsyslog seems to have some memory leak and it brings down the machine.
>
> I guess realtime logging should be a very common requirement. What is the best way for this
>
>
> Thanks
> Ram
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Realtime log reporting when postfix delivers mails

Florian Schaal
Am 16.03.2013 22:11, schrieb Reinaldo Gil Lima de Carvalho:
> We need a structured log to avoid parsing. I talk with Wietse in the year 2011 at FISL conference (Porto Alegre/Brasil).
>
> The second problem is load this data to a database. Rsyslog put the data in a single column, and use full text search is inevitable.
>
> While don't have a better solution, I wrote a daemon to parse and insert the data on a database. I will search this code and put on github on next week.
>
> []'s
>

I use something similar for apache with syslog-ng. It can adapted for
each message-string: http://blog.schaal-24.de/?p=769&lang=en

Using syslog-ng you can easily split each log-message in different parts
and than store each part of the message in a different column. Maybe
this is possible with ryslog, too.

regards
Florian