It is possible for Postfix logging to bypass journald?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

It is possible for Postfix logging to bypass journald?

Curtis-20
We recently switched our Postfix mail servers to Ubuntu Server 18, which
uses journald for logging. Since we have monitoring systems that parse
/var/log/maillog, we enabled rsyslog with imuxsock so we still can parse
the log like we did before journald.  But, it's unreliable.

Our monitoring systems are reporting failed deliveries of messages
because of missing log lines in /var/log/maillog.  When using journalctl
to query the journal, the missing lines can be found, but these queries
are too CPU intensive.

We also see that journald is occasionally logging messages such as this:

Jan 08 20:55:16 host123 systemd-journald[11136]: Forwarding to syslog
missed 2 messages.

Since this message doesn't provide any information as to why the
messages were missed, I have to wonder if it's related to this warning
message on the rsyslog site:

"Note: It must be noted, however, that the journal tends to drop
messages when it becomes busy instead of forwarding them to the system
log socket. This is because the journal uses an async log socket
interface for forwarding instead of the traditional synchronous one."

See:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#imuxsock-systemd-details-label

I'm aware we could switch to using imjournal, which might solve the
issue since it reads the journal directly (which does seem to contain
the missing messages), but I have to imagine that it would come at a
very high CPU cost.

See:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html

So, I'm trying to figure out if it would be possible to get Postfix to
use an alternate logging mechanism that would completely bypass journald
so that we can have reliable loggging in a manner that is less CPU
intensive than journald/imjournal.

Ideas?

Thanks,

Curtis
Reply | Threaded
Open this post in threaded view
|

Re: It is possible for Postfix logging to bypass journald?

Wietse Venema
Curtis:
> We recently switched our Postfix mail servers to Ubuntu Server 18, which
> uses journald for logging. Since we have monitoring systems that parse
> /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse
> the log like we did before journald.  But, it's unreliable.

I recall that system-effing-d has a rare-limiting feature that very
helpfully drops Postfix logging.

Here's one search result with suggestions for systemd.
https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/

Another search result: systemd and rsyslog both have rate limits.
https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disable-rsyslog-rate-limiting

It is time to update the Postfix page on LINUX logging brain damage.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: It is possible for Postfix logging to bypass journald?

Robert L Mathews
In reply to this post by Curtis-20
On 1/9/19 4:05 PM, Curtis wrote:
> We recently switched our Postfix mail servers to Ubuntu Server 18, which
> uses journald for logging. Since we have monitoring systems that parse
> /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse
> the log like we did before journald.  But, it's unreliable.
>
> Our monitoring systems are reporting failed deliveries of messages
> because of missing log lines in /var/log/maillog.

We had this problem. It was fixed by putting this in
/etc/systemd/journald.conf:

# allow for busy mail logs; allows 1000 per second
RateLimitInterval=5s
RateLimitBurst=5000

And/or by putting this into /etc/rsyslog.conf:

$SystemLogRateLimitInterval 0

(The latter is supposedly no longer necessary, but it used to be, and
does not appear to be harmful.)

--
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
Reply | Threaded
Open this post in threaded view
|

Re: It is possible for Postfix logging to bypass journald?

Matus UHLAR - fantomas
In reply to this post by Wietse Venema
>Curtis:
>> We recently switched our Postfix mail servers to Ubuntu Server 18, which
>> uses journald for logging. Since we have monitoring systems that parse
>> /var/log/maillog, we enabled rsyslog with imuxsock so we still can parse
>> the log like we did before journald.  But, it's unreliable.

On 09.01.19 19:38, Wietse Venema wrote:

>I recall that system-effing-d has a rare-limiting feature that very
>helpfully drops Postfix logging.
>
>Here's one search result with suggestions for systemd.
>https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/
>
>Another search result: systemd and rsyslog both have rate limits.
>https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disable-rsyslog-rate-limiting
>
>It is time to update the Postfix page on LINUX logging brain damage.

oh, please... systemd and rsyslog. I use sysvinit+syslog-ng wherever
possible, on linux
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
The 3 biggets disasters: Hiroshima 45, Tschernobyl 86, Windows 95