Check outgoing emails not using TLS

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Check outgoing emails not using TLS

Dominic Raferd
Using setting 'smtp_tls_security_level = may' (postfix 3.3.0) is there
a reliable way to see from log which outgoing emails were sent in the
clear i.e. *not* using TLS?
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Viktor Dukhovni
On Mon, Apr 06, 2020 at 08:21:32AM +0100, Dominic Raferd wrote:

> Using setting 'smtp_tls_security_level = may' (postfix 3.3.0) is there
> a reliable way to see from log which outgoing emails were sent in the
> clear i.e. *not* using TLS?

Yes, provided you don't lose too many log messages[1], and your logging
subsystem does not reorder them[1], set:

    smtp_tls_loglevel = 1

and use "collate":

    https://github.com/vdukhovni/postfix/tree/master/postfix/auxiliary/collate

whose output you'd send to the attached Perl script.  On my system for
example:

    # bzip2 -dcf $(ls -tr /var/log/maillog*) | perl collate | perl tlstype.pl

--
    Viktor.

[1] If your system is suffering under the yoke of systemd-journald, you
should strongly consider enabling the built-in logging in recent
versions of Postfix to bypass systemd's broken logging subsystem.

    - It is single-threaded, performs poorly on multi-cpu servers and
      may not be able to keep up with all the messages generated on a
      busy multi-cpu system.

    - By default has low message rate limits, dropping messages
      that exceed the limits.

    - Listens on stream socket rather than a dgram socket, which
      breaks message ordering from multi-process systems like
      Postfix.

tlstype.pl (911 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Dominic Raferd
On Mon, 6 Apr 2020 at 09:44, Viktor Dukhovni <[hidden email]> wrote:

>
> On Mon, Apr 06, 2020 at 08:21:32AM +0100, Dominic Raferd wrote:
>
> > Using setting 'smtp_tls_security_level = may' (postfix 3.3.0) is there
> > a reliable way to see from log which outgoing emails were sent in the
> > clear i.e. *not* using TLS?
>
> Yes, provided you don't lose too many log messages[1], and your logging
> subsystem does not reorder them[1], set:
>
>     smtp_tls_loglevel = 1
>
> and use "collate":
>
>     https://github.com/vdukhovni/postfix/tree/master/postfix/auxiliary/collate
>
> whose output you'd send to the attached Perl script.  On my system for
> example:
>
>     # bzip2 -dcf $(ls -tr /var/log/maillog*) | perl collate | perl tlstype.pl
>
> --
>     Viktor.
>
> [1] If your system is suffering under the yoke of systemd-journald, you
> should strongly consider enabling the built-in logging in recent
> versions of Postfix to bypass systemd's broken logging subsystem.
>
>     - It is single-threaded, performs poorly on multi-cpu servers and
>       may not be able to keep up with all the messages generated on a
>       busy multi-cpu system.
>
>     - By default has low message rate limits, dropping messages
>       that exceed the limits.
>
>     - Listens on stream socket rather than a dgram socket, which
>       breaks message ordering from multi-process systems like
>       Postfix.

Thanks Viktor, collate.pl and tlstype.pl work perfectly for this task,
and show that there are very few such messages (and hardly any that
are not automated replies to low-quality incoming mails).

I will keep in mind what you say about systemd and logging. The only
'systemd:.*suppress' messages I see are very occasionally in the
system log and don't relate to any mail activity, but our mail servers
are not heavily loaded. Our mail log file, where postfix messages end
up, is also a destination for logs from other mail-related services,
all handled by (systemd via) rsyslog, and I guess if postfix was doing
direct logging it should have exclusive rights to a log file?
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Wietse Venema
Dominic Raferd:
> I will keep in mind what you say about systemd and logging. The only
> 'systemd:.*suppress' messages I see are very occasionally in the
> system log and don't relate to any mail activity, but our mail servers
> are not heavily loaded. Our mail log file, where postfix messages end
> up, is also a destination for logs from other mail-related services,
> all handled by (systemd via) rsyslog, and I guess if postfix was doing
> direct logging it should have exclusive rights to a log file?

Postfix direct logging feature MUST NOT share files other programs.
You manage logs with the "postfix logrotate" command, usually from
a cronjob.

For more informantion, see http://www.postfix.org/MAILLOG_README.html
(Postfix logging to file or stdout).

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Dominic Raferd
On Mon, 6 Apr 2020 at 21:54, Wietse Venema <[hidden email]> wrote:

>
> Dominic Raferd:
> > I will keep in mind what you say about systemd and logging. The only
> > 'systemd:.*suppress' messages I see are very occasionally in the
> > system log and don't relate to any mail activity, but our mail servers
> > are not heavily loaded. Our mail log file, where postfix messages end
> > up, is also a destination for logs from other mail-related services,
> > all handled by (systemd via) rsyslog, and I guess if postfix was doing
> > direct logging it should have exclusive rights to a log file?
>
> Postfix direct logging feature MUST NOT share files other programs.
> You manage logs with the "postfix logrotate" command, usually from
> a cronjob.
>
> For more information, see http://www.postfix.org/MAILLOG_README.html
> (Postfix logging to file or stdout).

Thanks Wietse for the clarification
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Viktor Dukhovni
In reply to this post by Dominic Raferd
On Mon, Apr 06, 2020 at 02:53:25PM +0100, Dominic Raferd wrote:

> > whose output you'd send to the attached Perl script.  On my system for
> > example:
> >
> >     # bzip2 -dcf $(ls -tr /var/log/maillog*) | perl collate | perl tlstype.pl

I should perhaps mention that the "tlstype.pl" Perl script does not
handle TLS connection re-use.  I've not looked at what it would take
to do that.

We should perhaps consider logging some indication of TLS in the
core delivery summary line:

    postfix/smtp: <qid>: to=<...>,[ orig_to=<...>,] relay=...,
    [ tls=<level>:(Anonymous|Untrusted|Verified),]

that is, perhaps just the security level and verification status?

Collating the data from the logs is tricky, and likely more so with
connection reuse (but perhaps not too bad, exercise for the reader...).

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Viktor Dukhovni
In reply to this post by Dominic Raferd
On Mon, Apr 06, 2020 at 02:53:25PM +0100, Dominic Raferd wrote:

> Our mail log file, where postfix messages end
> up, is also a destination for logs from other mail-related services,
> all handled by (systemd via) rsyslog...

More accurately, handled by rsyslog (via systemd).  It is the
(mis)handling by systemd that I'm warning about.  While the overall
goals of systemd are generally laudable, the implementation does not
always merit praise.

In particular, the performance of the logging sub-system has been
substantially degraded, which combined with an increased risk of
reordering messages, means that applications now need to be prepared to
bypass the resulting mess, and manage their own logging. :-(

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Michael Storz
In reply to this post by Viktor Dukhovni
Am 2020-04-06 23:53, schrieb Viktor Dukhovni:

> On Mon, Apr 06, 2020 at 02:53:25PM +0100, Dominic Raferd wrote:
>
>> > whose output you'd send to the attached Perl script.  On my system for
>> > example:
>> >
>> >     # bzip2 -dcf $(ls -tr /var/log/maillog*) | perl collate | perl tlstype.pl
>
> I should perhaps mention that the "tlstype.pl" Perl script does not
> handle TLS connection re-use.  I've not looked at what it would take
> to do that.

And it does not work for mixed-case hostnames:

- TLS connection established to lower-case-hostname
- relay=mixed-case-hostname

>
> We should perhaps consider logging some indication of TLS in the
> core delivery summary line:
>
>     postfix/smtp: <qid>: to=<...>,[ orig_to=<...>,] relay=...,
>     [ tls=<level>:(Anonymous|Untrusted|Verified),]
>
> that is, perhaps just the security level and verification status?
>
> Collating the data from the logs is tricky, and likely more so with
> connection reuse (but perhaps not too bad, exercise for the reader...).

Regards,
Michael
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Viktor Dukhovni
On Tue, Apr 07, 2020 at 11:46:33AM +0200, Michael Storz wrote:

> > I should perhaps mention that the "tlstype.pl" Perl script does not
> > handle TLS connection re-use.  I've not looked at what it would take
> > to do that.
>
> And it does not work for mixed-case hostnames:
>
> - TLS connection established to lower-case-hostname
> - relay=mixed-case-hostname

Ah, thanks. Easily enough corrected, by wrapping Perl values in
lc($value).  If you fix this and more issues, feel free to put it up on
github somewhere...  I am not planning to become a "maintainer" of this
off-the-cuff script.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Wietse Venema
Viktor Dukhovni:

> On Tue, Apr 07, 2020 at 11:46:33AM +0200, Michael Storz wrote:
>
> > > I should perhaps mention that the "tlstype.pl" Perl script does not
> > > handle TLS connection re-use.  I've not looked at what it would take
> > > to do that.
> >
> > And it does not work for mixed-case hostnames:
> >
> > - TLS connection established to lower-case-hostname
> > - relay=mixed-case-hostname
>
> Ah, thanks. Easily enough corrected, by wrapping Perl values in
> lc($value).  If you fix this and more issues, feel free to put it up on
> github somewhere...  I am not planning to become a "maintainer" of this
> off-the-cuff script.

Also I'd be happy to bundle an uipdated version under $postfix/auxiliary.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Wietse Venema
In reply to this post by Viktor Dukhovni
Viktor Dukhovni:

> On Tue, Apr 07, 2020 at 11:46:33AM +0200, Michael Storz wrote:
>
> > > I should perhaps mention that the "tlstype.pl" Perl script does not
> > > handle TLS connection re-use.  I've not looked at what it would take
> > > to do that.
> >
> > And it does not work for mixed-case hostnames:
> >
> > - TLS connection established to lower-case-hostname
> > - relay=mixed-case-hostname
>
> Ah, thanks. Easily enough corrected, by wrapping Perl values in
> lc($value).  If you fix this and more issues, feel free to put it up on
> github somewhere...  I am not planning to become a "maintainer" of this
> off-the-cuff script.
Attached are an updated script, and a diff.

        Wietse

tlstype.pl (935 bytes) Download Attachment
tlstype.pl.diff (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Check outgoing emails not using TLS

Viktor Dukhovni
On Tue, Apr 07, 2020 at 07:06:41PM -0400, Wietse Venema wrote:

> Attached are an updated script, and a diff.

Looks good to me.

--
    Viktor.