(no subject)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

(no subject)

Greg Strange
Hi,

I am trying to track a single email throughout the entire postfix process. The idea is that when a customer calls us and says that a certain email never reached them, we can quickly trace the email through the logs and see that it died due to RBL, virus threshold, etc.

Ideally, I'd like to be able to get or set a unique message ID and then be able to match that ID in the logfiles to see what the outcome of a specific email was. Is there a way to trace a single email through everything postfix does to it?

TIA.
Reply | Threaded
Open this post in threaded view
|

Re: tracking through logs

/dev/rob0
[Subject: added]

On Mon, Jun 04, 2018 at 03:34:33PM -0500, Greg Strange wrote:

> I am trying to track a single email throughout the entire postfix
> process. The idea is that when a customer calls us and says that a
> certain email never reached them, we can quickly trace the email
> through the logs and see that it died due to RBL, virus threshold,
> etc.
>
> Ideally, I'd like to be able to get or set a unique message ID and
> then be able to match that ID in the logfiles to see what the
> outcome of a specific email was. Is there a way to trace a single
> email through everything postfix does to it?

Well, you want this:

enable_long_queue_ids = yes

... but that won't help in one of the cases you mentioned, that of
DNSBL blocking.  Also, the customer won't know the queue ID, but it
will be found in headers, unless of course it was blocked prior to
DATA, in which case there's no queue ID.
--
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Reply | Threaded
Open this post in threaded view
|

Re:

@lbutlr
In reply to this post by Greg Strange
On 4 Jun 2018, at 14:34, Greg Strange <[hidden email]> wrote:
> I am trying to track a single email throughout the entire postfix process. The idea is that when a customer calls us and says that a certain email never reached them, we can quickly trace the email through the logs and see that it died due to RBL, virus threshold, etc.

I send a daily mail report to the very few users who are super paranoid about missing mail. Each day they get a report that shows all the messages that were rejected, and all the messages that were accepted. the accepted email has the QueueID in the table, and the rejected emails show the IP and helo name that was rejected.

It's not very efficient, but it's good enough for the handful of those who want it.



Reply | Threaded
Open this post in threaded view
|

Re: Tracing single email through postfix

Jeff Abrahamson
In reply to this post by Greg Strange
On 04/06/18 22:34, Greg Strange wrote:

> Hi,
>
> I am trying to track a single email throughout the entire postfix
> process. The idea is that when a customer calls us and says that a
> certain email never reached them, we can quickly trace the email
> through the logs and see that it died due to RBL, virus threshold, etc.
>
> Ideally, I'd like to be able to get or set a unique message ID and
> then be able to match that ID in the logfiles to see what the outcome
> of a specific email was. Is there a way to trace a single email
> through everything postfix does to it?

We want to do the same thing.  The current candidate is to use logwatch
to gather the information and forward to a database where we post-process.

The project hasn't quite made it high enough on the priority list to get
implemented yet.  If you do something, please do report back to the list
how it works out.

(In passing, email message id's may be considered to contain PII. 
IANAL, just a techy heavily bombarded by GDPR mails of late.)

--

Jeff Abrahamson
+33 6 24 40 01 57
+44 7920 594 255

http://p27.eu/jeff/

Reply | Threaded
Open this post in threaded view
|

Re: your mail

Matus UHLAR - fantomas
In reply to this post by Greg Strange
On 04.06.18 15:34, Greg Strange wrote:
>I am trying to track a single email throughout the entire postfix process.
>The idea is that when a customer calls us and says that a certain email
>never reached them, we can quickly trace the email through the logs and see
>that it died due to RBL, virus threshold, etc.
>
>Ideally, I'd like to be able to get or set a unique message ID and then be
>able to match that ID in the logfiles to see what the outcome of a specific
>email was. Is there a way to trace a single email through everything
>postfix does to it?

in postyfix queue each mail does have its unique ID. However, when pushed
through any kind of content filter, the ID changes.
Also, when mail gets forwarded, the ID changes.

you apparently need interface to parse these logs to find even related queue
IDs.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Support bacteria - they're the only culture some people have.
Reply | Threaded
Open this post in threaded view
|

Re: your mail

@lbutlr
On 5 Jun 2018, at 02:22, Matus UHLAR - fantomas <[hidden email]> wrote:
> in postyfix queue each mail does have its unique ID. However, when pushed
> through any kind of content filter, the ID changes.
> Also, when mail gets forwarded, the ID changes.


A new ID will be ADDED, but the original one remains in the headers, at least for filters.

Received: by mail.covisp.net (Postfix, from userid 58)
        id 410Ptl2kC1zbRcb; Tue,  5 Jun 2018 02:22:39 -0600 (MDT)
Received: from russian-caravan.cloud9.net (russian-caravan.cloud9.net [168.100.1.4])
        (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by mail.covisp.net (Postfix) with ESMTPS id 410Ptj3CtMzbRbb
        for <[hidden email]>; Tue,  5 Jun 2018 02:22:37 -0600 (MDT)

The top header is after spamd with a new ID and the bottom shows the initial ID after it was received by my mail server.

It's not an issue to get the 'right' ID from the header, depending on which ID you consider the right one.

--
The hippo of recollection stirred in the muddy waters of the mind.

Reply | Threaded
Open this post in threaded view
|

Re: your mail

Matus UHLAR - fantomas
>On 5 Jun 2018, at 02:22, Matus UHLAR - fantomas <[hidden email]> wrote:
>> in postyfix queue each mail does have its unique ID. However, when pushed
>> through any kind of content filter, the ID changes.
>> Also, when mail gets forwarded, the ID changes.

On 05.06.18 02:30, @lbutlr wrote:
>A new ID will be ADDED, but the original one remains in the headers, at least for filters.

No, the new mail with new id is created and the old one is marked delivered.

This applies for content_filter (and also for smtp_proxy I believe, although
never tried it).

It can be avoided by using milters, however users will blame postmaster when
processing of mail takes long (which does for spam and virus filters).

>The top header is after spamd with a new ID and the bottom shows the initial ID after it was received by my mail server.
>
>It's not an issue to get the 'right' ID from the header, depending on which ID you consider the right one.

the request was for searching in mail logs, not headers.
It's quite hard to search in headers of mail when you don't know if it was
received at all.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
Reply | Threaded
Open this post in threaded view
|

Re: Tracing single email through postfix

Wietse Venema
In reply to this post by Jeff Abrahamson
Jeff Abrahamson:

> On 04/06/18 22:34, Greg Strange wrote:
> > Hi,
> >
> > I am trying to track a single email throughout the entire postfix
> > process. The idea is that when a customer calls us and says that a
> > certain email never reached them, we can quickly trace the email
> > through the logs and see that it died due to RBL, virus threshold, etc.
> >
> > Ideally, I'd like to be able to get or set a unique message ID and
> > then be able to match that ID in the logfiles to see what the outcome
> > of a specific email was. Is there a way to trace a single email
> > through everything postfix does to it?
>
> We want to do the same thing.? The current candidate is to use logwatch
> to gather the information and forward to a database where we post-process.
>
> The project hasn't quite made it high enough on the priority list to get
> implemented yet.? If you do something, please do report back to the list
> how it works out.
>
> (In passing, email message id's may be considered to contain PII.?
> IANAL, just a techy heavily bombarded by GDPR mails of late.)

This could build on the "collate" tool, which groups Postfix log
records one "session" at a time based on queue ID and process ID
information.

This tool ships with Postfix source code as:

auxiliary/collate/collate.pl
auxiliary/collate/README

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Mailtracking

Michael Reck
In reply to this post by Greg Strange
Hi,

Years ago, one user on this List talked about a tool he (co-) wrote.
AFAIR,it was for United Nation Helpdesk:

https://sourceforge.net/p/x-itools/wiki/Home/

Hth
Michael

Am 04.06.2018 um 22:34 schrieb Greg Strange:

> Hi,
>
> I am trying to track a single email throughout the entire postfix process. The
> idea is that when a customer calls us and says that a certain email never
> reached them, we can quickly trace the email through the logs and see that it
> died due to RBL, virus threshold, etc.
>
> Ideally, I'd like to be able to get or set a unique message ID and then be
> able to match that ID in the logfiles to see what the outcome of a specific
> email was. Is there a way to trace a single email through everything postfix
> does to it?
>
> TIA.