capture information for internal generated mails

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

capture information for internal generated mails

d tbsky
hi:
   I want to bcc all mails for archive purpose. one kind of mail is like below:

   outside user ([hidden email]) mail to  -> postfix alias with settings
to forward outside ([hidden email]) ->  forward to outside user
([hidden email])

   "always_bcc" and "recipient_bcc_maps" won't capture mails to "[hidden email]".

   "sender_bcc_maps" could capture the mail, but the whole mail didn't
have any header information about "[hidden email]".

   can I write "sender_bcc_maps" to reveal the information about "[hidden email]"?
   or there are other method I can use to record the mail and envelope
information?
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Dominic Raferd
On Thu, 20 Dec 2018 at 09:22, d tbsky <[hidden email]> wrote:
hi:
   I want to bcc all mails for archive purpose. one kind of mail is like below:

   outside user ([hidden email]) mail to  -> postfix alias with settings
to forward outside ([hidden email]) ->  forward to outside user
([hidden email])

   "always_bcc" and "recipient_bcc_maps" won't capture mails to "[hidden email]".

   "sender_bcc_maps" could capture the mail, but the whole mail didn't
have any header information about "[hidden email]".

   can I write "sender_bcc_maps" to reveal the information about "[hidden email]"?
   or there are other method I can use to record the mail and envelope
information?

In the scenario you describe I believe always_bcc should capture the email: i.e. where email originates from outside and is relayed to outside. It works for me.
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
Dominic Raferd <[hidden email]>

>
> On Thu, 20 Dec 2018 at 09:22, d tbsky <[hidden email]> wrote:
>>
>> hi:
>>    I want to bcc all mails for archive purpose. one kind of mail is like below:
>>
>>    outside user ([hidden email]) mail to  -> postfix alias with settings
>> to forward outside ([hidden email]) ->  forward to outside user
>> ([hidden email])
>>
>>    "always_bcc" and "recipient_bcc_maps" won't capture mails to "[hidden email]".
>>
>>    "sender_bcc_maps" could capture the mail, but the whole mail didn't
>> have any header information about "[hidden email]".
>>
>>    can I write "sender_bcc_maps" to reveal the information about "[hidden email]"?
>>    or there are other method I can use to record the mail and envelope
>> information?
>
>
> In the scenario you describe I believe always_bcc should capture the email: i.e. where email originates from outside and is relayed to outside. It works for me.

"always_bcc" will only capture mail from [hidden email] ->
[hidden email], but not [hidden email] -> [hidden email].
although it is the same mail.
but seems "always_bcc" didn't capture it, it won't know the
information about [hidden email] and reveal it at email header or other
place I think.
"sender_bcc_maps" will capture from [hidden email] ->
[hidden email], but I don't know how to let "sender_bcc_maps" reveal
"[hidden email]" in header or some other method to me.
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Dominic Raferd


On Thu, 20 Dec 2018 at 11:19, d tbsky <[hidden email]> wrote:
Dominic Raferd <[hidden email]>
>
> On Thu, 20 Dec 2018 at 09:22, d tbsky <[hidden email]> wrote:
>>
>> hi:
>>    I want to bcc all mails for archive purpose. one kind of mail is like below:
>>
>>    outside user ([hidden email]) mail to  -> postfix alias with settings
>> to forward outside ([hidden email]) ->  forward to outside user
>> ([hidden email])
>>
>>    "always_bcc" and "recipient_bcc_maps" won't capture mails to "[hidden email]".
>>
>>    "sender_bcc_maps" could capture the mail, but the whole mail didn't
>> have any header information about "[hidden email]".
>>
>>    can I write "sender_bcc_maps" to reveal the information about "[hidden email]"?
>>    or there are other method I can use to record the mail and envelope
>> information?
>
>
> In the scenario you describe I believe always_bcc should capture the email: i.e. where email originates from outside and is relayed to outside. It works for me.

"always_bcc" will only capture mail from [hidden email] ->
[hidden email], but not [hidden email] -> [hidden email].
although it is the same mail.
but seems "always_bcc" didn't capture it, it won't know the
information about [hidden email] and reveal it at email header or other
place I think.
"sender_bcc_maps" will capture from [hidden email] ->
[hidden email], but I don't know how to let "sender_bcc_maps" reveal
"[hidden email]" in header or some other method to me.

The incoming email is saved by always_bcc, why is it important to save it again when it is relayed (still I presume with the same 'To:' header, but different envelope recipient) to gsmtp? You can find some information about the relay transaction in the mail log (smtp). Example:

2018-12-20 11:27:56 streamingbats postfix/smtp[25382]: 3241861963: to=<[hidden email]>, orig_to=<[hidden email]>, relay=gmail-smtp-in.l.google.com[108.177.15.26]:25, delay=0.86, delays=0.06/0.02/0.26/0.52, dsn=2.0.0, status=sent (250 2.0.0 OK 1545305276 i15si6765428wrp.144 - gsmtp)


Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
Dominic Raferd <[hidden email]>
> The incoming email is saved by always_bcc, why is it important to save it again when it is relayed (still I presume with the same 'To:' header, but different envelope recipient) to gsmtp? You can find some information about the relay transaction in the mail log (smtp). Example:
>
> 2018-12-20 11:27:56 streamingbats postfix/smtp[25382]: 3241861963: to=<[hidden email]>, orig_to=<[hidden email]>, relay=gmail-smtp-in.l.google.com[108.177.15.26]:25, delay=0.86, delays=0.06/0.02/0.26/0.52, dsn=2.0.0, status=sent (250 2.0.0 OK 1545305276 i15si6765428wrp.144 - gsmtp)


I understand the information is in the log. but I need to archive this
information for auditing in the future. so I need this information
when postfix bcc the mail.
with other kind of received mails, I can use bcc_recipient_maps and
rewrite the envelope to archive email address. but I don't know how to
handle mails send out.
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Bastian Blank-3
On Thu, Dec 20, 2018 at 08:02:12PM +0800, d tbsky wrote:
> I understand the information is in the log. but I need to archive this
> information for auditing in the future. so I need this information
> when postfix bcc the mail.
> with other kind of received mails, I can use bcc_recipient_maps and
> rewrite the envelope to archive email address. but I don't know how to
> handle mails send out.

Please be clear what you want.  A mail after bcc shows the content, it
does not show the recipients or the sender.

Bastian

--
It is necessary to have purpose.
                -- Alice #1, "I, Mudd", stardate 4513.3
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Wietse Venema
In reply to this post by d tbsky
d tbsky:
> hi:
>    I want to bcc all mails for archive purpose. one kind of mail is like below:
>
>    outside user ([hidden email]) mail to  -> postfix alias with settings
> to forward outside ([hidden email]) ->  forward to outside user
> ([hidden email])
>
>    "always_bcc" and "recipient_bcc_maps" won't capture mails to "[hidden email]".

... and sender_bcc_maps add the BCC recipient when email ARRIVES
(i.e. input) not when mail is DELIVERED (i.e. output).

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Wietse Venema
In reply to this post by d tbsky
d tbsky:
> Dominic Raferd <[hidden email]>
> > The incoming email is saved by always_bcc, why is it important to save it again when it is relayed (still I presume with the same 'To:' header, but different envelope recipient) to gsmtp? You can find some information about the relay transaction in the mail log (smtp). Example:
> >
> > 2018-12-20 11:27:56 streamingbats postfix/smtp[25382]: 3241861963: to=<[hidden email]>, orig_to=<[hidden email]>, relay=gmail-smtp-in.l.google.com[108.177.15.26]:25, delay=0.86, delays=0.06/0.02/0.26/0.52, dsn=2.0.0, status=sent (250 2.0.0 OK 1545305276 i15si6765428wrp.144 - gsmtp)
>
>
> I understand the information is in the log. but I need to archive this
> information for auditing in the future. so I need this information
> when postfix bcc the mail.

FYI, none of the Postfix built-in BCC features archives the envelope
recipient.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
Wietse Venema <[hidden email]>

>
> d tbsky:
> > Dominic Raferd <[hidden email]>
> > > The incoming email is saved by always_bcc, why is it important to save it again when it is relayed (still I presume with the same 'To:' header, but different envelope recipient) to gsmtp? You can find some information about the relay transaction in the mail log (smtp). Example:
> > >
> > > 2018-12-20 11:27:56 streamingbats postfix/smtp[25382]: 3241861963: to=<[hidden email]>, orig_to=<[hidden email]>, relay=gmail-smtp-in.l.google.com[108.177.15.26]:25, delay=0.86, delays=0.06/0.02/0.26/0.52, dsn=2.0.0, status=sent (250 2.0.0 OK 1545305276 i15si6765428wrp.144 - gsmtp)
> >
> >
> > I understand the information is in the log. but I need to archive this
> > information for auditing in the future. so I need this information
> > when postfix bcc the mail.
>
> FYI, none of the Postfix built-in BCC features archives the envelope
> recipient.
>
>         Wietse

  although built-in bcc won't give envelope, there seems some tricky
ways to do. for example:
recipient_delimiter = +
recipient_bcc_maps = pcre:/etc/postfix/bcc_map.cf

and bcc_map.cf:
/^(.*)@(.*)$/      archive+$1=$2@localhost

then I can get the envelope for received mails.
but I can not find the tricky way for sender_bcc_maps.
if I need to write some script to handle this, any suggestion where to hook?
thanks a lot for help!!
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Matus UHLAR - fantomas
>Wietse Venema <[hidden email]>
>>
>> d tbsky:
>> > Dominic Raferd <[hidden email]>
>> > > The incoming email is saved by always_bcc, why is it important to save it again when it is relayed (still I presume with the same 'To:' header, but different envelope recipient) to gsmtp? You can find some information about the relay transaction in the mail log (smtp). Example:
>> > >
>> > > 2018-12-20 11:27:56 streamingbats postfix/smtp[25382]: 3241861963: to=<[hidden email]>, orig_to=<[hidden email]>, relay=gmail-smtp-in.l.google.com[108.177.15.26]:25, delay=0.86, delays=0.06/0.02/0.26/0.52, dsn=2.0.0, status=sent (250 2.0.0 OK 1545305276 i15si6765428wrp.144 - gsmtp)
>> >
>> >
>> > I understand the information is in the log. but I need to archive this
>> > information for auditing in the future. so I need this information
>> > when postfix bcc the mail.
>>
>> FYI, none of the Postfix built-in BCC features archives the envelope
>> recipient.

On 20.12.18 21:13, d tbsky wrote:

>  although built-in bcc won't give envelope, there seems some tricky
>ways to do. for example:
>recipient_delimiter = +
>recipient_bcc_maps = pcre:/etc/postfix/bcc_map.cf
>
>and bcc_map.cf:
>/^(.*)@(.*)$/      archive+$1=$2@localhost
>
>then I can get the envelope for received mails.
>but I can not find the tricky way for sender_bcc_maps.
>if I need to write some script to handle this, any suggestion where to hook?
>thanks a lot for help!!

isn;t it easier to save one copy of mail with the logs, instead of two
copied of mail, without logs?
Note that logs will show e.g. when mail was refused by destination server,
mail won't.


--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Linux - It's now safe to turn on your computer.
Linux - Teraz mozete pocitac bez obav zapnut.
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
In reply to this post by Wietse Venema
Wietse Venema <[hidden email]>
> ... and sender_bcc_maps add the BCC recipient when email ARRIVES
> (i.e. input) not when mail is DELIVERED (i.e. output).
>
>         Wietse

   thanks for the clarify.  so that's my misunderstanding.
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
In reply to this post by Matus UHLAR - fantomas
Matus UHLAR - fantomas <[hidden email]>
</[hidden email]>
> isn;t it easier to save one copy of mail with the logs, instead of two
> copied of mail, without logs?
> Note that logs will show e.g. when mail was refused by destination server,
> mail won't.

I don't know if it is easier. but what I want is three information:
the mail content, who send the mail, the mail send to whom.
so i can send these to email archive system for auditing in the future..
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Matus UHLAR - fantomas
>Matus UHLAR - fantomas <[hidden email]>
></[hidden email]>
>> isn;t it easier to save one copy of mail with the logs, instead of two
>> copied of mail, without logs?
>> Note that logs will show e.g. when mail was refused by destination server,
>> mail won't.

On 20.12.18 21:50, d tbsky wrote:
>I don't know if it is easier. but what I want is three information:
>the mail content, who send the mail, the mail send to whom.

the latter 2 information is not available in mail header, unless you use
dirty hacks.

>so i can send these to email archive system for auditing in the future..

it's better (and often safer) to archive logs instead of while mail.
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
We are but packets in the Internet of life (userfriendly.org)
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
Matus UHLAR - fantomas <[hidden email]>
>> On 20.12.18 21:50, d tbsky wrote:
>>I don't know if it is easier. but what I want is three information:
>>the mail content, who send the mail, the mail send to whom.
>
>the latter 2 information is not available in mail header, unless you use
>dirty hacks.

   yes  I think I already use some dirty hacks. unfortunately it
doesn't cover all the situation yet.
I need more hack suggestion..

> it's better (and often safer) to archive logs instead of while mail.

  I need to archive the whole mails and related info. I think  it's a
common requirement for many companies?
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Matus UHLAR - fantomas
>>> On 20.12.18 21:50, d tbsky wrote:
>>>I don't know if it is easier. but what I want is three information:
>>>the mail content, who send the mail, the mail send to whom.
>>
>>the latter 2 information is not available in mail header, unless you use
>>dirty hacks.
>
>   yes  I think I already use some dirty hacks. unfortunately it
>doesn't cover all the situation yet.
>I need more hack suggestion..

>Matus UHLAR - fantomas <[hidden email]>
>> it's better (and often safer) to archive logs instead of while mail.

On 20.12.18 22:22, d tbsky wrote:
>  I need to archive the whole mails and related info. I think  it's a
>common requirement for many companies?

archiving one copy of a mail is enough. If you need information about how
the mail was sent, you need archive logs, not another copy of e-mail.

another copy of e-mail won't show when and how was outgoing mail delivered.
insteat of archiving otgoing mail, archive mail logs.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I'm not interested in your website anymore.
If you need cookies, bake them yourself.
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

Dominic Raferd
In reply to this post by d tbsky


On Thu, 20 Dec 2018 at 14:23, d tbsky <[hidden email]> wrote:
Matus UHLAR - fantomas <[hidden email]>
>> On 20.12.18 21:50, d tbsky wrote:
>>I don't know if it is easier. but what I want is three information:
>>the mail content, who send the mail, the mail send to whom.
>
>the latter 2 information is not available in mail header, unless you use
>dirty hacks.

   yes  I think I already use some dirty hacks. unfortunately it
doesn't cover all the situation yet.
I need more hack suggestion..

> it's better (and often safer) to archive logs instead of while mail.

  I need to archive the whole mails and related info. I think  it's a
common requirement for many companies?

I never heard of such requirement before. But the QueueID can be found in the first Received: header in each archived email and you can match this with the relevant smtp line for the outgoing delivery in the log file. You could extract the relevant data from this line and feed it back into the saved email file as a dummy header. It's a hack, sure. If you use a content_filter such as amavis then it is a bit harder because there are different QueueIDs for the received and outgoing emails.

Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
Dominic Raferd <[hidden email]>
> I never heard of such requirement before. But the QueueID can be found in the first Received: header in each archived email and you can match this with the relevant smtp line for the outgoing delivery in the log file. You could extract the relevant data from this line and feed it back into the saved email file as a dummy header. It's a hack, sure. If you use a content_filter such as amavis then it is a bit harder because there are different QueueIDs for the received and outgoing emails.

when I was using Qmail before, I wrote such archive system myself.
there are many times people lost their mails and ask to restore all
the old emails belong to them. or time to time there are legal issues
need to check related emails. I know there are companies who need to
keep their mails for 7 years.

now there are many mature email archive system in the market so I can
just use one of them. but I need to feed the system necessary
information to finish the job.

QueueID maybe an idea. but if there is some way to hook postfix to get
the info it maybe easier. (when using configuration is not possible)
I am using postfixadmin+dovecot. for incoming mails I already hook
"virtual_transport" and write my script to get every receiver.
is there similar hook I can do for outgoing mails?
Reply | Threaded
Open this post in threaded view
|

Re: capture information for internal generated mails

d tbsky
In reply to this post by Matus UHLAR - fantomas
Matus UHLAR - fantomas <[hidden email]>
&gt; archiving one copy of a mail is enough. If you need information about how
&gt; the mail was sent, you need archive logs, not another copy of e-mail.

   yes one copy of mail is enough. other redundant mails are just for
extra information and will be abandoned after extract the information.
all the from,to, attachment will goto database for indexing. logs are
hard for future auditing.