How to copy all incoming and outgoing messages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to copy all incoming and outgoing messages

Forums-6
Hi Everyone,

How do I make it so postfix will copy all incoming and outgoing emails to another email invisibly? It's a SOX requirement actually. I have to save all the emails.

So, I create one account called "log". Every email that goes through the mail server has to be invisibly copied to that user whether incoming or outgoing. Thanks for the help. I'm new to postfix.

Peter
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

d.hill
On Thu, 1 May 2008 at 22:22 -0000, [hidden email] confabulated:

> Hi Everyone,
>
> How do I make it so postfix will copy all incoming and outgoing emails to another email invisibly? It's a SOX requirement actually. I have to save all the emails.
>
> So, I create one account called "log". Every email that goes through the mail server has to be invisibly copied to that user whether incoming or outgoing. Thanks for the help. I'm new to postfix.

Postfix configuration parameters:

   always_bcc, sender_bcc_maps, recipient_bcc_maps

control what you are asking for. You can do a search on this page:

   http://www.postfix.org/postconf.5.html

for further explination.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Curtis Vaughan-2


D Hill wrote:

> On Thu, 1 May 2008 at 22:22 -0000, [hidden email]
> confabulated:
>
>> Hi Everyone,
>>
>> How do I make it so postfix will copy all incoming and outgoing
>> emails to another email invisibly? It's a SOX requirement actually. I
>> have to save all the emails.
>>
>> So, I create one account called "log". Every email that goes through
>> the mail server has to be invisibly copied to that user whether
>> incoming or outgoing. Thanks for the help. I'm new to postfix.
>
> Postfix configuration parameters:
>
>   always_bcc, sender_bcc_maps, recipient_bcc_maps
>
> control what you are asking for. You can do a search on this page:
>
>   http://www.postfix.org/postconf.5.html
>
> for further explination.

This is great question, as I've been wondering how people meet gov.
requirements on storing corp. emails with Postfix.
As it is our problem is that some people use POP3 and delete as they
download (I've been fighting this for years, but I can't control the
foreign office!).
Anyhow.... Even with IMAP,  although most mail is saved, still there is
they can delete emails from Trash and then those emails are lost.

If all mail were just sent to another address, then it would make it
difficult to find emails in the event we need to recover specific ones.
It seems like there should be a way to say, regardless of whether
someone wants to delete -- well, don't delete it from the server. Maybe
do something else with it within the same users profile. I don't know.
I should note that we are not a large company. Only 60-odd employees
worldwide. And the budget for me to have continual backups doesn't
exist. We do nightly backups using BackupPC. So we have about a weeks
retention.

Any ideas are most welcome, but please take into consideration our
constrictions (budgetary).
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Victor Duchovni
On Thu, May 01, 2008 at 04:17:26PM -0700, Curtis Vaughan wrote:

> This is great question, as I've been wondering how people meet gov.
> requirements on storing corp. emails with Postfix.
> As it is our problem is that some people use POP3 and delete as they
> download (I've been fighting this for years, but I can't control the
> foreign office!).
> Anyhow.... Even with IMAP,  although most mail is saved, still there is
> they can delete emails from Trash and then those emails are lost.

You can get a fair wait with regexp based recipient_bcc_maps, which
allow you to capture the original envelope recipient. Merge message
copies are not sufficient IMHO as headers are not accurate/complete.

I usee a "tee" proxy that sends an encapsulated archive copy and the real
message in parallel (archive "." immediately precedes message "." and
blocks message delivery on failure).

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Sahil Tandon
* Victor Duchovni <[hidden email]> [2008-05-01 19:24:17 -0400]:

> On Thu, May 01, 2008 at 04:17:26PM -0700, Curtis Vaughan wrote:
>
> > This is great question, as I've been wondering how people meet gov.
> > requirements on storing corp. emails with Postfix.
> > As it is our problem is that some people use POP3 and delete as they
> > download (I've been fighting this for years, but I can't control the
> > foreign office!).
> > Anyhow.... Even with IMAP,  although most mail is saved, still there is
> > they can delete emails from Trash and then those emails are lost.
>
> You can get a fair wait with regexp based recipient_bcc_maps, which
> allow you to capture the original envelope recipient. Merge message
> copies are not sufficient IMHO as headers are not accurate/complete.
>
> I usee a "tee" proxy that sends an encapsulated archive copy and the real
> message in parallel (archive "." immediately precedes message "." and
> blocks message delivery on failure).
       
Is there publicly available documentation on how to implement the tee proxy
backup solution in Postfix?
       
--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Victor Duchovni
On Thu, May 01, 2008 at 08:04:17PM -0400, Sahil Tandon wrote:

> Is there publicly available documentation on how to implement the tee proxy
> backup solution in Postfix?

        1. Write a tee proxy
        2. Set content_filter to the tee proxy

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: How to copy all incoming and outgoing messages

Tandon, Sahil (IM)
> > Is there publicly available documentation on how to
> implement the tee
> > proxy backup solution in Postfix?
>
> 1. Write a tee proxy
> 2. Set content_filter to the tee proxy

Thanks, that is very helpful.

Sahil
--------------------------------------------------------

NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Victor Duchovni
On Fri, May 02, 2008 at 09:19:34AM -0400, Tandon, Sahil (IM) wrote:

> > > Is there publicly available documentation on how to
> > implement the tee
> > > proxy backup solution in Postfix?
> >
> > 1. Write a tee proxy
> > 2. Set content_filter to the tee proxy
>
> Thanks, that is very helpful.

It's the best I can do until I find the cycles to fully document and
then release the tee proxy I am using. I don't think it is appropriate
to release it in an undocumented state.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Charles Marcus
On 5/2/2008, Victor Duchovni ([hidden email]) wrote:
> It's the best I can do until I find the cycles to fully document and
> then release the tee proxy I am using. I don't think it is appropriate
> to release it in an undocumented state.

Understandable, and I'm sure I and many others look forward to this...

With the advancing requirements of burdensome regulations, this (some
kind of basic archiving capability) will become more and more important,
and in my opinion, it is not unreasonable for postfix to provide an
integrated/built-in method of performing this function, to ensure a
robust and safe - ie, don't bounce if there is a problem with the
mirror, but queue until it is back up, etc - functionality.

It would also be nice if it could easily deliver to an appropriate
sub-folder - ie, one named after the local address part of the original
envelope recipient (the one(s) being tested for during recipient
validation stage when the primary server accepted the message for final
delivery)...

--

Best regards,

Charles
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Victor Duchovni
On Fri, May 02, 2008 at 10:41:40AM -0400, Charles Marcus wrote:

> On 5/2/2008, Victor Duchovni ([hidden email]) wrote:
> >It's the best I can do until I find the cycles to fully document and
> >then release the tee proxy I am using. I don't think it is appropriate
> >to release it in an undocumented state.
>
> Understandable, and I'm sure I and many others look forward to this...
>
> With the advancing requirements of burdensome regulations, this (some
> kind of basic archiving capability) will become more and more important,
> and in my opinion, it is not unreasonable for postfix to provide an
> integrated/built-in method of performing this function, to ensure a
> robust and safe - ie, don't bounce if there is a problem with the
> mirror, but queue until it is back up, etc - functionality.

I don't expect the proxy in question to ever be part of Postfix. Postfix
supports at least 4 extension mechanisms:

        - Post-queue content filters
        - Pre-queue proxy filters
        - Milters
        - Policy servers

It is up to the Postfix community and vendors to create add-on tools
that make use of these features. So I don't see a "buit-in" archive
feature any time soon.

> It would also be nice if it could easily deliver to an appropriate
> sub-folder - ie, one named after the local address part of the original
> envelope recipient (the one(s) being tested for during recipient
> validation stage when the primary server accepted the message for final
> delivery)...

The right mechanism leaves this choice to the administrator, the
archive copy is created and queued, after that you can deliver it
where-ever you want (configure the archive Postfix instance transport
rules accoringly). It would be wrong to make delivery decisions in the
archive module, they would never be sufficiently comprehensive.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Michael Katz-2
In reply to this post by Forums-6
Forums wrote:
> Hi Everyone,
>
> How do I make it so postfix will copy all incoming and outgoing emails to another email invisibly? It's a SOX requirement actually. I have to save all the emails.
>
> So, I create one account called "log". Every email that goes through the mail server has to be invisibly copied to that user whether incoming or outgoing. Thanks for the help. I'm new to postfix.
>
> Peter

You can do exactly this with MPP.  I get a million lashes for mentioning
our solution on the list, but until someone funds my life we have to
charge for the software. messagepartners.com .

M Katz

>
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Wietse Venema
In reply to this post by Victor Duchovni
Victor Duchovni:

> On Fri, May 02, 2008 at 10:41:40AM -0400, Charles Marcus wrote:
>
> > On 5/2/2008, Victor Duchovni ([hidden email]) wrote:
> > >It's the best I can do until I find the cycles to fully document and
> > >then release the tee proxy I am using. I don't think it is appropriate
> > >to release it in an undocumented state.
> >
> > Understandable, and I'm sure I and many others look forward to this...
> >
> > With the advancing requirements of burdensome regulations, this (some
> > kind of basic archiving capability) will become more and more important,
> > and in my opinion, it is not unreasonable for postfix to provide an
> > integrated/built-in method of performing this function, to ensure a
> > robust and safe - ie, don't bounce if there is a problem with the
> > mirror, but queue until it is back up, etc - functionality.
>
> I don't expect the proxy in question to ever be part of Postfix. Postfix
> supports at least 4 extension mechanisms:
>
> - Post-queue content filters
> - Pre-queue proxy filters
> - Milters
> - Policy servers
>
> It is up to the Postfix community and vendors to create add-on tools
> that make use of these features. So I don't see a "buit-in" archive
> feature any time soon.
>
> > It would also be nice if it could easily deliver to an appropriate
> > sub-folder - ie, one named after the local address part of the original
> > envelope recipient (the one(s) being tested for during recipient
> > validation stage when the primary server accepted the message for final
> > delivery)...
>
> The right mechanism leaves this choice to the administrator, the
> archive copy is created and queued, after that you can deliver it
> where-ever you want (configure the archive Postfix instance transport
> rules accoringly). It would be wrong to make delivery decisions in the
> archive module, they would never be sufficiently comprehensive.

If we can agree on a usable MIME encapsulation, then it should be
possible to spawn off a message in the cleanup server, after the
Milter processing has happened, and before the (SMTP) client is
notified that the mail transaction is complete.

However, Postfix is a general-purpose MTA, and you can already
configure dedicated delivery channels (with transport maps and
master.cf) that have soft-bounce turned on, so there is no need
for built-in special delivery modes that never bounce. Just set
the maximal queue time large enough.

        Wietse
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Victor Duchovni
On Fri, May 02, 2008 at 01:04:16PM -0400, Wietse Venema wrote:

> If we can agree on a usable MIME encapsulation, then it should be
> possible to spawn off a message in the cleanup server, after the
> Milter processing has happened, and before the (SMTP) client is
> notified that the mail transaction is complete.

This is a bit tricky, because the is not necessarily a "right" answer.

    - IMHO, The natural format for an archive message is a success DSN,
    with the original message attached in full (not just headers). Having
    the format defined by a standard is IMHO rather attractive.

    - Sendmail have for many years been supplying  "copier" milter that
    is perhaps a defacto standard for such encapsulation. This is a
    multipart/mixed, with the envelope in the first part and the message
    in the second. The first part encodes the sender and recipients one
    per line as follows:

        --boundary
        Content-Type: text/plain
        Content-Transfer-Encoding: 7bit

        Attached is a copy of a message being sent by Sendmail Message Copier
        Sendmail Copier.
        via the proxy on <hostname.without.the.angle.brackets>.

        Original sender:       [hidden email]
        Original recipient(s): [hidden email]
                               [hidden email]
                               [hidden email]

        Original message is attached.

        --boundary

    It would be far more natural to drop the verbiage and prefix each
    address with a type:

        Sender: <address>
        Recipient: <address>
        ...

    The second part is the attached message.

    - Microsoft Exchange has an archive format called
      Exchange Envelope Journalling. This too is not ideal, but is
      broadly implemented.

Any format we would choose would either be ugly (ad-hoc format from
existing vendor products) or Postfix-specific (at least initially).

The archive module for my 'tee' proxy punts the issue by using
format templates for the envelope part.

        --boundary
        Content-Type: text/plain
        Content-Transfer-Encoding: 7bit

        $topmatter
        `printf "$senderfmt", $sender`
        `printf "$1strcptfmt", $rcpt1`
        `printf "$nxtrcptfmt", $rcpt2`
        ...
        $footer

        --boundary

So I can generate the Sendmail compatible format without hard-coding
it.

This can generate any 2-part envelope/message encapsulation with some
stuff above the envelope, then the sender address in some form, then
the recipients with the 1st formatted differently if need be, a footer
and a separately attached message.

I also capture "ORCPT" values, but don't currently distinguish between
these and real recipients, because I don't accept "ORCPT" from outside,
rather ORCPT is the recipient just before my own virtual address
expansion. So capture of ORCPT would need to be optional and probably
deserves its own format template.

        $topmatter
        `printf "$senderfmt", $sender`
        `printf "$1strcptfmt", $rcpt1`
        [`printf "$orcptfmt", $orcpt1`]
        `printf "$nxtrcptfmt", $rcpt2`
        [`printf "$orcptfmt", $orcpt2`]
        ...
        $footer

With so much rope, is this still a reasonable Postfix feature?

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:[hidden email]?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to copy all incoming and outgoing messages

Wietse Venema
[About forking off an acrhive-copy message just before the cleanup
server commits the queue file transaction]

Victor Duchovni:
> With so much rope, is this still a reasonable Postfix feature?

It should be able to provide one text/plain MIME segment with
original sender, sender, dsn original recipient, Postfix original
recipient; and one message/rfc822 MIME segment with the content.

As long as the fields in the first MIME segment have distinct
labels, I don't see this as particularly challenging. Given the
bounce templates as an example, the first MIME segment could even
be made configurable.  But it would in all likelihood be a separate
daemon process.

        Wietse
Loading...