Stripping Received: headers

[Geoff Shang]

Hi,

I'm trying to strip Received: headers from mail at various parts of our
processing, for security reasons.

I'm starting with mail that comes in from authenticated clients.  I tried
doing the following:

master.cf:

submission inet n       -       -       -       -       smtpd
   -o smtpd_enforce_tls=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o milter_macro_daemon_name=ORIGINATING
   -o header_checks=pcre:/etc/postfix/header_checks

/etc/postfix/header_checks:

/^Received:/    IGNORE

I ran this through Postmap with a query from a message I sent myself, and
the IGNORE key is correctly returned.  But if I actually send myself a
message, it comes through with the Received: line intact.

I did some searching and found
http://marc.info/?l=postfix-users&m=122106227124195&w=2

I'm curious to know why this would work and the above wouldn't.  Am I just
trying to do it too early in the process?

A related question, is it possible to prevent Postfix from generating
lines like this?

Geoff.

[Wietse Venema]

Geoff Shang:
> submission inet n       -       -       -       -       smtpd
>    -o smtpd_enforce_tls=yes
>    -o smtpd_sasl_auth_enable=yes
>    -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>    -o milter_macro_daemon_name=ORIGINATING
>    -o header_checks=pcre:/etc/postfix/header_checks

As documented header_checks is not an smtpd(8) feature, it is
a cleanup(8) feature.

The easiest way to give separate treatment to mail from the
internal network versus mail from outside is to use separate
Postfix instances.

Otherwise,

    submission inet n       -       -       -       -       smtpd
        -o cleanup_service=submission_cleanup

    submission_cleanup unix n  ............................ cleanup
        -o header_checks=pcre:/etc/postfix/header_checks

would do the job.

        Wietse

[Benny Pedersen-2]

Geoff Shang skrev den 2013-04-11 16:33:

> Hi,
>
> I'm trying to strip Received: headers from mail at various parts of
> our processing, for security reasons.
>
> I'm starting with mail that comes in from authenticated clients.  I
> tried doing the following:
>
> master.cf:
>
> submission inet n       -       -       -       -       smtpd
>   -o smtpd_enforce_tls=yes
>   -o smtpd_sasl_auth_enable=yes
>   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>   -o milter_macro_daemon_name=ORIGINATING
>   -o header_checks=pcre:/etc/postfix/header_checks

header_checks is incomming on smtpd, but you use submission

so you must change to to smtp_header_checks

http://www.postfix.org/header_checks.5.html

>
> /etc/postfix/header_checks:
>
> /^Received:/    IGNORE

this one is to gready, dont use it on header_checks

>
> I ran this through Postmap with a query from a message I sent myself,
> and the IGNORE key is correctly returned.  But if I actually send
> myself a message, it comes through with the Received: line intact.
>
> I did some searching and found
> http://marc.info/?l=postfix-users&m=122106227124195&w=2
>
> I'm curious to know why this would work and the above wouldn't.  Am I
> just trying to do it too early in the process?
>
> A related question, is it possible to prevent Postfix from generating
> lines like this?

what problems do you like to resolve ?

>
> Geoff.

--
senders that put my email into body content will deliver it to my own
trashcan, so if you like to get reply, dont do it

[Noel Jones-2]

On 4/11/2013 10:05 AM, Benny Pedersen wrote:

> Geoff Shang skrev den 2013-04-11 16:33:
>> Hi,
>>
>> I'm trying to strip Received: headers from mail at various parts of
>> our processing, for security reasons.
>>
>> I'm starting with mail that comes in from authenticated clients.  I
>> tried doing the following:
>>
>> master.cf:
>>
>> submission inet n       -       -       -       -       smtpd
>>   -o smtpd_enforce_tls=yes
>>   -o smtpd_sasl_auth_enable=yes
>>   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>>   -o milter_macro_daemon_name=ORIGINATING
>>   -o header_checks=pcre:/etc/postfix/header_checks
>
> header_checks is incomming on smtpd, but you use submission

No, header_checks are performed on all incoming mail.

As already explained, the problem above is that "-o
header_checks=..." has no effect on smtpd(5).

>
> so you must change to to smtp_header_checks

smtp_header_checks are performed on outgoing mail during smtp(5)
delivery.

But you're sort of on the right track.  You can use
smtp_header_checks to remove the Received: headers from
authenticated mail before external delivery with something like:
/^Received: .*by myserver.example.com \(Postfix\) with ESMTPS?A
id.*$/  IGNORE

Geoff, please note I've seen some overanxious anti-spam systems that
consider mail with no Received: headers as spam.



  -- Noel Jones

>
> http://www.postfix.org/header_checks.5.html
>
>>
>> /etc/postfix/header_checks:
>>
>> /^Received:/    IGNORE
>
> this one is to gready, dont use it on header_checks
>
>>
>> I ran this through Postmap with a query from a message I sent myself,
>> and the IGNORE key is correctly returned.  But if I actually send
>> myself a message, it comes through with the Received: line intact.
>>
>> I did some searching and found
>> http://marc.info/?l=postfix-users&m=122106227124195&w=2
>>
>> I'm curious to know why this would work and the above wouldn't.  Am I
>> just trying to do it too early in the process?
>>
>> A related question, is it possible to prevent Postfix from generating
>> lines like this?
>
> what problems do you like to resolve ?
>
>>
>> Geoff.
>

[Benny Pedersen-2]

Noel Jones skrev den 2013-04-11 18:29:

> No, header_checks are performed on all incoming mail.

+1

> As already explained, the problem above is that "-o
> header_checks=..." has no effect on smtpd(5).

yes it included as it used all incomming, but not directly with smtpd

>> so you must change to to smtp_header_checks
>
> smtp_header_checks are performed on outgoing mail during smtp(5)
> delivery.

is submission not using smtp_header_checks ?

> But you're sort of on the right track.  You can use
> smtp_header_checks to remove the Received: headers from
> authenticated mail before external delivery with something like:
> /^Received: .*by myserver.example.com \(Postfix\) with ESMTPS?A
> id.*$/  IGNORE

yep will soon try to apply it here

> Geoff, please note I've seen some overanxious anti-spam systems that
> consider mail with no Received: headers as spam.

can one show an example main.cf that remove all recieved headers on
remote senders ?, there would allways be one last hop imho

--
senders that put my email into body content will deliver it to my own
trashcan, so if you like to get reply, dont do it

[DTNX Postmaster]

On Apr 11, 2013, at 18:29, Noel Jones <[hidden email]> wrote:

>> so you must change to to smtp_header_checks
>
> smtp_header_checks are performed on outgoing mail during smtp(5)
> delivery.
>
> But you're sort of on the right track.  You can use
> smtp_header_checks to remove the Received: headers from
> authenticated mail before external delivery with something like:
> /^Received: .*by myserver.example.com \(Postfix\) with ESMTPS?A
> id.*$/  IGNORE
>
> Geoff, please note I've seen some overanxious anti-spam systems that
> consider mail with no Received: headers as spam.

In our case, the problem was with overzealous ones that filter on all
Received: headers, and therefore block legitimate mail because the
authenticated client is connecting from an access provider range listed
by Spamhaus, or something similar.

Our solution so far is to strip a few of the internal Received:
headers, and 'REPLACE' the one that contains the connecting IP with a
'Received: by hostname.domain.tld (from authenticated client)' header.
Since the submission hosts never send directly, it will always have at
least three or four Received: headers when offered to the destination
MX.

Since the regular expression is fairly specific, this is done with
'header_checks' in our case.

HTH,
Jona

[Reindl Harald-2]

Am 11.04.2013 18:55, schrieb Benny Pedersen:
>> smtp_header_checks are performed on outgoing mail during smtp(5)
>> delivery.
>
> is submission not using smtp_header_checks?

has your submission service smtp or smtpd in master.cf?
mine has smtpd as all other working ones out there

[Reindl Harald-2]

Am 11.04.2013 19:20, schrieb Reindl Harald:

>
>
> Am 11.04.2013 18:55, schrieb Benny Pedersen:
>>> smtp_header_checks are performed on outgoing mail during smtp(5)
>>> delivery.
>>
>> is submission not using smtp_header_checks?
>
> has your submission service smtp or smtpd in master.cf?
> mine has smtpd as all other working ones out there

to make it clear:

submission is nothing else as smtpd on port 587
and if you want not rely on /etc/services you would
even write 587 instead submission

the only difference between port 25 and 587 is
usually that you require authentication on 587

[harry@srv-rhsoft:~]$ cat /etc/services | grep submission
submission      587/tcp         msa             # mail message submission
submission      587/udp         msa             # mail message submission

[Noel Jones-2]

On 4/11/2013 11:55 AM, Benny Pedersen wrote:
> Noel Jones skrev den 2013-04-11 18:29:

>> smtp_header_checks are performed on outgoing mail during smtp(5)
>> delivery.
>
> is submission not using smtp_header_checks ?

No.

submission uses the smtpd(5) service to receive mail, which uses
header_checks (indirectly, through the cleanup service).

smtp_header_checks are used by the smtp(5) transport when sending
mail to remote systems.


http://www.postfix.org/OVERVIEW.html


  -- Noel Jones

[Geoff Shang]

On Thu, 11 Apr 2013, Wietse Venema wrote:

> Geoff Shang:
>> submission inet n       -       -       -       -       smtpd
>>    -o smtpd_enforce_tls=yes
>>    -o smtpd_sasl_auth_enable=yes
>>    -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>>    -o milter_macro_daemon_name=ORIGINATING
>>    -o header_checks=pcre:/etc/postfix/header_checks
>
> As documented header_checks is not an smtpd(8) feature, it is
> a cleanup(8) feature.

Oh duh!  Thanks for pointing this out.

> The easiest way to give separate treatment to mail from the
> internal network versus mail from outside is to use separate
> Postfix instances.
>
> Otherwise,
>
>    submission inet n       -       -       -       -       smtpd
> -o cleanup_service=submission_cleanup
>
>    submission_cleanup unix n  ............................ cleanup
> -o header_checks=pcre:/etc/postfix/header_checks
>
> would do the job.

Thanks Wietse.  I think I will opt for this latter option.

Some have suggested smtp_header_checks, and I may use this in some places.
But since this box will deliver some mail locally as well as externally, I
think I will implement the separate cleanup process.

Thanks everyone for your input.

Geoff.

[Geoff Shang]

On Fri, 12 Apr 2013, Geoff Shang wrote:

>>    submission inet n       -       -       -       -       smtpd
>> -o cleanup_service=submission_cleanup
>>
>>    submission_cleanup unix n  ............................ cleanup
>> -o header_checks=pcre:/etc/postfix/header_checks
>>
>> would do the job.
>
> Thanks Wietse.  I think I will opt for this latter option.

hmm.  This didn't work.  I'm a bit stuck as to why.  I thought that
perhaps it might be running before the Received: header is created, but in
that case, I don't know why the example I linked to earlier that searches
for an authenticated header would work, while this would not.

Geoff.

[Wietse Venema]

Geoff Shang:

> On Fri, 12 Apr 2013, Geoff Shang wrote:
>
> >>    submission inet n       -       -       -       -       smtpd
> >> -o cleanup_service=submission_cleanup
> >>
> >>    submission_cleanup unix n  ............................ cleanup
> >> -o header_checks=pcre:/etc/postfix/header_checks
> >>
> >> would do the job.
> >
> > Thanks Wietse.  I think I will opt for this latter option.
>
> hmm.  This didn't work.  I'm a bit stuck as to why.  I thought that
> perhaps it might be running before the Received: header is created, but in
> that case, I don't know why the example I linked to earlier that searches
> for an authenticated header would work, while this would not.

You need to do "postfix reload" after editing master.cf.

The submission_cleanup service will see the Received: header that
was prepended by the submission server.

However, if your Milter adds headers then those aren't seen by
header_checks; you would need to use milter_header_checks.

        Wietse

[Geoff Shang]

On Fri, 12 Apr 2013, Wietse Venema wrote:

> You need to do "postfix reload" after editing master.cf.

I did.  I did it again for good measure - no difference.

> The submission_cleanup service will see the Received: header that
> was prepended by the submission server.

Is there any way I can be sure that the special cleanup agent is running?
I see the socket /var/spool/postfix/public/submission_cleanup

> However, if your Milter adds headers then those aren't seen by
> header_checks; you would need to use milter_header_checks.

We don't appear to be using any milters, despite the
'milter_macro_daemon_name=ORIGINATING'

Here's what I did in case I messed up:

master.cf:

# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
#
==========================================================================
smtp      inet  n       -       -       -       -       smtpd
submission inet n       -       -       -       -       smtpd
   -o smtpd_enforce_tls=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o milter_macro_daemon_name=ORIGINATING
   # Use a special cleanup service so we can strip headers.
   -o cleanup_service=submission_cleanup

smtps     inet  n       -       -       -       -       smtpd
   -o smtpd_tls_wrappermode=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o milter_macro_daemon_name=ORIGINATING
   # Use a special cleanup service so we can strip headers.
   -o cleanup_service=submission_cleanup

submission_cleanup   unix  n       -       -       -       -       cleanup
   # Strip Received: lines from authenticated mail
   -o header_checks=pcre:/etc/postfix/header_checks



/etc/postfix/header_checks:

# Remove any Received: headers from authenticated mail.
/^Received:/    IGNORE



An example message.  The line is matched if I run it through postmap.
Some details have to be obscured, sorry.  I'm on holiday so I'm not
worried about letting the hostname through, you can all get it from my
headers anyway.  Obviously I'm not posting from my work address.

Return-Path: <[hidden email]>
X-Original-To: [hidden email]
Delivered-To: [hidden email]
Received: from [192.168.0.20] (dsl-mlibrasgw2-50de1c-161.dhcp.inet.fi
[80.222.28.161])
     by mail.example.com (Postfix) with ESMTPSA id DED281C40E9
     for <[hidden email]>; Fri, 12 Apr 2013 14:35:47
+0000 (UTC)
Date: Fri, 12 Apr 2013 17:35:44 +0300 (EEST)
From: Geoff Shang <[hidden email]>
X-X-Sender: [hidden email]
To: [hidden email]
Subject: test
Message-ID: <[hidden email]>
User-Agent: Alpine 2.02 (DEB 1266 2009-07-14)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII

  Geoff.\

[/dev/rob0]

A word at the outset here: I predict this will come back to bite you
in a most painful way. As Noel suggested, you're going to run afoul
of some clueless spam checks. Some years back I know that Hotmail/MSN
actually *discarded* such mail silently!

Note also that Postfix itself uses Received: headers as a protection
against mail loops. Let's hope you don't get a loop going!

On Fri, Apr 12, 2013 at 05:49:47PM +0300, Geoff Shang wrote:
> Is there any way I can be sure that the special cleanup agent
> is running? I see the socket
> /var/spool/postfix/public/submission_cleanup

It's running. To see what it does:

> master.cf:

> submission_cleanup   unix  n    -    -    -    -    cleanup
>   # Strip Received: lines from authenticated mail
>   -o header_checks=pcre:/etc/postfix/header_checks
    -o syslog_name=postfix/submission/cleanup

Every non-default service should have its own syslog_name to enhance
your log searches.

> /etc/postfix/header_checks:
>
> # Remove any Received: headers from authenticated mail.
> /^Received:/    IGNORE
/./ WARN

That might get too noisy in the logs, but at least you will know your
alternate cleanup service is being used.
--
  http://rob0.nodns4.us/ -- system administration and consulting
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

[Wietse Venema]

Geoff Shang:
> On Fri, 12 Apr 2013, Wietse Venema wrote:
>
> > You need to do "postfix reload" after editing master.cf.
>
> I did.  I did it again for good measure - no difference.

Are you using receive_override_options? in main.cf or master.cf?

        Wietse

[Geoff Shang]

On Fri, 12 Apr 2013, Wietse Venema wrote:

> Geoff Shang:
>> On Fri, 12 Apr 2013, Wietse Venema wrote:
>>
>>> You need to do "postfix reload" after editing master.cf.
>>
>> I did.  I did it again for good measure - no difference.
>
> Are you using receive_override_options? in main.cf or master.cf?

No.

Geoff.

[Den1]

my apologies to post here again as the thread is quite old but I am having аn absolutely identical issue. I cannot seem to remove any headers at all no matter what I do.

I tried setting this in master.cf:
submission inet n       -       -       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=subcleanup

and this:
subcleanup unix n       -       -       -       0       cleanup
  -o smtp_header_checks=regexp:/etc/postfix/header_checks

then changed smtp_header_checks to header_checks just to test it:
subcleanup unix n       -       -       -       0       cleanup
  -o header_checks=regexp:/etc/postfix/headerch

with my both header_checks and headerch files that contain this:
/^Received: .*/     IGNORE
/^X-Originating-IP:/    IGNORE
/^Received:.*from:/ IGNORE
/^Received:.*\s*from/ IGNORE
/^Received:.*\sfrom/ IGNORE
/^Received:\s.*from/ IGNORE
/^Received:\sfrom/ IGNORE
/^Received: from .*/  IGNORE

and I have this in my main.cf :
smtp_header_checks = regexp:/etc/postfix/header_checks
header_checks = regexp:/etc/postfix/headerch

but all the headers are still there in both incoming and outgoing emails. I was wondering was this somehow solved by Geoff? Would appreciate any advises at all. Many thanks in advance! I am running Postfix 2.9.6. I also tried to remove this:

-o cleanup_service_name=subcleanup

and this:

subcleanup unix n       -       -       -       0       cleanup
  -o smtp_header_checks=regexp:/etc/postfix/header_checks

and this:

subcleanup unix n       -       -       -       0       cleanup
  -o header_checks=regexp:/etc/postfix/headerch

keeping just

smtp_header_checks = regexp:/etc/postfix/header_checks
header_checks = regexp:/etc/postfix/headеrch

in my main.cf but it's still the same, no headers are removed in both incoming and outgoing emails.

[Noel Jones-2]

On 10/27/2016 1:57 PM, Den1 wrote:

> my apologies to post here again as the thread is quite old but I am having аn
> absolutely identical issue. I cannot seem to remove any headers at all no
> matter what I do.
>
> I tried setting this in master.cf:
> submission inet n       -       -       -       -       smtpd
>   -o smtpd_tls_security_level=encrypt
>   -o smtpd_sasl_auth_enable=yes
>   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>   -o milter_macro_daemon_name=ORIGINATING
>   -o cleanup_service_name=subcleanup

This looks reasonably correct.

>
> and this:
> subcleanup unix n       -       -       -       0       cleanup
>   -o smtp_header_checks=regexp:/etc/postfix/header_checks

smtp_header_checks is a property of the smtp delivery agent, not
cleanup.  Not surprising it doesn't work here.

>
> then changed smtp_header_checks to header_checks just to test it:
> subcleanup unix n       -       -       -       0       cleanup
>   -o header_checks=regexp:/etc/postfix/headerch

This looks reasonably correct.

>
> with my both header_checks and headerch files that contain this:
> /^Received: .*/     IGNORE
> /^X-Originating-IP:/    IGNORE
> /^Received:.*from:/ IGNORE
> /^Received:.*\s*from/ IGNORE
> /^Received:.*\sfrom/ IGNORE
> /^Received:\s.*from/ IGNORE
> /^Received:\sfrom/ IGNORE
> /^Received: from .*/  IGNORE

You can replace all the above with;
/^Received: /  IGNORE


>
> and I have this in my main.cf :
> smtp_header_checks = regexp:/etc/postfix/header_checks
> header_checks = regexp:/etc/postfix/headerch

unwise, that will mangle all your mail, both incoming and outgoing.

>
> but all the headers are still there in both incoming and outgoing emails.


Do you have receive_override_options set in main.cf?
What's "postconf -n" show?
what's "postconf -Mf" show?




  -- Noel Jones

[Den1]

>>Do you have receive_override_options set in main.cf?

Yes, I do, but I disabled them for testing purposes.

>>What's "postconf -n" show?
>>what's "postconf -Mf" show?

Would you like me to post the entire output of postconf -n and postconf -Mf? Won't your list ban me for flooding because of it will it? :-)))

>
> and this:
> subcleanup unix n       -       -       -       0       cleanup
>   -o smtp_header_checks=regexp:/etc/postfix/header_checks

smtp_header_checks is a property of the smtp delivery agent, not
cleanup.  Not surprising it doesn't work here.

Where do you suggest I have to put those then? Thank you.

[Wietse Venema]

Den1:
> >>Do you have receive_override_options set in main.cf?
>
> Yes, I do, but I disabled them for testing purposes.
>
> >>What's "postconf -n" show?
> >>what's "postconf -Mf" show?
>
> Would you like me to post the entire output of postconf -n and postconf -Mf?
> Won't your list ban me for flooding because of it will it? :-)))

"postconf -n" output should be no more than a few dozen lines.

        Wietse