Rewriting Subject line, adding an X-header?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Rewriting Subject line, adding an X-header?

Ville Walveranta
I'm setting up Postfix 2.5.1, Dovecot 1.1.1 on FreeBSD 7 and will be
using an external (commercial) spam filtering service that forwards
the emails to my Postfix/Dovecot mailserver. I have the option to
either keep the spam at the external service, or forward them to the
local system. I'm thinking of bringing them local, and automatically
placing them to spam folder (probably by using the Sieve plugin, but
that's another issue). The external service tags the spam Subject
lines with **SPAM**, and currently there is no option to change that
to a header tag on their side. I would like to do that locally so that
the Subjects of all of the spam messages in their spam boxes would not
start with "**SPAM**".

So the question is: How do I detect "**SPAM**" on the subject line,
remove it (i.e. rewrite the Subject line), and add something like
"X-Spam: yes" into the header of the messages whose Subject includes
"**SPAM**"?

Thanks for any advise, insights on this!

Ville
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Wietse Venema
Ville Walveranta:

> I'm setting up Postfix 2.5.1, Dovecot 1.1.1 on FreeBSD 7 and will be
> using an external (commercial) spam filtering service that forwards
> the emails to my Postfix/Dovecot mailserver. I have the option to
> either keep the spam at the external service, or forward them to the
> local system. I'm thinking of bringing them local, and automatically
> placing them to spam folder (probably by using the Sieve plugin, but
> that's another issue). The external service tags the spam Subject
> lines with **SPAM**, and currently there is no option to change that
> to a header tag on their side. I would like to do that locally so that
> the Subjects of all of the spam messages in their spam boxes would not
> start with "**SPAM**".
>
> So the question is: How do I detect "**SPAM**" on the subject line,
> remove it (i.e. rewrite the Subject line), and add something like
> "X-Spam: yes" into the header of the messages whose Subject includes
> "**SPAM**"?

This requires both a "prepend" and a "replace" action.
Postfix header_checks perform only one action per header.
Therefore, you need an external content filter.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Sahil Tandon
In reply to this post by Ville Walveranta
Ville Walveranta <[hidden email]> wrote:

> I'm setting up Postfix 2.5.1, Dovecot 1.1.1 on FreeBSD 7 and will be
> using an external (commercial) spam filtering service that forwards
> the emails to my Postfix/Dovecot mailserver. I have the option to
> either keep the spam at the external service, or forward them to the
> local system. I'm thinking of bringing them local, and automatically
> placing them to spam folder (probably by using the Sieve plugin, but
> that's another issue). The external service tags the spam Subject
> lines with **SPAM**, and currently there is no option to change that
> to a header tag on their side. I would like to do that locally so that
> the Subjects of all of the spam messages in their spam boxes would not
> start with "**SPAM**".
>
> So the question is: How do I detect "**SPAM**" on the subject line,
> remove it (i.e. rewrite the Subject line), and add something like
> "X-Spam: yes" into the header of the messages whose Subject includes
> "**SPAM**"?
>
> Thanks for any advise, insights on this!
                 
AFAIK this cannot be done within Postfix.

--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Ville Walveranta
In reply to this post by Wietse Venema
Thanks for the responses! I printed out bunch of Postfix READMEs to
read while waiting for my wife and daughter were shopping, and while
reading "Before-Queue" and "After-Queue" content filtering README
files I was pretty sure that I'd find in my mailbox recommendations to
use the content filters. I'll give the After-Queue Content filter a
try; it seems to be the best way to go, perhaps utilizing Bennett
Todd's SMTP content filtering framework to make the setup simpler.

Though not exacly simple for a newcomer, I find Postfix/Dovecot setup
exciting -- unlike with qmail where I often had no idea why something
worked the way it did, with Postfix/Dovecot everything can be
understood with relative ease.

Ville
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Leonardo Rodrigues Magalhães


Ville Walveranta escreveu:
> Though not exacly simple for a newcomer, I find Postfix/Dovecot setup
> exciting -- unlike with qmail where I often had no idea why something
> worked the way it did, with Postfix/Dovecot everything can be
> understood with relative ease.
>  

    Exactly same thing i tought some years ago when started migrating
things from sendmail to postfix. Things in postfix werent always easy.
And arent always easy until now, several years of postfix experience
later. But even the complex configurations are 'understandable', VERY
different of those non-human-understandable macros in sendmail.cf .....

    never used qmail ... and, because of postfix, probably (and i hope)
i never will :)

    thanks all postfix developers for bringing such a quality and
featured software for us !!

--


        Atenciosamente / Sincerily,
        Leonardo Rodrigues
        Solutti Tecnologia
        http://www.solutti.com.br

        Minha armadilha de SPAM, NÃO mandem email
        [hidden email]
        My SPAMTRAP, do not email it




Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Ville Walveranta
As discussed in "Header check and script" thread, I modified the
yammer sub in Client.pm (yes, I'm sure it's not very pretty; my Perl
skills are very rusty.. but it seems to work). It doesn't yet check
for the end of the header (i.e. for a blank line) to prevent
replacement in the body as well, but I did both the addition of
"X-Spam: yes" header and cleanup of the Subject line in the content
filter. My question is, does this violate any RFCs? The resultant
header includes the "X-Spam: yes" in between of Delivered-To and From
lines, like so:

------
Delivered-To: [hidden email]
X-Spam: yes  <------------ this was added
From: Some User <[hidden email]>
To: Ville Walveranta <[hidden email]>
Subject: This is spam!  <------------ "**SPAM**" was removed
Date: Mon, 28 Jul 2008 09:58:26 +0000 (UTC)
------

I'd rather do both in the content filter as down the line there are
likely other change actions that needs to be done, and thus the single
action that can be done with header_checks isn't very useful.

Here's the modified yammer sub:

------[begin excerpt from smtpprox MSDW/SMTP/Client.pm]------
sub yammer {
    my ($self, $fh) = (@_);
    my $spamheader = "X-Spam: yes\r\n";
    my $spam = 0;
    local (*_);
    local ($/) = "\r\n";
    while (<$fh>) {
        if ($_ =~ m/^Subject:\s*\*\*SPAM\*\*\s+/i) {
        $spam = 1;
        }
    }
    seek( $fh, 0, 0);
    if ($spam == 1) {
    $self->{sock}->print($spamheader) or die "$0: write error: $!\n";
    }
    while (<$fh>) {
        s/^\./../;
        if ($spam == 1) {
        s/^Subject:\s*\*\*SPAM\*\*\s+/Subject: /i;
        }
        $self->{sock}->print($_) or die "$0: write error: $!\n";
    }
    $self->{sock}->print(".\r\n") or die "$0: write error: $!\n";
}
------[end excerpt from smtpprox MSDW/SMTP/Client.pm]------
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Ville Walveranta
Here's the completed code that includes the "blank line check" -- the
body is neither scanned for "Subject: **SPAM**" (saves time and
prevents false positives) nor is the substitution string run against
the body (so that any occurrences of "Subject: **SPAM**" are not
touched in the body).

If someone feels like beautifying my Perl code, or optimizing the
logic, I don't mind. ;-)

Ville

------[begin excerpt from smtpprox MSDW/SMTP/Client.pm]------
sub yammer {
    my ($self, $fh) = (@_);
    my $spamheader = "X-Spam: yes\r\n";
    my $spam = 0;
    local (*_);
    local ($/) = "\r\n";
    while (<$fh>) {
        if ($_ =~ m/^Subject:\s*\*\*SPAM\*\*\s+/i) {
        $spam = 1;
        }
        if ($_ =~ m/^\s*$/) {
        last;
        }
    }
    seek( $fh, 0, 0);
    if ($spam == 1) {
    $self->{sock}->print($spamheader) or die "$0: write error: $!\n";
    }
    while (<$fh>) {
        s/^\./../;
        if ($_ =~ m/^\s*$/) {
        $spam = 0;
        }
        if ($spam == 1) {
        s/^Subject:\s*\*\*SPAM\*\*\s+/Subject: /i;
        }
        $self->{sock}->print($_) or die "$0: write error: $!\n";
    }
    $self->{sock}->print(".\r\n") or die "$0: write error: $!\n";
}
------[end excerpt from smtpprox MSDW/SMTP/Client.pm]------
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

mouss-2
Ville Walveranta wrote:

> Here's the completed code that includes the "blank line check" -- the
> body is neither scanned for "Subject: **SPAM**" (saves time and
> prevents false positives) nor is the substitution string run against
> the body (so that any occurrences of "Subject: **SPAM**" are not
> touched in the body).
>
> If someone feels like beautifying my Perl code, or optimizing the
> logic, I don't mind. ;-)
>
> Ville
>
> ------[begin excerpt from smtpprox MSDW/SMTP/Client.pm]------
> sub yammer {
>     my ($self, $fh) = (@_);
>     my $spamheader = "X-Spam: yes\r\n";
>     my $spam = 0;
>     local (*_);
>     local ($/) = "\r\n";
>     while (<$fh>) {
> if ($_ =~ m/^Subject:\s*\*\*SPAM\*\*\s+/i) {
> $spam = 1;
> }
> if ($_ =~ m/^\s*$/) {
> last;
> }
>     }
>     seek( $fh, 0, 0);
>     if ($spam == 1) {
>     $self->{sock}->print($spamheader) or die "$0: write error: $!\n";
>     }
>     while (<$fh>) {
> s/^\./../;
> if ($_ =~ m/^\s*$/) {
> $spam = 0;
> }
> if ($spam == 1) {
> s/^Subject:\s*\*\*SPAM\*\*\s+/Subject: /i;
> }
> $self->{sock}->print($_) or die "$0: write error: $!\n";
>     }
>     $self->{sock}->print(".\r\n") or die "$0: write error: $!\n";
> }


you're removing the blank line. always be careful with "last".

why do you split the while loop? you're changing the code too much...


        my $spamheader="X-Spam: Yes";
        ...
        while (<$fh>) {
          s/^\./../;
          if ($spamheader) {
                if (/^Subject:\s*\*\*SPAM\*\*\s+/i) {
                        $_ = "Subject $'";
                        $self->{sock}->print($spamheader) ...
                } elsif { /^$/) {
                        $spamheader = undef;
                }
          }
          $self->{sock}->print($_) or die "$0: write error: $!\n";
        }
        ...

BTW. I think the proxy adds \r\n by itself (and removes them when it
reads the mail). you need to check this though.

anyway, count me out now.






Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Robert Spencer-3
In reply to this post by Ville Walveranta
On 7/26/08, Ville Walveranta <[hidden email]> wrote:
<...>
> The external service tags the spam Subject
> lines with **SPAM**, and currently there is no option to change that
> to a header tag on their side. I would like to do that locally so that
> the Subjects of all of the spam messages in their spam boxes would not
> start with "**SPAM**".

Good idea, I too detest the subject line change.

> So the question is: How do I detect "**SPAM**" on the subject line,
> remove it (i.e. rewrite the Subject line), and add something like
> "X-Spam: yes" into the header of the messages whose Subject includes
> "**SPAM**"?

Can't they add "X-Spam: yes"? For a paid service they're offering you
remarkably little options.

Next question, why add it? If all your spam is coming from one source,
you can just filter on that or have the util that retrieves the spam
dump it straight into a spam folder.

--
Robert Spencer
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Ville Walveranta
In reply to this post by mouss-2
On Mon, Jul 28, 2008 at 3:08 PM, mouss <[hidden email]> wrote:
> you're removing the blank line. always be careful with "last".

No, the first pass was just to collect information (i.e. whether the
"**SPAM**" exists on the Subject line). It didn't remove the blank
line in the end of the header.

> why do you split the while loop?

Ok, I revised my version in order to not split the while loop. Same
result as your code, except that the X-Spam header is placed as the
last item of the headers (perhaps cleaner headers, if anyone cares).


my $spamheader = "X-Spam: yes\r\n";
my $check_spam = 1;
my $add_X_spam = 0;
...
while (<$fh>) {
    s/^\./../;
    if ($check_spam) {
      if (s/^Subject:\s*\*\*SPAM\*\*\s+/Subject: /i) {
        $add_X_spam = 1;
      }
      if (/^\s*$/) {
        $check_spam = 0;
      }
    }
    if ($add_X_spam && m/^\s*$/) {
      $_ = $spamheader;
      $add_X_spam = 0;
    }
    $self->{sock}->print($_) or die "$0: write error: $!\n";
}


> BTW. I think the proxy adds \r\n by itself (and removes them when it reads
> the mail). you need to check this though.

If there is no "\r\n" in the end of the spamheader, the following line
in the resultant header is concatenated (or, in above code, the blank
line and thus the separation between the header and the message body,
is lost).

> anyway, count me out now.

Ok. Thanks for advise on this issue; it helped a lot!

Ville
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Ville Walveranta
In reply to this post by Robert Spencer-3
On Mon, Jul 28, 2008 at 4:55 PM, Robert Spencer <[hidden email]> wrote:
> Can't they add "X-Spam: yes"? For a paid service they're offering you
> remarkably little options.

I'm requesting that change as it would positively identify the spam
messages. They generally seem to buffer the spam on their side and
allow users to access the spam folder though a web interface. However,
some less technical users find it easier to go looking for messages
that have been potentially mistagged as spam in a "Spam" folder in
Outlook rather than by logging to the web interface.

> Next question, why add it? If all your spam is coming from one source,
> you can just filter on that or have the util that retrieves the spam
> dump it straight into a spam folder.

Yeah, spam dump would be an option, but it would be more like a digest
that I would then have to parse and place in the Spam folder, so it
would involve work as well. Going that route there would also be a
delay with the arrival of the messages tagged as spam. If someone is
expecting an email which erroneously gets tagged as spam, at least
they will now have access to it immediately.

The only downside with this setup is, I suppose, that if someone sends
a message with a subject beginning "**SPAM**" it will erroneously go
into the spam folder. But then, if someone sends such a message in
this Age of Spam, perhaps their email deserves to end up in the spam
folder ;-).

Ville
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Robert Spencer-3
On 7/29/08, Ville Walveranta <[hidden email]> wrote:

> On Mon, Jul 28, 2008 at 4:55 PM, Robert Spencer <[hidden email]>
> wrote:
>> Can't they add "X-Spam: yes"? For a paid service they're offering you
>> remarkably little options.
>
> I'm requesting that change as it would positively identify the spam
> messages. They generally seem to buffer the spam on their side and
> allow users to access the spam folder though a web interface. However,
> some less technical users find it easier to go looking for messages
> that have been potentially mistagged as spam in a "Spam" folder in
> Outlook rather than by logging to the web interface.

Your "less technical" users are right, it takes less time to look in a
spam folder than it is to open a browser window and navigate to the
web interface. I would have found it irritating in a very short time
and some users would even given up on the task after a while.

Unfortunately I've known users who never check their spam folder, so
the spam db gets poisoned over time. I hope your has some workaround
for that.

>> Next question, why add it? If all your spam is coming from one source,
>> you can just filter on that or have the util that retrieves the spam
>> dump it straight into a spam folder.
>
> Yeah, spam dump would be an option, but it would be more like a digest
> that I would then have to parse and place in the Spam folder, so it
> would involve work as well.

There are scripts on the Net for splitting digests and you can also
use a script to turn it into a mbox mail file.

> Going that route there would also be a
> delay with the arrival of the messages tagged as spam. If someone is
> expecting an email which erroneously gets tagged as spam, at least
> they will now have access to it immediately.

Okay.

> The only downside with this setup is, I suppose, that if someone sends
> a message with a subject beginning "**SPAM**" it will erroneously go
> into the spam folder. But then, if someone sends such a message in
> this Age of Spam, perhaps their email deserves to end up in the spam
> folder ;-).

I've received legitimate mail with "**SPAM**"  in the subject line and
it wasn't due to software on my side.  I've even seen mail on a
mailing list were a clueless newbie quoted the entire message body in
his mail and every time it went through his spam filter another
"**SPAM**" was added to the subject line. Ouch!

It might help if you check for something in front of it like "Re", but
"Re" is English and if you receive mail from Europe it can land up
becoming something else.

--
Robert Spencer
Reply | Threaded
Open this post in threaded view
|

Re: Rewriting Subject line, adding an X-header?

Ville Walveranta
On Mon, Jul 28, 2008 at 7:30 PM, Robert Spencer <[hidden email]> wrote:
> Your "less technical" users are right, it takes less time to look in a
> spam folder than it is to open a browser window and navigate to the
> web interface. I would have found it irritating in a very short time
> and some users would even given up on the task after a while.

That's true of course. I just probably would not have explored this
option without some users really disliking separate path to view mail
that was tagged spam.

> Unfortunately I've known users who never check their spam folder, so
> the spam db gets poisoned over time. I hope your has some workaround
> for that.

The service(s) that I'm looking at don't seem to so much rely on user
feedback to identify spam. They use RBLs, Bayesian filtering, etc. And
obviously the users have the option to probably black/whitelist items
(and greylisting is also available).

> I've received legitimate mail with "**SPAM**"  in the subject line and
> it wasn't due to software on my side.  I've even seen mail on a
> mailing list were a clueless newbie quoted the entire message body in
> his mail and every time it went through his spam filter another
> "**SPAM**" was added to the subject line. Ouch!

For now those unfortunate emails just have to go to the spam folder.
Maybe I'll either end up with a filtering service that does offer
header tagging, or this current one I'm testing will add the feature
in.

Ville