Mail server recently became an open relay

[Rich Wales]

Hi.  My mail server (memoryalpha.richw.org), running Postfix 3.3.0,
recently started attracting open relay spam.  I thought I had done all
the appropriate things in Postfix to block open relay traffic, and I
hadn't seen any such traffic for a very long time, but suddenly I've
gotten three attacks in the last two days (plus another one a couple of
weeks ago).

I'm attaching the output of "postconf -nf".

You'll note that I'm using amavisd-new as a spam filter (which has
worked fine for a very long time).  The log info from amavisd-new
identifies the messages in question as probably coming via an open
relay, but it still passes them.  What confuses me is that I would
expect Postfix to have identified and rejected these messages during the
initial SMTP dialogue with the sender, and they should never reach
amavisd-new.

Any suggestions gratefully welcome.

Rich Wales
[hidden email]

[Wietse Venema]

Rich Wales:
> Hi.  My mail server (memoryalpha.richw.org), running Postfix 3.3.0,
> recently started attracting open relay spam.  I thought I had done all

Why do you believe that your server is an open relay, as in, it
will forward messages FROM spammers TO remote destinations.

        Wietse

I would think running an open relay test would be step one.

https://mxtoolbox.com/diagnostic.aspx


There are probably half a dozen online services that do this. Which brings me to my question: Is there an open relay test website that is considered the best? I have noticed some run multiple tests which I assume means different methods.





  Original Message  


From: [hidden email]
Sent: October 16, 2020 3:27 PM
To: [hidden email]
Reply-to: [hidden email]
Subject: Re: Mail server recently became an open relay


Rich Wales:
> Hi.  My mail server (memoryalpha.richw.org), running Postfix 3.3.0,
> recently started attracting open relay spam.  I thought I had done all

Why do you believe that your server is an open relay, as in, it
will forward messages FROM spammers TO remote destinations.

Wietse

[Rich Wales]

> Why do you believe that your server is an open relay, as in, it
> will forward messages FROM spammers TO remote destinations.
> Wietse

Because it *is* accepting messages from outsiders (spammers) and is
using my server to relay those messages to remote destinations.  It was
(and still is) my understanding that such messages should be rejected by
Postfix during the SMTP dialogue, with an error to the client saying
relay access is denied -- though I assume I inadvertently either failed
to enable this check or have somehow broken it.

I've run several open relay test sites on my server, and all of them
claim it is clean, but I have seen numerous mail queue listings with
such messages clogging up my system because the destination sites are
correctly identifying them as being relayed and won't accept them.  I
didn't think such messages were supposed to make it into the queue, but
should instead have been rejected by me with an SMTP error sent back to
the sending client.  One of these incidents, btw (two weeks ago),
created a mail queue of over 50,000 messages before I noticed it and
cleaned up the mess.  It took about a day after that for my server to
get taken off the GBUDB blacklist site.

The next time I see this happen -- could be tomorrow, could be weeks
from now, I have no idea when -- I'll gladly forward a copy of my
"mailq" output.  I deleted my earlier evidence, I'm afraid.

Rich Wales
[hidden email]

[Viktor Dukhovni]

On Oct 16, 2020, at 10:28 PM, Rich Wales <[hidden email]> wrote:
>
> The next time I see this happen -- could be tomorrow, could be weeks
> from now, I have no idea when -- I'll gladly forward a copy of my
> "mailq" output.  I deleted my earlier evidence, I'm afraid.

No "mailq" output needed.  Just the relevant entries from your logs.
Have you deleted your logs?

Also of course your configuration, per:

        http://www.postfix.org/DEBUG_README.html#mail

--
        Viktor.

[Rich Wales]

No, Viktor, I have not deleted my logs.  However, there is so much stuff
in the Postfix log (/var/log/mail.log on my system) -- including both
good e-mail messages and bad, overlapped every which-way, multiple
Postfix processes, etc. -- that I don't think I can reasonably hope for
anyone to be able to decipher even a time-delimited excerpt.

Are there any tools available to make sense of the Postfix log?  In the
DEBUG_README document, there is mention of something called postfinger,
but it appears to be missing (the ftp.wl0.org site is nonexistent).

I've enabled verbosity (smtpd -v) for the "smtpd" line in my master.cf,
in hopes that this may capture some additional detail of inbound SMTP
sessions.  Any other debugging suggestions would be welcomed.

I'll be back when I have something reasonably useful for you to look at.

Rich Wales
[hidden email]

[Viktor Dukhovni]

On Oct 16, 2020, at 11:17 PM, Rich Wales <[hidden email]> wrote:
>
> No, Viktor, I have not deleted my logs.  However, there is so much stuff
> in the Postfix log (/var/log/mail.log on my system) -- including both
> good e-mail messages and bad, overlapped every which-way, multiple
> Postfix processes, etc. -- that I don't think I can reasonably hope for
> anyone to be able to decipher even a time-delimited excerpt.

Well, of course I wasn't asking for the raw log file, just a sample of
a few problem messages...

> Are there any tools available to make sense of the Postfix log?  In the
> DEBUG_README document, there is mention of something called postfinger,
> but it appears to be missing (the ftp.wl0.org site is nonexistent).

https://github.com/vdukhovni/postfix/tree/master/postfix/auxiliary/collate

> I've enabled verbosity (smtpd -v) for the "smtpd" line in my master.cf,
> in hopes that this may capture some additional detail of inbound SMTP
> sessions.  Any other debugging suggestions would be welcomed.

No, turn that off.  It just makes it much harder to see the important
bits.

--
        Viktor.

[Bill Cole-3]

On 16 Oct 2020, at 18:20, Rich Wales wrote:

> Hi.  My mail server (memoryalpha.richw.org), running Postfix 3.3.0,
> recently started attracting open relay spam.  I thought I had done all
> the appropriate things in Postfix to block open relay traffic, and I
> hadn't seen any such traffic for a very long time, but suddenly I've
> gotten three attacks in the last two days (plus another one a couple
> of
> weeks ago).
>
> I'm attaching the output of "postconf -nf".
>
> You'll note that I'm using amavisd-new as a spam filter (which has
> worked fine for a very long time).  The log info from amavisd-new
> identifies the messages in question as probably coming via an open
> relay, but it still passes them.  What confuses me is that I would
> expect Postfix to have identified and rejected these messages during
> the
> initial SMTP dialogue with the sender, and they should never reach
> amavisd-new.
>
> Any suggestions gratefully welcome.

Based on your config and descriptions, it smells like a compromised
account being used to pump mail through your submission service. A full
set of log lines for one of the messages should reveal that. The
master.cf lines for smtpd and submission would also help.


--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

[Rich Wales]

On 2020-10-16 21:16, Bill Cole wrote:

> Based on your config and descriptions, it smells like a compromised
> account being used to pump mail through your submission service. A full
> set of log lines for one of the messages should reveal that. The
> master.cf lines for smtpd and submission would also help.

Thanks.  I'll look into this.

Rich Wales
[hidden email]

[Wietse Venema]

Rich Wales:

> > Why do you believe that your server is an open relay, as in, it
> > will forward messages FROM spammers TO remote destinations.
> > Wietse
>
> Because it *is* accepting messages from outsiders (spammers) and is
> using my server to relay those messages to remote destinations.  It was
> (and still is) my understanding that such messages should be rejected by
> Postfix during the SMTP dialogue, with an error to the client saying
> relay access is denied -- though I assume I inadvertently either failed
> to enable this check or have somehow broken it.

Show evidence (logging) and turn of verbose logging.

        Wietse

[Rich Wales]

Show evidence (logging) and turn of verbose logging. Wietse

OK, here is the message header for one of the spam e-mails (which did not get deleted during my mass cleanup efforts because a copy was saved in my amavisd-new quarantine database):

X-Envelope-From: [hidden email]
X-Envelope-To: [hidden email]
X-Envelope-To-Blocked: [hidden email]
X-Quarantine-ID: <D0t9j6VORyNH>
X-Spam-Flag: YES
X-Spam-Score: 5.488
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.488 tag=x tag2=4.5 kill=4.5 tests=[ALL_TRUSTED=-1,
    BAYES_00=-0.8, BAYES_00_RAZOR=0.8, FSL_BULK_SIG=0.001,
    HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, MIME_HTML_ONLY=1,
    RAZOR2_CF_RANGE_51_100=1.886, RAZOR2_CHECK=3, SUBJ_ALL_CAPS=0.5]
    autolearn=disabled
Received: from memoryalpha.richw.org ([127.0.0.1])
    by localhost (memoryalpha.richw.org [127.0.0.1]) (amavisd-new, port 10024)
    with ESMTP id D0t9j6VORyNH for [hidden email];
    Thu, 15 Oct 2020 14:48:06 -0700 (PDT)
Received: from [154.91.34.144] (localhost [127.0.0.1])
    by memoryalpha.richw.org (Postfix) with ESMTP id 4CC2vp5WmFz87Jy
    for [hidden email]; Thu, 15 Oct 2020 14:48:06 -0700 (PDT)
From: ScotiaInfoAlerts Communications [hidden email]
Content-Type: text/html
To: [hidden email]
Content-Transfer-Encoding: 7bit
X-Read: MailingV.4371434NJ
Subject: CONFIRMATION REQUIRED - CUSTOMER ID: SS437143
Message-Id: [hidden email]
Date: Thu, 15 Oct 2020 14:48:06 -0700 (PDT)

Note that the chronologically last "Received:" line says the message was received from 154.91.34.144 -- an IP address with no hostname, in a range assigned (according to WHOIS) to Hong Kong.

I'm not sure what the parenthesized reference to "(localhost [127.0.0.1])" in this "Received:" line means.  Does this mean that the client host falsely identified itself with "HELO localhost"?

Now, here are the lines in my mail log corresponding to this message:

Oct 15 14:48:06 memoryalpha postfix/postscreen[18030]: CONNECT from [127.0.0.1]:52138 to [127.0.0.1]:25
Oct 15 14:48:06 memoryalpha postfix/postscreen[18030]: WHITELISTED [127.0.0.1]:52138
Oct 15 14:48:06 memoryalpha postfix/smtpd[6414]: connect from localhost[127.0.0.1]
Oct 15 14:48:06 memoryalpha postfix/smtpd[6414]: 4CC2vp5WmFz87Jy: client=localhost[127.0.0.1]
Oct 15 14:48:06 memoryalpha postfix/cleanup[7158]: 4CC2vp5WmFz87Jy: message-id=[hidden email]
Oct 15 14:48:06 memoryalpha postfix/qmgr[26090]: 4CC2vp5WmFz87Jy: from=[hidden email], size=9292, nrcpt=1 (queue active)
Oct 15 14:48:08 memoryalpha amavis[8375]: (08375-11) Blocked SPAM {DiscardedOpenRelay,Quarantined}, [127.0.0.1]:52138 [hidden email] -> [hidden email], quarantine: D0t9j6VORyNH, Queue-ID: 4CC2vp5WmFz87Jy, Message-ID: [hidden email], mail_id: D0t9j6VORyNH, Hits: 5.488, size: 9276, 1290 ms
Oct 15 14:48:08 memoryalpha postfix/smtp[8703]: 4CC2vp5WmFz87Jy: to=[hidden email], relay=127.0.0.1[127.0.0.1]:10024, delay=1.5, delays=0.19/0/0/1.3, dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=08375-11 - spam)

(plus some more amavis-specific log lines which I assume people here don't care about).

Note here that the "client=" line (first line in the above) gives the sending host as "localhost[127.0.0.1]".  I know that Postfix connects to amavisd-new via [127.0.0.1]:10024, so I can understand references to this IP address in and after the "amavis[8375]" log line.  But why does the very first line (smtpd[6414], before any amavis processing) have localhost as the client?

If my server is getting confused and thinks the message in question originally came from localhost, I can easily understand why the open relay checks are being skipped, since my main.cf file includes 127.0.0.0/8 amongst the "mynetworks" values.

So, am I doing something wrong that is allowing spammers to say "HELO localhost" and get away with it?  Or is something else causing my Postfix to think the e-mail came inbound from localhost even though it didn't?

Other, valid e-mail coming into and delivered via this server retain the sending host's identity, btw, and are not rewritten to claim they came from localhost.

Rich Wales
[hidden email]

[Rich Wales]

Sorry, when I said "chronologically last 'Received:' line" in my earlier e-mail, I meant to say "chronologically first (physically last)".  Mea culpa.

Rich Wales
[hidden email]

[Viktor Dukhovni]

On Sat, Oct 17, 2020 at 08:41:25PM -0700, Rich Wales wrote:

>     Received: from memoryalpha.richw.org ([127.0.0.1])
>         by localhost (memoryalpha.richw.org [127.0.0.1]) (amavisd-new, port 10024)
>         with ESMTP id D0t9j6VORyNH for <[hidden email]>;
>         Thu, 15 Oct 2020 14:48:06 -0700 (PDT)
>     Received: from [154.91.34.144] (localhost [127.0.0.1])
>         by memoryalpha.richw.org (Postfix) with ESMTP id 4CC2vp5WmFz87Jy
>         for <[hidden email]>; Thu, 15 Oct 2020 14:48:06 -0700 (PDT)
>     From: ScotiaInfoAlerts Communications <[hidden email]>
>     Message-Id: <[hidden email]>
>
> Note that the chronologically last "Received:" line says the message was
> received from 154.91.34.144 -- an IP address with no hostname, in a
> range assigned (according to WHOIS) to Hong Kong.

No, it says no such thing.  It says the EHLO name was [154.91.34.144],
the client IP was however 127.0.0.1.  It seems you have some sort of
proxy or NAT in place that masks the real external IP address, making
all connections appear to originate from 127.0.0.1.  That would sure
explain your spam innundation problem.

> I'm not sure what the parenthesized reference to "(localhost
> [127.0.0.1])" in this "Received:" line means.  Does this mean that the
> client host falsely identified itself with "HELO localhost"?

No, the other way around.

>     Oct 15 14:48:06 memoryalpha postfix/postscreen[18030]: CONNECT from
>     [127.0.0.1]:52138 to [127.0.0.1]:25

Same NAT or proxy issue.

--
    Viktor.

[Rich Wales]

> No, it says no such thing.  It says the EHLO name was [154.91.34.144],
> the client IP was however 127.0.0.1.  It seems you have some sort of
> proxy or NAT in place that masks the real external IP address, making
> all connections appear to originate from 127.0.0.1.  That would sure
> explain your spam innundation problem.

Thanks.  I was actually thinking something of the sort myself -- my
server is indeed behind a separate firewall appliance.

However, other e-mail (such as your recent reply to my inquiry) is NOT
exhibiting this same NAT/proxy addressing problem.  The relevant
"Received:" line in my copy of your reply says the following (with line
wrapping to make it legible in an ASCII environment):

    Received: from english-breakfast.cloud9.net
        (english-breakfast.cloud9.net [168.100.1.7])
        (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by memoryalpha.richw.org (Postfix)
        with ESMTPS id 4CDQt72CNxz7t88
        for <[hidden email]>; Sat, 17 Oct 2020 20:51:27 -0700 (PDT)

Your e-mail (along with lots and lots of valid e-mail) appears to be
entering my server via exactly the same NAT/proxy path as the spam did.

I'll continue searching for any possible security hole on my firewall
appliance, though.

Rich Wales
[hidden email]

[Viktor Dukhovni]

On Sat, Oct 17, 2020 at 09:14:50PM -0700, Rich Wales wrote:

> Thanks.  I was actually thinking something of the sort myself -- my
> server is indeed behind a separate firewall appliance.
>
> However, other e-mail (such as your recent reply to my inquiry) is NOT
> exhibiting this same NAT/proxy addressing problem.  The relevant
> "Received:" line in my copy of your reply says the following (with line
> wrapping to make it legible in an ASCII environment):

Well, that shows that a proxy is the more likely scenario, some process
listening on a non-loopback IP that passes SMTP connections through to
127.0.0.1, or a NAT rule in your iptables...

> I'll continue searching for any possible security hole on my firewall
> appliance, though.

The firewall appliance (if a separate device) cannot make connections
appear to originate from 127.0.0.1, only something running on your
machine itself can do that.  So not much point looking there.

--
    Viktor.

[John Fawcett]

On 18/10/2020 06:32, Viktor Dukhovni wrote:

> On Sat, Oct 17, 2020 at 09:14:50PM -0700, Rich Wales wrote:
>
>> Thanks.  I was actually thinking something of the sort myself -- my
>> server is indeed behind a separate firewall appliance.
>>
>> However, other e-mail (such as your recent reply to my inquiry) is NOT
>> exhibiting this same NAT/proxy addressing problem.  The relevant
>> "Received:" line in my copy of your reply says the following (with line
>> wrapping to make it legible in an ASCII environment):
> Well, that shows that a proxy is the more likely scenario, some process
> listening on a non-loopback IP that passes SMTP connections through to
> 127.0.0.1, or a NAT rule in your iptables...
>
>> I'll continue searching for any possible security hole on my firewall
>> appliance, though.
> The firewall appliance (if a separate device) cannot make connections
> appear to originate from 127.0.0.1, only something running on your
> machine itself can do that.  So not much point looking there.
>

One thing I would suggest looking at is if there is a web server running
on the same host it may be allowing email to be injected into postfix
via smtp on the loopback interface using some scripting language like
php or others.

John

[Rich Wales]

John Fawcett wrote:

> One thing I would suggest looking at is if there is a web server running
> on the same host it may be allowing email to be injected into postfix
> via smtp on the loopback interface using some scripting language like
> php or others.

I suppose that's possible.

I spent some time last night cleaning up old stuff from the server in
question -- and also rebooting the box for good measure -- so the
problem *might* just go away at this point.

Before I can say anything more about this, unfortunately, I'll probably
need to wait for another incident similar to the preceding ones, and try
to capture more evidence while the problem is ongoing.  If it never
happens again, then maybe it was the fault of an old PHP web page which
I have removed.

If the problem were in fact due to a hijacked PHP page, btw, would this
necessarily require the page to be using e-mail or TCP connections
already for its own legitimate purposes, but being co-opted by a hacker
to nefarious ends?  Or could *any* PHP script theoretically be infected
in a way that would cause this misbehaviour?

Rich Wales
[hidden email]

[John Fawcett]

On 19/10/2020 20:50, Rich Wales wrote:

> John Fawcett wrote:
>
>> One thing I would suggest looking at is if there is a web server running
>> on the same host it may be allowing email to be injected into postfix
>> via smtp on the loopback interface using some scripting language like
>> php or others.
> I suppose that's possible.
>
> I spent some time last night cleaning up old stuff from the server in
> question -- and also rebooting the box for good measure -- so the
> problem *might* just go away at this point.
>
> Before I can say anything more about this, unfortunately, I'll probably
> need to wait for another incident similar to the preceding ones, and try
> to capture more evidence while the problem is ongoing.  If it never
> happens again, then maybe it was the fault of an old PHP web page which
> I have removed.
>
> If the problem were in fact due to a hijacked PHP page, btw, would this
> necessarily require the page to be using e-mail or TCP connections
> already for its own legitimate purposes, but being co-opted by a hacker
> to nefarious ends?  Or could *any* PHP script theoretically be infected
> in a way that would cause this misbehaviour?
>
> Rich Wales
> [hidden email]

Sorry not to be able to give a definitive answer. Typical mail injection
via php will use a script that already calls the php mail function or
similar functions that open the smtp connection. But there are other
attack vectors that are possible that allow hackers to gain the
privileges of the web server user. The web server user is often allowed
to inject mail to localhost without any authentication (under the
permit_mynetworks syntax in postfix main.cf).

John

[Jaroslaw Rafa]

Dnia 19.10.2020 o godz. 21:12:20 John Fawcett pisze:
> Sorry not to be able to give a definitive answer. Typical mail injection
> via php will use a script that already calls the php mail function or
> similar functions that open the smtp connection. But there are other
> attack vectors that are possible that allow hackers to gain the
> privileges of the web server user.

Very often hackers abuse web pages that allow users to upload files to the
web server. If the input is not correctly sanitized, it may be possible to
upload an arbitrary php script and get it executed.

There were multiple attacks based on this scenario.
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Bob Proulx]

Rich Wales wrote:
> If the problem were in fact due to a hijacked PHP page, btw, would this
> necessarily require the page to be using e-mail or TCP connections
> already for its own legitimate purposes, but being co-opted by a hacker
> to nefarious ends?  Or could *any* PHP script theoretically be infected
> in a way that would cause this misbehaviour?

*If* the host is running a site, such as Wordpress but there are also
many other possibilities, and if it is not absolutely up to date with
security upgrades current as of *TODAY*, then it is very likely that
the site has been compromised.

That's just the history of WP and other similar frameworks!  They are
allowed to do brain surgery on themselves without restriction and they
consist of a community of thousands and thousands of inexperienced
developers all submitting modules without a security focus.  When I
read John Fawcett's suggestion that it might be a web server
compromise I thought immediately, "Oh good suggestion!", since that is
so often a typical compromise case!

The default PHP "mail()" method sends mail by using the system's
/usr/sbin/sendmail interface rather than SMTP.

    https://www.php.net/manual/en/mail.requirements.php
    https://www.php.net/manual/en/function.mail.php

Bob