Hi. My mail server (memoryalpha.richw.org), running Postfix 3.3.0,
recently started attracting open relay spam. I thought I had done all the appropriate things in Postfix to block open relay traffic, and I hadn't seen any such traffic for a very long time, but suddenly I've gotten three attacks in the last two days (plus another one a couple of weeks ago). I'm attaching the output of "postconf -nf". You'll note that I'm using amavisd-new as a spam filter (which has worked fine for a very long time). The log info from amavisd-new identifies the messages in question as probably coming via an open relay, but it still passes them. What confuses me is that I would expect Postfix to have identified and rejected these messages during the initial SMTP dialogue with the sender, and they should never reach amavisd-new. Any suggestions gratefully welcome. Rich Wales [hidden email] |
Rich Wales:
> Hi. My mail server (memoryalpha.richw.org), running Postfix 3.3.0, > recently started attracting open relay spam. I thought I had done all Why do you believe that your server is an open relay, as in, it will forward messages FROM spammers TO remote destinations. Wietse |
I would think running an open relay test would be step one.
https://mxtoolbox.com/diagnostic.aspx There are probably half a dozen online services that do this. Which brings me to my question: Is there an open relay test website that is considered the best? I have noticed some run multiple tests which I assume means different methods. Original Message From: [hidden email] Sent: October 16, 2020 3:27 PM To: [hidden email] Reply-to: [hidden email] Subject: Re: Mail server recently became an open relay Rich Wales: > Hi. My mail server (memoryalpha.richw.org), running Postfix 3.3.0, > recently started attracting open relay spam. I thought I had done all Why do you believe that your server is an open relay, as in, it will forward messages FROM spammers TO remote destinations. Wietse |
> Why do you believe that your server is an open relay, as in, it
> will forward messages FROM spammers TO remote destinations. > Wietse Because it *is* accepting messages from outsiders (spammers) and is using my server to relay those messages to remote destinations. It was (and still is) my understanding that such messages should be rejected by Postfix during the SMTP dialogue, with an error to the client saying relay access is denied -- though I assume I inadvertently either failed to enable this check or have somehow broken it. I've run several open relay test sites on my server, and all of them claim it is clean, but I have seen numerous mail queue listings with such messages clogging up my system because the destination sites are correctly identifying them as being relayed and won't accept them. I didn't think such messages were supposed to make it into the queue, but should instead have been rejected by me with an SMTP error sent back to the sending client. One of these incidents, btw (two weeks ago), created a mail queue of over 50,000 messages before I noticed it and cleaned up the mess. It took about a day after that for my server to get taken off the GBUDB blacklist site. The next time I see this happen -- could be tomorrow, could be weeks from now, I have no idea when -- I'll gladly forward a copy of my "mailq" output. I deleted my earlier evidence, I'm afraid. Rich Wales [hidden email] |
On Oct 16, 2020, at 10:28 PM, Rich Wales <[hidden email]> wrote:
> > The next time I see this happen -- could be tomorrow, could be weeks > from now, I have no idea when -- I'll gladly forward a copy of my > "mailq" output. I deleted my earlier evidence, I'm afraid. No "mailq" output needed. Just the relevant entries from your logs. Have you deleted your logs? Also of course your configuration, per: http://www.postfix.org/DEBUG_README.html#mail -- Viktor. |
No, Viktor, I have not deleted my logs. However, there is so much stuff
in the Postfix log (/var/log/mail.log on my system) -- including both good e-mail messages and bad, overlapped every which-way, multiple Postfix processes, etc. -- that I don't think I can reasonably hope for anyone to be able to decipher even a time-delimited excerpt. Are there any tools available to make sense of the Postfix log? In the DEBUG_README document, there is mention of something called postfinger, but it appears to be missing (the ftp.wl0.org site is nonexistent). I've enabled verbosity (smtpd -v) for the "smtpd" line in my master.cf, in hopes that this may capture some additional detail of inbound SMTP sessions. Any other debugging suggestions would be welcomed. I'll be back when I have something reasonably useful for you to look at. Rich Wales [hidden email] |
On Oct 16, 2020, at 11:17 PM, Rich Wales <[hidden email]> wrote:
> > No, Viktor, I have not deleted my logs. However, there is so much stuff > in the Postfix log (/var/log/mail.log on my system) -- including both > good e-mail messages and bad, overlapped every which-way, multiple > Postfix processes, etc. -- that I don't think I can reasonably hope for > anyone to be able to decipher even a time-delimited excerpt. Well, of course I wasn't asking for the raw log file, just a sample of a few problem messages... > Are there any tools available to make sense of the Postfix log? In the > DEBUG_README document, there is mention of something called postfinger, > but it appears to be missing (the ftp.wl0.org site is nonexistent). https://github.com/vdukhovni/postfix/tree/master/postfix/auxiliary/collate > I've enabled verbosity (smtpd -v) for the "smtpd" line in my master.cf, > in hopes that this may capture some additional detail of inbound SMTP > sessions. Any other debugging suggestions would be welcomed. No, turn that off. It just makes it much harder to see the important bits. -- Viktor. |
In reply to this post by Rich Wales
On 16 Oct 2020, at 18:20, Rich Wales wrote:
> Hi. My mail server (memoryalpha.richw.org), running Postfix 3.3.0, > recently started attracting open relay spam. I thought I had done all > the appropriate things in Postfix to block open relay traffic, and I > hadn't seen any such traffic for a very long time, but suddenly I've > gotten three attacks in the last two days (plus another one a couple > of > weeks ago). > > I'm attaching the output of "postconf -nf". > > You'll note that I'm using amavisd-new as a spam filter (which has > worked fine for a very long time). The log info from amavisd-new > identifies the messages in question as probably coming via an open > relay, but it still passes them. What confuses me is that I would > expect Postfix to have identified and rejected these messages during > the > initial SMTP dialogue with the sender, and they should never reach > amavisd-new. > > Any suggestions gratefully welcome. Based on your config and descriptions, it smells like a compromised account being used to pump mail through your submission service. A full set of log lines for one of the messages should reveal that. The master.cf lines for smtpd and submission would also help. -- Bill Cole [hidden email] or [hidden email] (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire |
On 2020-10-16 21:16, Bill Cole wrote:
> Based on your config and descriptions, it smells like a compromised > account being used to pump mail through your submission service. A full > set of log lines for one of the messages should reveal that. The > master.cf lines for smtpd and submission would also help. Thanks. I'll look into this. Rich Wales [hidden email] |
In reply to this post by Rich Wales
Rich Wales:
> > Why do you believe that your server is an open relay, as in, it > > will forward messages FROM spammers TO remote destinations. > > Wietse > > Because it *is* accepting messages from outsiders (spammers) and is > using my server to relay those messages to remote destinations. It was > (and still is) my understanding that such messages should be rejected by > Postfix during the SMTP dialogue, with an error to the client saying > relay access is denied -- though I assume I inadvertently either failed > to enable this check or have somehow broken it. Show evidence (logging) and turn of verbose logging. Wietse |
Show evidence (logging) and turn of verbose logging. Wietse OK, here is the message header for one of the spam e-mails (which did not get deleted during my mass cleanup efforts because a copy was saved in my amavisd-new quarantine database): X-Envelope-From: [hidden email]Note that the chronologically last "Received:" line says the message was received from 154.91.34.144 -- an IP address with no hostname, in a range assigned (according to WHOIS) to Hong Kong. I'm not sure what the parenthesized reference to "(localhost [127.0.0.1])" in this "Received:" line means. Does this mean that the client host falsely identified itself with "HELO localhost"? Now, here are the lines in my mail log corresponding to this message: Oct 15 14:48:06 memoryalpha postfix/postscreen[18030]: CONNECT from [127.0.0.1]:52138 to [127.0.0.1]:25(plus some more amavis-specific log lines which I assume people here don't care about). Note here that the "client=" line (first line in the above) gives the sending host as "localhost[127.0.0.1]". I know that Postfix connects to amavisd-new via [127.0.0.1]:10024, so I can understand references to this IP address in and after the "amavis[8375]" log line. But why does the very first line (smtpd[6414], before any amavis processing) have localhost as the client? If my server is getting confused and thinks the message in question originally came from localhost, I can easily understand why the open relay checks are being skipped, since my main.cf file includes 127.0.0.0/8 amongst the "mynetworks" values. So, am I doing something wrong that is allowing spammers to say "HELO localhost" and get away with it? Or is something else causing my Postfix to think the e-mail came inbound from localhost even though it didn't? Other, valid e-mail coming into and delivered via this server retain the sending host's identity, btw, and are not rewritten to claim they came from localhost. Rich Wales [hidden email] |
Sorry, when I said "chronologically last 'Received:'
line" in my earlier e-mail, I meant to say "chronologically first
(physically last)". Mea culpa.
Rich Wales [hidden email] |
In reply to this post by Rich Wales
On Sat, Oct 17, 2020 at 08:41:25PM -0700, Rich Wales wrote:
> Received: from memoryalpha.richw.org ([127.0.0.1]) > by localhost (memoryalpha.richw.org [127.0.0.1]) (amavisd-new, port 10024) > with ESMTP id D0t9j6VORyNH for <[hidden email]>; > Thu, 15 Oct 2020 14:48:06 -0700 (PDT) > Received: from [154.91.34.144] (localhost [127.0.0.1]) > by memoryalpha.richw.org (Postfix) with ESMTP id 4CC2vp5WmFz87Jy > for <[hidden email]>; Thu, 15 Oct 2020 14:48:06 -0700 (PDT) > From: ScotiaInfoAlerts Communications <[hidden email]> > Message-Id: <[hidden email]> > > Note that the chronologically last "Received:" line says the message was > received from 154.91.34.144 -- an IP address with no hostname, in a > range assigned (according to WHOIS) to Hong Kong. No, it says no such thing. It says the EHLO name was [154.91.34.144], the client IP was however 127.0.0.1. It seems you have some sort of proxy or NAT in place that masks the real external IP address, making all connections appear to originate from 127.0.0.1. That would sure explain your spam innundation problem. > I'm not sure what the parenthesized reference to "(localhost > [127.0.0.1])" in this "Received:" line means. Does this mean that the > client host falsely identified itself with "HELO localhost"? No, the other way around. > Oct 15 14:48:06 memoryalpha postfix/postscreen[18030]: CONNECT from > [127.0.0.1]:52138 to [127.0.0.1]:25 Same NAT or proxy issue. -- Viktor. |
> No, it says no such thing. It says the EHLO name was [154.91.34.144], > the client IP was however 127.0.0.1. It seems you have some sort of > proxy or NAT in place that masks the real external IP address, making > all connections appear to originate from 127.0.0.1. That would sure > explain your spam innundation problem. Thanks. I was actually thinking something of the sort myself -- my server is indeed behind a separate firewall appliance. However, other e-mail (such as your recent reply to my inquiry) is NOT exhibiting this same NAT/proxy addressing problem. The relevant "Received:" line in my copy of your reply says the following (with line wrapping to make it legible in an ASCII environment): Received: from english-breakfast.cloud9.net (english-breakfast.cloud9.net [168.100.1.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by memoryalpha.richw.org (Postfix) with ESMTPS id 4CDQt72CNxz7t88 for <[hidden email]>; Sat, 17 Oct 2020 20:51:27 -0700 (PDT) Your e-mail (along with lots and lots of valid e-mail) appears to be entering my server via exactly the same NAT/proxy path as the spam did. I'll continue searching for any possible security hole on my firewall appliance, though. Rich Wales [hidden email] |
On Sat, Oct 17, 2020 at 09:14:50PM -0700, Rich Wales wrote:
> Thanks. I was actually thinking something of the sort myself -- my > server is indeed behind a separate firewall appliance. > > However, other e-mail (such as your recent reply to my inquiry) is NOT > exhibiting this same NAT/proxy addressing problem. The relevant > "Received:" line in my copy of your reply says the following (with line > wrapping to make it legible in an ASCII environment): Well, that shows that a proxy is the more likely scenario, some process listening on a non-loopback IP that passes SMTP connections through to 127.0.0.1, or a NAT rule in your iptables... > I'll continue searching for any possible security hole on my firewall > appliance, though. The firewall appliance (if a separate device) cannot make connections appear to originate from 127.0.0.1, only something running on your machine itself can do that. So not much point looking there. -- Viktor. |
On 18/10/2020 06:32, Viktor Dukhovni wrote:
> On Sat, Oct 17, 2020 at 09:14:50PM -0700, Rich Wales wrote: > >> Thanks. I was actually thinking something of the sort myself -- my >> server is indeed behind a separate firewall appliance. >> >> However, other e-mail (such as your recent reply to my inquiry) is NOT >> exhibiting this same NAT/proxy addressing problem. The relevant >> "Received:" line in my copy of your reply says the following (with line >> wrapping to make it legible in an ASCII environment): > Well, that shows that a proxy is the more likely scenario, some process > listening on a non-loopback IP that passes SMTP connections through to > 127.0.0.1, or a NAT rule in your iptables... > >> I'll continue searching for any possible security hole on my firewall >> appliance, though. > The firewall appliance (if a separate device) cannot make connections > appear to originate from 127.0.0.1, only something running on your > machine itself can do that. So not much point looking there. > on the same host it may be allowing email to be injected into postfix via smtp on the loopback interface using some scripting language like php or others. John |
John Fawcett wrote:
> One thing I would suggest looking at is if there is a web server running > on the same host it may be allowing email to be injected into postfix > via smtp on the loopback interface using some scripting language like > php or others. I suppose that's possible. I spent some time last night cleaning up old stuff from the server in question -- and also rebooting the box for good measure -- so the problem *might* just go away at this point. Before I can say anything more about this, unfortunately, I'll probably need to wait for another incident similar to the preceding ones, and try to capture more evidence while the problem is ongoing. If it never happens again, then maybe it was the fault of an old PHP web page which I have removed. If the problem were in fact due to a hijacked PHP page, btw, would this necessarily require the page to be using e-mail or TCP connections already for its own legitimate purposes, but being co-opted by a hacker to nefarious ends? Or could *any* PHP script theoretically be infected in a way that would cause this misbehaviour? Rich Wales [hidden email] |
On 19/10/2020 20:50, Rich Wales wrote:
> John Fawcett wrote: > >> One thing I would suggest looking at is if there is a web server running >> on the same host it may be allowing email to be injected into postfix >> via smtp on the loopback interface using some scripting language like >> php or others. > I suppose that's possible. > > I spent some time last night cleaning up old stuff from the server in > question -- and also rebooting the box for good measure -- so the > problem *might* just go away at this point. > > Before I can say anything more about this, unfortunately, I'll probably > need to wait for another incident similar to the preceding ones, and try > to capture more evidence while the problem is ongoing. If it never > happens again, then maybe it was the fault of an old PHP web page which > I have removed. > > If the problem were in fact due to a hijacked PHP page, btw, would this > necessarily require the page to be using e-mail or TCP connections > already for its own legitimate purposes, but being co-opted by a hacker > to nefarious ends? Or could *any* PHP script theoretically be infected > in a way that would cause this misbehaviour? > > Rich Wales > [hidden email] Sorry not to be able to give a definitive answer. Typical mail injection via php will use a script that already calls the php mail function or similar functions that open the smtp connection. But there are other attack vectors that are possible that allow hackers to gain the privileges of the web server user. The web server user is often allowed to inject mail to localhost without any authentication (under the permit_mynetworks syntax in postfix main.cf). John |
Dnia 19.10.2020 o godz. 21:12:20 John Fawcett pisze:
> Sorry not to be able to give a definitive answer. Typical mail injection > via php will use a script that already calls the php mail function or > similar functions that open the smtp connection. But there are other > attack vectors that are possible that allow hackers to gain the > privileges of the web server user. Very often hackers abuse web pages that allow users to upload files to the web server. If the input is not correctly sanitized, it may be possible to upload an arbitrary php script and get it executed. There were multiple attacks based on this scenario. -- Regards, Jaroslaw Rafa [hidden email] -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." |
In reply to this post by Rich Wales
Rich Wales wrote:
> If the problem were in fact due to a hijacked PHP page, btw, would this > necessarily require the page to be using e-mail or TCP connections > already for its own legitimate purposes, but being co-opted by a hacker > to nefarious ends? Or could *any* PHP script theoretically be infected > in a way that would cause this misbehaviour? *If* the host is running a site, such as Wordpress but there are also many other possibilities, and if it is not absolutely up to date with security upgrades current as of *TODAY*, then it is very likely that the site has been compromised. That's just the history of WP and other similar frameworks! They are allowed to do brain surgery on themselves without restriction and they consist of a community of thousands and thousands of inexperienced developers all submitting modules without a security focus. When I read John Fawcett's suggestion that it might be a web server compromise I thought immediately, "Oh good suggestion!", since that is so often a typical compromise case! The default PHP "mail()" method sends mail by using the system's /usr/sbin/sendmail interface rather than SMTP. https://www.php.net/manual/en/mail.requirements.php https://www.php.net/manual/en/function.mail.php Bob |
Free forum by Nabble | Edit this page |