Limiting HELO spoofing in Postfix?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Limiting HELO spoofing in Postfix?

Rich Wales
This question is kind-of related to my recent open relay problem, which
at the moment seems most likely caused by a hacker invading my mail
server (possibly through a web service running on the same box) and
injecting fraudulent messages into my server directly via the localhost
interface.

Postfix "sort-of" recognizes this, because each of the fake messages
appears to contain a "Received:" header line (created by my Postfix)
saying the sender sent out a host name or address in its HELO, but my
Postfix knew that the message was really coming from 127.0.0.1.

My question is, is there any configuration option for Postfix to reject
mail in a situation where the sender is clearly spoofing its identity
like this during the SMTP conversation?

I suppose it might possibly be good enough in this situation to use an
option that rejects inbound mail claiming to be from some outside host
when in fact it's really coming from localhost.  I've looked at the
various smtpd_helo_restrictions, but it's not obvious to me which (if
any) of these can do what I want.

Obviously, this option (if it exists) would need to be used judiciously
so as not to block legitimate e-mail passing through milters and such.
But what I want to know is if any such option exists at all.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Wietse Venema
Rich Wales:

> This question is kind-of related to my recent open relay problem, which
> at the moment seems most likely caused by a hacker invading my mail
> server (possibly through a web service running on the same box) and
> injecting fraudulent messages into my server directly via the localhost
> interface.
>
> Postfix "sort-of" recognizes this, because each of the fake messages
> appears to contain a "Received:" header line (created by my Postfix)
> saying the sender sent out a host name or address in its HELO, but my
> Postfix knew that the message was really coming from 127.0.0.1.
>
> My question is, is there any configuration option for Postfix to reject
> mail in a situation where the sender is clearly spoofing its identity
> like this during the SMTP conversation?
>
> I suppose it might possibly be good enough in this situation to use an
> option that rejects inbound mail claiming to be from some outside host
> when in fact it's really coming from localhost.  I've looked at the
> various smtpd_helo_restrictions, but it's not obvious to me which (if
> any) of these can do what I want.
>
> Obviously, this option (if it exists) would need to be used judiciously
> so as not to block legitimate e-mail passing through milters and such.
> But what I want to know is if any such option exists at all.

I would suggest that you fix the exploited script. Look for time
stamps that appear in both web server logging and Postfix logging.

1) Use awk or perl to extract time stamps from web server logs and
normalize them so that they look the same. Produce sorted unqiue
timestamps with "sort -u" and save the result to file. Ditto for
time stamps from Postfix logs.

2) Look for time stamps that appear in both files.

    comm -12 webserver-timestamps postfix-timestamps

3) Look up the corresponding records in webserver logs and Postfix logs.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales

> I would suggest that you fix the exploited script. Look for time
> stamps that appear in both web server logging and Postfix logging.

Thanks, Wietse.  That would obviously be the best approach, if it
worked, but I tried it and (so far at least) haven't been able to find
any matching entries.

I did find some generally suspicious things in my web server logs --
including lots of clients looking for the following item:

        /nette.micro?callback=shell_exec&cmd=ifconfig

but there isn't any /nette.micro anywhere on my server, and all these
GET requests failed with 404 or 302 SMTP response codes.  I don't
currently think these have anything to do with the recent attacks on my
server, and I'm mentioning them just in case anyone on this list might
recognize this kind of thing.

As I said earlier, I did clean out a bunch of old, obsolete, possibly
buggy content from my web server's directories, and I rebooted the box
just for good measure.  I haven't seen any more occurrences of the
problem so far, but I am keeping a close eye on the server, and if I do
see another flood of apparent open-relay messages, I hope I can capture
more (and more useful) information about the problem this time.

In the meantime, I would still like to know if there is any way to make
Postfix check for and reject messages which are coming from localhost
but which are falsely claiming (in the HELO / EHLO) to be originating
from some outside host.  I realize this wouldn't solve every problem,
but it seems to me like a very useful thing for Postfix to be able to
do.  If this option is intentionally not and most likely never will be
part of Postfix, I would be grateful for an explanation of why it is not
actually helpful, even if it might appear to be at first glance.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Bob Proulx
Rich Wales wrote:
> I did find some generally suspicious things in my web server logs --
> including lots of clients looking for the following item:
>
>         /nette.micro?callback=shell_exec&cmd=ifconfig
>
> but there isn't any /nette.micro anywhere on my server, and all these
> GET requests failed with 404 or 302 SMTP response codes.

If the web server logs said it was 404 then that is an HTTP 404 return
code and not an SMTP 404 return code since it is in the web server
logs not the smtp server logs.

    https://en.wikipedia.org/wiki/HTTP_404

That's the normal response to a probe.  There are a lot of probes of
web servers on the net.

    rwp@havoc:/var/log$ awk '$9=="404"' /var/log/nginx/access.log | wc -l
    1124

That's one day of log file on one of my systems.  The probes are
endless.  Script kiddies.  It's why we can't have nice things.

Examples of the types of probes.  None of these are a problem on my
system but malicious attackers probe to see if there are any
vulnerabilities.  These are probes.

    GET /data/admin/allowurl.txt
    GET /html/public/index.php
    GET /phpinfo.php
    GET /wp-config.php
    GET /wp-content/plugins/wp-file-manager/readme.txt
    GET /wp-login.php
    POST /wp-admins.php

The best thing you can do for these is to configure 'fail2ban' to
watch those log files and to drop the malicious probes at the firewall
level.  I think most of us would not do without fail2ban!

The HTTP 302 responses need more investigation before an analyses
could be made however.  Because for example they might be simply a
normal http redirection from http to https.  Or similar.  So they all
might turn out to be perfectly normal.  You would have to look and
see.

Bob
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales
Bob Proulx wrote:

> If the web server logs said it was 404 then that is an HTTP 404 return
> code and not an SMTP 404 return code since it is in the web server
> logs not the smtp server logs....

I know that.  (Sorry that I accidentally said they were SMTP response
codes, I do know better, they were HTTP response codes, mea culpa, it's
late at night and I'm dealing with pain from a kidney stone right now.)

Again, for clarity, I was asked by Wietse to examine both the web logs
and the Postfix logs on my server (both services are running on a single
box, I'm not using Docker), in order to find correlations.  I wasn't
able to find any correlations.  I did see various extraneous HTTP GET
requests in my web logs, but none of them related to my mail incidents.

I singled out the "GET /nette.micro" requests in particular because they
stood out as likely instances of attempted reverse tunneling attacks.
Also (something I didn't mention before), one of these corresponded in
time pretty closely to one of the fake messages I received -- though I
am inclined to dismiss this as mere coincidence, since the GET request
failed with a 404 HTTP return code.

The HTTP 302 responses to "GET /nette.micro" requests appear, as best I
can tell, to have all been simple redirections from HTTP to HTTPS.  The
corresponding HTTPS GET requests were all rejected with 404 codes.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Wietse Venema
In reply to this post by Rich Wales
Rich Wales:

>
> > I would suggest that you fix the exploited script. Look for time
> > stamps that appear in both web server logging and Postfix logging.
>
> Thanks, Wietse.  That would obviously be the best approach, if it
> worked, but I tried it and (so far at least) haven't been able to find
> any matching entries.
>
> I did find some generally suspicious things in my web server logs --
> including lots of clients looking for the following item:
>
>         /nette.micro?callback=shell_exec&cmd=ifconfig

Well here is, an idea:

    Extract time stamps for NON-ERROR web server responses, and
    correlate those time stamnps with activity in Postfix logs.

I prefer to spend my brain cycles on other things than to kook up
schemes that could perhaps slow down a hypothetical exploit.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales
On 2020-10-20 06:45, Wietse Venema wrote:

> Extract time stamps for NON-ERROR web server responses, and
> correlate those time stamnps with activity in Postfix logs.

Working on this now.  There are log entries for several GET requests
asking for nonsensical things like the following:

/index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP

/index.php?s=index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=phpinfo&vars[1][]=1

/index.php?m=admin&c=index&a=login&dosubmit=1

/?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php>

A couple of the above are near the dates/times when I was having the
e-mail problem.  But this could just as easily be a coincidence -- and
as far as I can tell, none of the above would accomplish anything -- the
supplied parameters are completely different from what the "index.php"
script in question is expecting.  Are these strange GET requests still
something which I should investigate further?

Some other observations (none apparently pointing to any problem):

My server runs a web site which sells a book on shoemaking which my
mother wrote long ago.  The site uses PHP, plus one JavaScript file.
There are, however, NO FORMS -- it's all done by clicking buttons, and
the financial transactions are handled by PayPal.  Lots and lots of GETs
in the log for this site, but no PUTs or POSTs, and the files themselves
are all read-only, so I can't really see how they could have been
exploited (though I'm open to enlightenment on this).  All of the above
weird GETs with random options tacked onto the URL were for this site.
And for what it may be worth, this site consists of raw PHP and JS which
I wrote from scratch, without using any frameworks or toolkits.

Lots of attempts to GET a script named "wp-login.php" in several
directories.  In fact, there are not (and never have been) ANY
"wp-login.php files on this server (not running WordPress).  Strangely,
though, many of the GETs return a 200 HTTP status code -- not something
I would expect when a requested file doesn't exist.  Were it not for the
200 HTTP status code, I would have just dismissed these as irrelevant.
In any case, none of these "wp-login.php" attempts correspond to the
dates when I was having the e-mail problem.

I had a couple of VERY old PHP scripts supporting "Project Honey Pot".
I've removed them, though, and will review my security before putting
them back (or, more properly, installing fresh scripts from the
project).  The logs showed about 20 accesses to my honeypot scripts, but
none around the dates of interest.

And I have still not seen any further instances of the hacker attack in
the last several days.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

ilyak
> /index.php?s=index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=phpinfo&vars[1][]=1
That is fine: networks are constantly scanned by bots. They are trying to hack any site using well-known vulnerabilities.

I have a lot of similar entries, although I do not have PHP on my site)

I have never been hacked, but if I were, here is what I would do:
* Reformat drive and install the latest stable version of your favorite OS. Be sure to upgrade it on the regular basis. Many OSes can do that using cron.
* Use the latest stable version of some mature framework and also update it. If you aren't using one, then make sure you understand how to write secure code and how to run it correctly
* Close all ports except http, https and ssh (which you should move away from 22 port because 22 port is also scanned by bots). Disable password authentication for ssh (use keys instead)
* Check your server from the remote one to be sure all other ports are closed.
* Configure lowatch (or something like that) to send your logs every day. Check logs carefully.






On Wed, Oct 21, 2020 at 2:03 AM Rich Wales <[hidden email]> wrote:
On 2020-10-20 06:45, Wietse Venema wrote:

> Extract time stamps for NON-ERROR web server responses, and
> correlate those time stamnps with activity in Postfix logs.

Working on this now.  There are log entries for several GET requests
asking for nonsensical things like the following:

/index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP

/index.php?s=index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=phpinfo&vars[1][]=1

/index.php?m=admin&c=index&a=login&dosubmit=1

/?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php>

A couple of the above are near the dates/times when I was having the
e-mail problem.  But this could just as easily be a coincidence -- and
as far as I can tell, none of the above would accomplish anything -- the
supplied parameters are completely different from what the "index.php"
script in question is expecting.  Are these strange GET requests still
something which I should investigate further?

Some other observations (none apparently pointing to any problem):

My server runs a web site which sells a book on shoemaking which my
mother wrote long ago.  The site uses PHP, plus one JavaScript file.
There are, however, NO FORMS -- it's all done by clicking buttons, and
the financial transactions are handled by PayPal.  Lots and lots of GETs
in the log for this site, but no PUTs or POSTs, and the files themselves
are all read-only, so I can't really see how they could have been
exploited (though I'm open to enlightenment on this).  All of the above
weird GETs with random options tacked onto the URL were for this site.
And for what it may be worth, this site consists of raw PHP and JS which
I wrote from scratch, without using any frameworks or toolkits.

Lots of attempts to GET a script named "wp-login.php" in several
directories.  In fact, there are not (and never have been) ANY
"wp-login.php files on this server (not running WordPress).  Strangely,
though, many of the GETs return a 200 HTTP status code -- not something
I would expect when a requested file doesn't exist.  Were it not for the
200 HTTP status code, I would have just dismissed these as irrelevant.
In any case, none of these "wp-login.php" attempts correspond to the
dates when I was having the e-mail problem.

I had a couple of VERY old PHP scripts supporting "Project Honey Pot".
I've removed them, though, and will review my security before putting
them back (or, more properly, installing fresh scripts from the
project).  The logs showed about 20 accesses to my honeypot scripts, but
none around the dates of interest.

And I have still not seen any further instances of the hacker attack in
the last several days.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Wietse Venema
In reply to this post by Rich Wales
Rich Wales:
> On 2020-10-20 06:45, Wietse Venema wrote:
>
> > Extract time stamps for NON-ERROR web server responses, and
> > correlate those time stamnps with activity in Postfix logs.
>
> Working on this now.  There are log entries for several GET requests
> asking for nonsensical things like the following:
>
> /index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP

Now we're getting somewhere :-)

According to a well-known search engine:

Query: HelloThinkPHP
Result:  ThinkPHP Remote Code Execution (RCE) bug

> /?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php>

Query: HelloThinkCMF
Result: WordPress exploit.

> /index.php?s=index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=phpinfo&vars[1][]=1

Another remote code execution exploit.

If any of those got a 200 HHTP response then you have been owned.

        Wietse

> A couple of the above are near the dates/times when I was having the
> e-mail problem.  But this could just as easily be a coincidence -- and
> as far as I can tell, none of the above would accomplish anything -- the
> supplied parameters are completely different from what the "index.php"
> script in question is expecting.  Are these strange GET requests still
> something which I should investigate further?
>
> Some other observations (none apparently pointing to any problem):
>
> My server runs a web site which sells a book on shoemaking which my
> mother wrote long ago.  The site uses PHP, plus one JavaScript file.
> There are, however, NO FORMS -- it's all done by clicking buttons, and
> the financial transactions are handled by PayPal.  Lots and lots of GETs
> in the log for this site, but no PUTs or POSTs, and the files themselves
> are all read-only, so I can't really see how they could have been
> exploited (though I'm open to enlightenment on this).  All of the above
> weird GETs with random options tacked onto the URL were for this site.
> And for what it may be worth, this site consists of raw PHP and JS which
> I wrote from scratch, without using any frameworks or toolkits.
>
> Lots of attempts to GET a script named "wp-login.php" in several
> directories.  In fact, there are not (and never have been) ANY
> "wp-login.php files on this server (not running WordPress).  Strangely,
> though, many of the GETs return a 200 HTTP status code -- not something
> I would expect when a requested file doesn't exist.  Were it not for the
> 200 HTTP status code, I would have just dismissed these as irrelevant.
> In any case, none of these "wp-login.php" attempts correspond to the
> dates when I was having the e-mail problem.
>
> I had a couple of VERY old PHP scripts supporting "Project Honey Pot".
> I've removed them, though, and will review my security before putting
> them back (or, more properly, installing fresh scripts from the
> project).  The logs showed about 20 accesses to my honeypot scripts, but
> none around the dates of interest.
>
> And I have still not seen any further instances of the hacker attack in
> the last several days.
>
> Rich Wales
> [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales
On 2020-10-21 06:17, Wietse Venema wrote:

> If any of those got a 200 HHTP response then you have been owned.

Acknowledged.  In this case, though, I honestly don't think so.  Let me
explain why.

My server IS NOT RUNNING THINKPHP OR WORDPRESS.  Never has.

The PHP site in question here was not built using WordPress, ThinkPHP,
or any other framework or toolkit.  And the /index.php file for the site
does not expect, and isn't written to process, command-line parameters
such as "s", "a", "content", "function", or "vars".

Apache would, to be sure, return a 200 HTTP status code for these
queries, but I think all that means is that the /index.php file was
found.  However, the PHP code in the /index.php file (plus the functions
in the other PHP files on the site invoked from /index.php) isn't
expecting any of the above command-line parameters and, as best I can
tell, should simply be ignoring them.

Unless there's something in raw PHP (not using any frameworks) that
recognizes and acts on these or other parameters outside the context of
whatever the raw PHP code in question is doing, nothing untoward should
happen, right?

Please feel free to try attacking the site in question for yourself,
adding any parameters you like to the URL, and let me know, and I'll go
check the site and confirm if I saw anything strange.

                https://www.marywalesloomis.com

The only command-line parameter which this web site is supposed to
recognize and do anything with is a "page=" parameter.  Everything else
on the command line / URL should be disregarded.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Demi M. Obenour
In reply to this post by ilyak
On 10/20/20 8:20 PM, IL Ka wrote:

>>
> /index.php?s=index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=phpinfo&vars[1][]=1
> That is fine: networks are constantly scanned by bots. They are trying to
> hack any site using well-known vulnerabilities.
>
> I have a lot of similar entries, although I do not have PHP on my site)
>
> I have never been hacked, but if I were, here is what I would do:
> * Reformat drive and install the latest stable version of your favorite OS.
> Be sure to upgrade it on the regular basis. Many OSes can do that using
> cron.
I agree, with the caveat that an attacker would need to have obtained
root access to implant a rootkit.  I consider the likelihood of this
high enough that wiping and reinstalling is justified.

> * Use the latest stable version of some mature framework and also update
> it. If you aren't using one, then make sure you understand how to write
> secure code and how to run it correctly
> * Close all ports except http, https and ssh (which you should move away
> from 22 port because 22 port is also scanned by bots). Disable password
> authentication for ssh (use keys instead)

If password and challenge-response authentication for SSH are
disabled, it isn't necessary to move SSH off of port 22.  SSH keys
are not vulnerable to brute-force attack, and last pre-authentication
vulnerability (other than denial of service) that I am aware of in
OpenSSH was in 2003.

Moving the SSH port can, however, reduce noise in your logs.
fail2ban and friends can help as well.

Sincerely,

Demi

OpenPGP_0xB288B55FFF9C22C1.asc (3K) Download Attachment
OpenPGP_signature (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Viktor Dukhovni
In reply to this post by Rich Wales
On Wed, Oct 21, 2020 at 08:50:54AM -0700, Rich Wales wrote:

> My server IS NOT RUNNING THINKPHP OR WORDPRESS.  Never has.

While the webserver is a good bet, before investing too much time
chasing ghosts, enumerate all the network listeners on your listem.
Something along the lines of (for Linux):

    # netstat -anp --inet --tcp | grep LISTEN
    # netstat -anp --inet6 --tcp | grep LISTEN

What LISTENERS do you see?  Then check your iptables for any unexpected
source IP mappings.

    # iptables -4 -t nat -n -L

These are hard to read, look carefully for anything that might NAT
remote IPs to 127.0.0.1.

But of course the web server is still a good bet.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Benny Pedersen-2
In reply to this post by Rich Wales
Rich Wales skrev den 2020-10-21 17:50:
> On 2020-10-21 06:17, Wietse Venema wrote:

> My server IS NOT RUNNING THINKPHP OR WORDPRESS.  Never has.

why do you SHOUT at Wietse ?

it does not matter if you have or not have, remote will try to detect if
you do or did, long time frame or not

try wget 127.0.0.1:25 and se what postfix responce to http remotes

to protect websites, i begin to think about make my own problem with
apache directoryindex .... websites does not need to expose
directoryindex files since http://example.org/?url still works same as
http://example.org/index.php?url

lets say the directoryindex subdir is pr dir and not used any other
places, then hackers have to be naviseels :=)
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales
On 2020-10-21 09:18, Benny Pedersen wrote:

> why do you SHOUT at Wietse ?

I was not shouting -- at least not intentionally.  I was being emphatic,
and perhaps a little frustrated at the suggestion that I might have been
hit by a WordPress exploit even after I had said earlier in this thread
that I'm not using WordPress.

I intended no offence to Wietse or anyone else, and I wish to apologize
to anyone I might have inadvertently offended.

> try wget 127.0.0.1:25 and se what postfix responce to http remotes

$ wget 127.0.0.1:25
--2020-10-21 10:20:27--  http://127.0.0.1:25/
Connecting to 127.0.0.1:25... connected.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
index.html: Permission denied

Cannot write to ‘index.html’ (Permission denied).
$

Not surprising that this command failed, I suppose, since Postfix isn't
an HTTP server.

I've also checked for illicit listeners -- though I will check again.
Since this box lives behind a physically separate firewall appliance
system, in addition to having its own host-based firewall rules in
place, a breakin -- though obviously not impossible -- is probably less
likely than if the server were connected directly to the Internet.

For the time being, I am going to continue to monitor the system in
question and see if any further instances of the "fake open relay spam
flood" problem occur.  No more incidents in the last several days (cross
fingers and touch wood).  The next time it happens (assuming it does), I
will take a much more careful look at what is going on, in hopes of
catching the offender in the act.  In order not to scatter any more spam
onto the Internet, I'll temporarily block inbound and outbound SMTP in
the firewall while I'm investigating the next incident (again, assuming
there is one) -- previously, I stopped Postfix, but this may have made
it harder for me to track down the issue in real time.

I would still like to figure out a way, btw, to catch locally generated
spam of this sort in Postfix.  I've already asked here about rejecting
HELO/EHLO when the client is localhost but the HELO/EHLO host is not
localhost -- I still think this would make sense, but I'm getting the
clear impression that it's just not there and just isn't going to get
added.  Or maybe I can reduce my use of permit_mynetworks in my
configuration -- I am currently invoking permit_mynetworks in my client,
HELO, sender, relay, and recipient smtpd restrictions, maybe this is
excessive.

I'll also check on other lists, do more extensive web searches, etc., to
see if anyone else out there has encountered this kind of attack.  As a
very last resort, I may consider wiping and rebuilding the system, but
I'm not willing to expend the time and energy to do that without first
having some reasonably specific evidence indicating exactly what has
happened.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Nick Tait
On 22/10/20 7:24 am, Rich Wales wrote:
I would still like to figure out a way, btw, to catch locally generated
spam of this sort in Postfix.  I've already asked here about rejecting
HELO/EHLO when the client is localhost but the HELO/EHLO host is not
localhost -- I still think this would make sense, but I'm getting the
clear impression that it's just not there and just isn't going to get
added.  Or maybe I can reduce my use of permit_mynetworks in my
configuration -- I am currently invoking permit_mynetworks in my client,
HELO, sender, relay, and recipient smtpd restrictions, maybe this is
excessive.

This sounds like the perfect candidate for a simple access policy. See: http://www.postfix.org/SMTPD_POLICY_README.html

In summary, you'd want to create a script in a language of your choice, which in the simplest case does this:

  1. Reads in lines until a blank line.
  2. Then sees if the lines that it read included the line "client_address=127.0.0.1".
  3. If it did, then it checks if it also received the line "helo_name=localhost".
  4. Then it outputs a result based on the results of steps #2 & #3:
    • If #2 matched and #3 matched, then it prints "dunno", followed by a blank line.
    • If #2 matched but #3 didn't, then it prints "reject You look like you're trying to get me to send spam", followed by a blank line.
    • If #2 didn't match, then it prints "dunno", followed by a blank line.

NB: The reason for using "dunno" (rather than "ok") is so that other following checks will still be performed.

Then:

  1. Configure a new service in master.cf to run your script (using spawn).
  2. Add "check_policy_service" into your smtpd_helo_restrictions option (before permit_mynetworks) to tell postfix to use the script.

Of course I've glossed over a lot of detail, and so I'd recommend reading the Postfix documentation and/or looking at some example policies (e.g. I use postfix-policyd-spf-python for SPF) to get your head around how policies work, before implementing what I suggested above.

Also remember "warn_if_reject" is your best friend when writing policy scripts! :-)

Nick.
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales
In reply to this post by Rich Wales
I've made one change to my configuration which may help handle the
locally generated spam problem, at least in the case of the "fake open
relay" mail.

I have removed permit_mynetworks from my smtpd_relay_restrictions.  (I
still have permit_mynetworks in the smtpd client, HELO, sender, and
recipient restrictions.)  In case this change might have broken
something (which it doesn't appear to have done), I also enabled
soft_bounce=yes.

Shortly thereafter, I found one (and, so far, only one) incident in my
log where an open relay message apparently originated from my server
itself.  It looks strange, though.  Check out the following log excerpt
and note particularly what happened with regard to the postscreen process:

Oct 21 20:22:33 memoryalpha postfix/postscreen[4751]: CONNECT from
[193.169.253.190]:63634 to [10.0.229.197]:25
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4752]: addr 193.169.253.190
listed by domain hostkarma.junkemailfilter.com as 127.0.1.1
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4752]: addr 193.169.253.190
listed by domain hostkarma.junkemailfilter.com as 127.0.0.2
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4758]: addr 193.169.253.190
listed by domain zen.spamhaus.org as 127.0.0.4
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4758]: addr 193.169.253.190
listed by domain zen.spamhaus.org as 127.0.0.3
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4752]: addr 193.169.253.190
listed by domain dnsbl.justspam.org as 127.0.0.2
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4757]: addr 193.169.253.190
listed by domain bl.spamcop.net as 127.0.0.2
Oct 21 20:22:33 memoryalpha postfix/postscreen[4751]: CONNECT from
[127.0.0.1]:40434 to [127.0.0.1]:25
Oct 21 20:22:33 memoryalpha postfix/postscreen[4751]: WHITELISTED
[127.0.0.1]:40434
Oct 21 20:22:33 memoryalpha postfix/smtpd[4764]: connect from
localhost[127.0.0.1]
Oct 21 20:22:33 memoryalpha postfix/dnsblog[4759]: addr 193.169.253.190
listed by domain score.senderscore.com as 127.0.4.0
Oct 21 20:22:34 memoryalpha postfix/dnsblog[4760]: addr 193.169.253.190
listed by domain truncate.gbudb.net as 127.0.0.2
Oct 21 20:22:34 memoryalpha postfix/smtpd[4764]: NOQUEUE: reject: RCPT
from localhost[127.0.0.1]: 454 4.7.1 <[hidden email]>: Relay access
denied; from=<[hidden email]> to=<[hidden email]> proto=ESMTP
helo=<WIN-NT9DHV1HPCJ>
Oct 21 20:22:34 memoryalpha postfix/smtpd[4764]: disconnect from
localhost[127.0.0.1] ehlo=1 mail=1 rcpt=0/1 rset=1 quit=1 commands=4/5
Oct 21 20:22:39 memoryalpha dovecot: imap-login: Aborted login (no auth
attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured,
session=<Tvvn+TmyNq9/AAAB>
Oct 21 20:22:39 memoryalpha postfix/postscreen[4751]: DNSBL rank 78 for
[193.169.253.190]:63634
Oct 21 20:22:40 memoryalpha postfix/postscreen[4751]: NOQUEUE: reject:
RCPT from [193.169.253.190]:63634: 450 4.7.1 Service unavailable; client
[193.169.253.190] blocked using zen.spamhaus.org;
from=<[hidden email]>, to=<[hidden email]>, proto=ESMTP,
helo=<WIN-NT9DHV1HPCJ>
Oct 21 20:22:40 memoryalpha postfix/postscreen[4751]: DISCONNECT
[193.169.253.190]:63634

The postscreen process (PID 4751) initially fielded a connection from
193.169.253.190 (port 63634) -- an IP address, btw and fwiw, which is
assigned to a hosting service in Estonia.  But before rejecting this
connection (because the IP address was blacklisted), another connection
sprang into life from 127.0.0.1 (port 40434).  Basically, it looks to me
as if the connection from 127.0.0.1 was somehow nested inside the
connection from 193.169.253.190.  This could just be a coincidence, but
the fact that all this activity happened within a single postscreen
process (PID 4751) confuses me -- can anyone explain this?  For what
it's worth, there is no other activity with PID 4751 anywhere else in
sight in my log.  Also, the sender and recipient e-mail addresses for
the 193.169.253.190 and 127.0.0.1 connections are the same -- another
seemingly very strong indication that they are somehow related, though
it's not clear to me how.

Correlating the above with other logs on my server, an inbound SMTP
connection from 193.169.253.190 on remote port 63634 was accepted and
logged by iptables.  No connections from 193.169.253.190 show up in my
server's Apache logs.

So, again, can anyone suggest an explanation for why a complete Postfix
connection from 127.0.0.1 is seemingly embedded inside a complete
Postfix connection from 193.169.253.190?

In case it matters, I'm running Postfix 3.3.0, installed as a package on
an Ubuntu 18.04.5 LTS system.  I'm not knowingly enabling XCLIENT in my
Postfix configuration for what this might be worth.

Thanks for any thoughts.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Viktor Dukhovni
On Thu, Oct 22, 2020 at 10:44:06PM -0700, Rich Wales wrote:

> I have removed permit_mynetworks from my smtpd_relay_restrictions.  (I
> still have permit_mynetworks in the smtpd client, HELO, sender, and
> recipient restrictions.)  In case this change might have broken
> something (which it doesn't appear to have done), I also enabled
> soft_bounce=yes.

This of course disables outbound submission on port 25, but that's
generally a good idea anyway.

> Shortly thereafter, I found one (and, so far, only one) incident in my
> log where an open relay message apparently originated from my server
> itself. 
>
> Oct 21 20:22:33 memoryalpha postfix/postscreen[4751]: CONNECT from
>   [193.169.253.190]:63634 to [10.0.229.197]:25
> [...]

The source IP is listed by multiple RBLs (perhaps a few too many
configured, but whatever...)

> Oct 21 20:22:33 memoryalpha postfix/postscreen[4751]: CONNECT from
> [127.0.0.1]:40434 to [127.0.0.1]:25
> Oct 21 20:22:33 memoryalpha postfix/postscreen[4751]: WHITELISTED
> [127.0.0.1]:40434
> Oct 21 20:22:33 memoryalpha postfix/smtpd[4764]: connect from
> localhost[127.0.0.1]

Fascinating, this would be a good time to post your master.cf settings,
i.e. verbatim, without refolding of lines output of:

    $ postconf -Mf

and also your main.cf settings, i.e. verbatim, without refolding of
lines output of:

    $ posconf -nf

> Oct 21 20:22:34 memoryalpha postfix/smtpd[4764]: NOQUEUE: reject: RCPT
> from localhost[127.0.0.1]: 454 4.7.1 <[hidden email]>: Relay access
> denied; from=<[hidden email]> to=<[hidden email]> proto=ESMTP
> helo=<WIN-NT9DHV1HPCJ>

Yes, the HELO name, sender and recipient above match the corresponding
information in the postscreen reject message below.

> Oct 21 20:22:39 memoryalpha dovecot: imap-login: Aborted login (no auth
> attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured,
> session=<Tvvn+TmyNq9/AAAB>

But note also "dovecot" showing up here.  It looks like the spammer is
using some feature of Dovecot to originate a connection to the Postfix
SMTP server.

> Oct 21 20:22:39 memoryalpha postfix/postscreen[4751]: DNSBL rank 78 for
> [193.169.253.190]:63634

Impressive DNSBL score!

> Oct 21 20:22:40 memoryalpha postfix/postscreen[4751]: NOQUEUE: reject:
> RCPT from [193.169.253.190]:63634: 450 4.7.1 Service unavailable; client
> [193.169.253.190] blocked using zen.spamhaus.org;
> from=<[hidden email]>, to=<[hidden email]>, proto=ESMTP,
> helo=<WIN-NT9DHV1HPCJ>

Note the matching "helo".  The two connections sure look related.

> The postscreen process (PID 4751) initially fielded a connection from
> 193.169.253.190 (port 63634) -- an IP address, btw and fwiw, which is
> assigned to a hosting service in Estonia.  But before rejecting this
> connection (because the IP address was blacklisted), another connection
> sprang into life from 127.0.0.1 (port 40434).

Perhaps via some component of Dovecot?  Is your Dovecot up to date and
patched?

> Basically, it looks to me
> as if the connection from 127.0.0.1 was somehow nested inside the
> connection from 193.169.253.190.

No, not nested inside, just concurrent.

> This could just be a coincidence, but
> the fact that all this activity happened within a single postscreen
> process (PID 4751) confuses me -- can anyone explain this?

There's only ever one postscreen process, it handles multiple concurrent
connections, that is its purpose in life.  See POSTSCREEN_README.

> Correlating the above with other logs on my server, an inbound SMTP
> connection from 193.169.253.190 on remote port 63634 was accepted and
> logged by iptables.  No connections from 193.169.253.190 show up in my
> server's Apache logs.

It looks more like connection proxying then a web server exploit.

> So, again, can anyone suggest an explanation for why a complete Postfix
> connection from 127.0.0.1 is seemingly embedded inside a complete
> Postfix connection from 193.169.253.190?

Your Postfix and Dovecot configurations would be useful at this point.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Viktor Dukhovni
On Fri, Oct 23, 2020 at 02:14:58AM -0400, Viktor Dukhovni wrote:
> On Thu, Oct 22, 2020 at 10:44:06PM -0700, Rich Wales wrote:

> > Oct 21 20:22:39 memoryalpha dovecot: imap-login: Aborted login (no auth
> > attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured,
> > session=<Tvvn+TmyNq9/AAAB>
>
> But note also "dovecot" showing up here.  It looks like the spammer is
> using some feature of Dovecot to originate a connection to the Postfix
> SMTP server.

Of course that could just be smtpd enabling Dovecot auth, though you
should probably not enable SASL on port 25.  In which case "Dovecot"
may be a distraction.

>
> > Oct 21 20:22:40 memoryalpha postfix/postscreen[4751]: NOQUEUE: reject:
> > RCPT from [193.169.253.190]:63634: 450 4.7.1 Service unavailable; client
> > [193.169.253.190] blocked using zen.spamhaus.org;
> > from=<[hidden email]>, to=<[hidden email]>, proto=ESMTP,
> > helo=<WIN-NT9DHV1HPCJ>
>
> Note the matching "helo".  The two connections sure look related.

One interesting question is whether you somehow have TCP fast open
enabled by default.  If the purpose of the first connection is to obtain
a TFO cookie, and somehow the second connection is able to use this
cookie (weak validation?) to send an entire SMTP transaction along with
the SYN packet, but this time with 127.0.0.1 as the source address, then
the use of two connections for the attack starts to make sense.

Of course TFO is supposed to provide some cryptographic protection
against spoofing the source address.  So something odd is happening
if that's the actual story.

Do you have any non-default network kernel parameter settings that
might be related to TCP fastopen?

Perhaps a previously implanted backdoor makes this form of attack
possible.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Rich Wales
In reply to this post by Viktor Dukhovni

>> Oct 21 20:22:39 memoryalpha dovecot: imap-login: Aborted login (no auth
>> attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured,
>> session=<Tvvn+TmyNq9/AAAB>
>
> But note also "dovecot" showing up here.  It looks like the spammer is
> using some feature of Dovecot to originate a connection to the Postfix
> SMTP server.

Actually, it's most likely a log entry from a periodic probe of my
server's functions (including its IMAP service) by Nagios.

I captured another incident last night, and there was no mention of
Dovecot in the log.  I'm not attaching the log for this second incident
right now (I had "smtpd -v" in effect and the log data is really long),
but I can share it if people really want to see it.

I've made another change to my Postfix configuration -- I changed the
value of postscreen_blacklist_action to "drop".  I'm still waiting to
see if I have any more instances of open relay attempts from localhost
after having made this change.  If the earlier open relay attempts are
in fact somehow (still unsure how?) being generated as a consequence of
the blacklisted connection, then maybe having postscreen drop right away
will nip the open relay attempts in the bud.

Rich Wales
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Limiting HELO spoofing in Postfix?

Viktor Dukhovni
On Fri, Oct 23, 2020 at 12:57:06PM -0700, Rich Wales wrote:

> > But note also "dovecot" showing up here.  It looks like the spammer is
> > using some feature of Dovecot to originate a connection to the Postfix
> > SMTP server.
>
> Actually, it's most likely a log entry from a periodic probe of my
> server's functions (including its IMAP service) by Nagios.

Yes, I noted that Dovecot could have been a red-herring.

> I captured another incident last night, and there was no mention of
> Dovecot in the log.  I'm not attaching the log for this second incident
> right now (I had "smtpd -v" in effect and the log data is really long),
> but I can share it if people really want to see it.

I don't recall whether you have as yet posted the requested (sans any
reformatting of line breaks) outputs of:

    $ postconf -Mf
    $ postconf -nf

More Postfix log data is not going to help.  Somehow the attacker is
managing to create an apparent connection from 127.0.0.1.  This is
either a bad rule in your iptables, some interesting abuse of your
TCP stack, or an as yet unreported proxy process on your system.

> I've made another change to my Postfix configuration -- I changed the
> value of postscreen_blacklist_action to "drop".

That won't change anything.  What would be helpful is a full packet
capture, which may mean recording everything coming in/out of your
machine for some hours, if you have the space to store full network
recordings.

> I'm still waiting to
> see if I have any more instances of open relay attempts from localhost
> after having made this change.  If the earlier open relay attempts are
> in fact somehow (still unsure how?) being generated as a consequence of
> the blacklisted connection, then maybe having postscreen drop right away
> will nip the open relay attempts in the bud.

The logic of this escapes me.  Dropping the Postscreen connection
slightly more quickly is unlikely to change anything.  Also instead
of rejecting email from 127.0.0.1, it would be more useful to accept
and quarantine it (put it on HOLD).

   smtpd_client_restrictions =
    check_client_access inline:{ { 127.0.0.1 = HOLD } }

    smtpd_relay_restrictions =
       permit_mynetworks
       reject_unauth_destination

--
    Viktor.
12