Emails from localhost

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Emails from localhost

Proxy
Hello,

I'm seeing lot of emails coming from local IP address trying to send
message to non existing accounts. Sending accounts are valid and even
authenticated. They all try to send messages to domain matching the
sending one. For example:

[hidden email] -> [hidden email]
[hidden email] -> [hidden email]

and so on. support@* is valid, user@* is not. In logs they are coming
from inet_interfaces address set in main.cf. This is the handshake part:

 Out: 220 mail.example.com ESMTP Postfix
 In:  EHLO localhost.localdomain
 Out: 250-mail.example.com
 Out: 250-PIPELINING
 Out: 250-SIZE 24800000
 Out: 250-ETRN
 Out: 250-STARTTLS
 Out: 250-ENHANCEDSTATUSCODES
 Out: 250-8BITMIME
 Out: 250 DSN
 In:  STARTTLS
 Out: 220 2.0.0 Ready to start TLS
 In:  EHLO localhost.localdomain
 Out: 250-mail.example.com
 Out: 250-PIPELINING
 Out: 250-SIZE 24800000
 Out: 250-ETRN
 Out: 250-AUTH PLAIN LOGIN
 Out: 250-AUTH=PLAIN LOGIN
 Out: 250-ENHANCEDSTATUSCODES
 Out: 250-8BITMIME
 Out: 250 DSN
 In:  AUTH LOGIN
 Out: 334 fjUIrwvlXCkR
 In:  t3VncG6ydiBwpGZ2v3ducmRjb476ZXJ0ZXIub3Jn
 Out: 334 dfjklaeuYFGL
 In:  dEgzfjklsaliQwMxl
 Out: 235 2.7.0 Authentication successful
 In:  MAIL FROM:<[hidden email]>
 Out: 250 2.1.0 Ok
 In:  RCPT TO:<[hidden email]>
 Out: 451 4.3.0 <[hidden email]>: Temporary lookup failure

Session aborted, reason: lost connection

Jun  3 06:12:04 mail postfix/smtpd[26186]: connect from mail.example.com[DD.DDD.DD.DDD]
Jun  3 06:12:04 mail postfix/smtpd[26186]: setting up TLS connection from mail.example.com[DD.DDD.DD.DDD]
Jun  3 06:12:04 mail postfix/smtpd[26186]: Anonymous TLS connection established from mail.example.com[DD.DDD.DD.DDD]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)
Jun  3 06:12:04 mail postfix/smtpd[26186]: NOQUEUE: reject: RCPT from mail.example.com[DD.DDD.DD.DDD]: 550 5.1.1 <[hidden email]>: Recipient address rejected: User unknown in virtual mailbox table; from=<[hidden email]> to=<[hidden email]> proto=ESMTP helo=<localhost.localdomain>
Jun  3 06:12:04 mail postfix/smtpd[26186]: lost connection after RCPT from mail.example.com[DD.DDD.DD.DDD]
Jun  3 06:12:04 mail postfix/smtpd[26186]: disconnect from mail.example.com[DD.DDD.DD.DDD]


My postconf -n (Postfix 2.6.6) is in the attachment.

How can I find out from where these emails are coming? If they are really from
localhost, what program/script? If from outside how to prevent IP spoofing?
Seing that it tries several passwords and succeed make me worried even more.



postconf-n.log (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Wietse Venema
Proxy:

> Hello,
>
> I'm seeing lot of emails coming from local IP address trying to send
> message to non existing accounts. Sending accounts are valid and even
> authenticated. They all try to send messages to domain matching the
> sending one. For example:
>
> [hidden email] -> [hidden email]
> [hidden email] -> [hidden email]
>
> and so on. support@* is valid, user@* is not. In logs they are coming
> from inet_interfaces address set in main.cf. This is the handshake part:
>
>  Out: 220 mail.example.com ESMTP Postfix
>  In:  EHLO localhost.localdomain
>  Out: 250-mail.example.com
>  Out: 250-PIPELINING
>  Out: 250-SIZE 24800000

To find out which processes have a connection to or from port 25,

    # lsof -Pi | grep :25 (must run as root to see all processes)

On Linux,

    # netstat -np | grep :25 (must run as root to see all processes)

is an alternative if your system does not have lsof.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Benny Pedersen-2
In reply to this post by Proxy
Proxy skrev den 2018-06-03 13:25:

....

to get more help post postconf -n

and relevandt logs that shows the problem
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Proxy
In reply to this post by Wietse Venema
On 2018-Jun-03 11:43, Wietse Venema wrote:
> To find out which processes have a connection to or from port 25,
>
>     # lsof -Pi | grep :25 (must run as root to see all processes)
 
Thanks Wietse, actualy I needed to grep :587 as this is mail sent after
authentication and I got pid that I searched with ps command and got to
see full command including script responsible for mail sending.
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Bill Cole-3
In reply to this post by Proxy
On 3 Jun 2018, at 7:25 (-0400), Proxy wrote:

> Hello,
>
> I'm seeing lot of emails coming from local IP address trying to send
> message to non existing accounts. Sending accounts are valid and even
> authenticated. They all try to send messages to domain matching the
> sending one. For example:
>
> [hidden email] -> [hidden email]
> [hidden email] -> [hidden email]
>
> and so on. support@* is valid, user@* is not. In logs they are coming
> from inet_interfaces address set in main.cf.

Your system has been compromised. The most common vectors are vulnerable
web applications (e.g. carelessly-written PHP or CGI scripts) but there
are many other possible modes of attack.

> This is the handshake part:
>
>  Out: 220 mail.example.com ESMTP Postfix
>  In:  EHLO localhost.localdomain
>  Out: 250-mail.example.com
>  Out: 250-PIPELINING
>  Out: 250-SIZE 24800000
>  Out: 250-ETRN
>  Out: 250-STARTTLS
>  Out: 250-ENHANCEDSTATUSCODES
>  Out: 250-8BITMIME
>  Out: 250 DSN
>  In:  STARTTLS
>  Out: 220 2.0.0 Ready to start TLS
>  In:  EHLO localhost.localdomain
>  Out: 250-mail.example.com
>  Out: 250-PIPELINING
>  Out: 250-SIZE 24800000
>  Out: 250-ETRN
>  Out: 250-AUTH PLAIN LOGIN
>  Out: 250-AUTH=PLAIN LOGIN
>  Out: 250-ENHANCEDSTATUSCODES
>  Out: 250-8BITMIME
>  Out: 250 DSN
>  In:  AUTH LOGIN
>  Out: 334 fjUIrwvlXCkR
>  In:  t3VncG6ydiBwpGZ2v3ducmRjb476ZXJ0ZXIub3Jn
>  Out: 334 dfjklaeuYFGL
>  In:  dEgzfjklsaliQwMxl
>  Out: 235 2.7.0 Authentication successful
>  In:  MAIL FROM:<[hidden email]>
>  Out: 250 2.1.0 Ok
>  In:  RCPT TO:<[hidden email]>
>  Out: 451 4.3.0 <[hidden email]>: Temporary lookup failure
>
> Session aborted, reason: lost connection
>
> Jun  3 06:12:04 mail postfix/smtpd[26186]: connect from
> mail.example.com[DD.DDD.DD.DDD]
> Jun  3 06:12:04 mail postfix/smtpd[26186]: setting up TLS connection
> from mail.example.com[DD.DDD.DD.DDD]
> Jun  3 06:12:04 mail postfix/smtpd[26186]: Anonymous TLS connection
> established from mail.example.com[DD.DDD.DD.DDD]: TLSv1 with cipher
> DHE-RSA-AES256-SHA (256/256 bits)
> Jun  3 06:12:04 mail postfix/smtpd[26186]: NOQUEUE: reject: RCPT from
> mail.example.com[DD.DDD.DD.DDD]: 550 5.1.1 <[hidden email]>:
> Recipient address rejected: User unknown in virtual mailbox table;
> from=<[hidden email]> to=<[hidden email]> proto=ESMTP
> helo=<localhost.localdomain>
> Jun  3 06:12:04 mail postfix/smtpd[26186]: lost connection after RCPT
> from mail.example.com[DD.DDD.DD.DDD]
> Jun  3 06:12:04 mail postfix/smtpd[26186]: disconnect from
> mail.example.com[DD.DDD.DD.DDD]
>

Obfuscating IP addresses and hostnames in email log snippets here is
pointless and removes any value these log lines might have had in
analyzing your problem.

Also, those log lines are NOT from a session similar to the purported
SMTP chat you included above. If Postfix logs that it sent a 550 5.1.1  
reply, it did NOT send a 451 4.3.0 reply as in the SMTP chat.

> My postconf -n (Postfix 2.6.6) is in the attachment.

Why are you using obsolete software? 2.6.6 was released over 8 years
ago. The last 2.6.x support release was 2.6.19, over 5 years ago.

If this is generally how software on your system is maintained, it is
unsurprising that it has been taken over by a spammer.

> How can I find out from where these emails are coming?

You've already said it: they are coming from your own broken system.

> If they are really from
> localhost, what program/script?

That's not a question that can be answered from the outside.

> If from outside how to prevent IP spoofing?

Even a system so old that it is running Postfix 2.6.6 is extremely
unlikely to be vulnerable to an external attacker spoofing a local IP
address for a TCP-based protocol like SMTP. Functional IP spoofing
generally is a UDP or ICMP trick, not TCP (at least not in THIS
millennieum...)

> Seing that it tries several passwords and succeed make me worried even
> more.

That's common for spammers.

Again: A spammer has taken over your server, either in a limited way
through a vulnerable web inteerface of some sort or possibly in an
unlimited way, restricted only by the risk of discovery.

Unmunged log entries and output from postconf -n and postconf -M might
help us to help you make this easier to analyze but there is a very
strong possibility that the first step towards an actaul fix is to wipe
the system clean and reinstall everything from the ground up (hopefuly
in non-obsolete versions.)

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steadier Work: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Wietse Venema
Bill Cole:
> > My postconf -n (Postfix 2.6.6) is in the attachment.
>
> Why are you using obsolete software? 2.6.6 was released over 8 years
> ago. The last 2.6.x support release was 2.6.19, over 5 years ago.
>
> If this is generally how software on your system is maintained, it is
> unsurprising that it has been taken over by a spammer.

Hey, calm down, Bill.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Proxy
In reply to this post by Bill Cole-3
On 2018-Jun-03 17:06, Bill Cole wrote:
> Your system has been compromised. The most common vectors are vulnerable web
> applications (e.g. carelessly-written PHP or CGI scripts) but there are many
> other possible modes of attack.

It's most likely our own script, the one that have these credentials in
it. How it is exploited I still don't know, but not likely that it's
possible to send after I have disabled this account.

> Obfuscating IP addresses and hostnames in email log snippets here is
> pointless and removes any value these log lines might have had in analyzing
> your problem.

Not sure why. Care to elaborate? According to
http://www.postfix.org/DEBUG_README.html#mail it's the way to go. You
just need to use same substitutions. So DD.DDD.DD.DDD is always the one
and same real world IP. Different IP would get different value.

> Also, those log lines are NOT from a session similar to the purported SMTP
> chat you included above. If Postfix logs that it sent a 550 5.1.1  reply, it
> did NOT send a 451 4.3.0 reply as in the SMTP chat.

Probably not. Handshake is sent to postmaster@me from MAILER-DAEMON.
That's how I noticed the problem. I'm just not sure what is the value
for spammer in sending to the user@mydomain. It would make more sense to
send to some outside victims.

> > My postconf -n (Postfix 2.6.6) is in the attachment.
>
> Why are you using obsolete software? 2.6.6 was released over 8 years ago.
> The last 2.6.x support release was 2.6.19, over 5 years ago.
>
> If this is generally how software on your system is maintained, it is
> unsurprising that it has been taken over by a spammer.

I assume you are using some bleeding edge distro, but for us using
production servers, it makes more sense to use long supported distro
with regular security patches. Like CentOS 6. I'm confident that CentOS
security team does a good job providing latest security patches RedHat
releases including those related to Postfix.

> > How can I find out from where these emails are coming?
>
> You've already said it: they are coming from your own broken system.
>
> > If they are really from
> > localhost, what program/script?
>
> That's not a question that can be answered from the outside.
>
> > If from outside how to prevent IP spoofing?
>
> Even a system so old that it is running Postfix 2.6.6 is extremely unlikely
> to be vulnerable to an external attacker spoofing a local IP address for a
> TCP-based protocol like SMTP. Functional IP spoofing generally is a UDP or
> ICMP trick, not TCP (at least not in THIS millennieum...)
>
> > Seing that it tries several passwords and succeed make me worried even
> > more.
>
> That's common for spammers.

Hm, now that I took a closer look, it actually don't try several
passwords. It's just base64 for username, password, actual username and
actual password.

> Again: A spammer has taken over your server, either in a limited way through
> a vulnerable web inteerface of some sort or possibly in an unlimited way,
> restricted only by the risk of discovery.
>
> Unmunged log entries and output from postconf -n and postconf -M might help
> us to help you make this easier to analyze but there is a very strong
> possibility that the first step towards an actaul fix is to wipe the system
> clean and reinstall everything from the ground up (hopefuly in non-obsolete
> versions.)

You probably know that there is no postconf -M on my ancient Postfix, so
you're just pulling my leg, right?


Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Bill Cole-3
On 3 Jun 2018, at 18:08 (-0400), Proxy wrote:

> On 2018-Jun-03 17:06, Bill Cole wrote:
>> Your system has been compromised. The most common vectors are
>> vulnerable web
>> applications (e.g. carelessly-written PHP or CGI scripts) but there
>> are many
>> other possible modes of attack.
>
> It's most likely our own script, the one that have these credentials
> in
> it. How it is exploited I still don't know, but not likely that it's
> possible to send after I have disabled this account.
>
>> Obfuscating IP addresses and hostnames in email log snippets here is
>> pointless and removes any value these log lines might have had in
>> analyzing
>> your problem.
>
> Not sure why. Care to elaborate?

You were inquiring about traffic being really from a local IP address or
not. If you wipe out all distinctions between different IP addresses and
hostnames, possible issues relevant to the problem become invisible.

Also: there's almost never any marginal risk in "exposing" an IP address
of a mail server. It might be prudent to obfuscate an IP address of a
SMTP client in some circumstances, but not in this case.

> According to
> http://www.postfix.org/DEBUG_README.html#mail it's the way to go.

That does not recommend obfuscating IP addresses.

It speaks of anonymizing email addresses. IP addresses are NOT email
addresses.

> You
> just need to use same substitutions. So DD.DDD.DD.DDD is always the
> one
> and same real world IP. Different IP would get different value.

That is not obvious from the obfuscated log & config.

Note that we have seen problems here entirely due to transposing or
omitting characters in an IP address or hostname.

>
>> Also, those log lines are NOT from a session similar to the purported
>> SMTP
>> chat you included above. If Postfix logs that it sent a 550 5.1.1  
>> reply, it
>> did NOT send a 451 4.3.0 reply as in the SMTP chat.
>
> Probably not. Handshake is sent to postmaster@me from MAILER-DAEMON.
> That's how I noticed the problem. I'm just not sure what is the value
> for spammer in sending to the user@mydomain. It would make more sense
> to
> send to some outside victims.
>
>>> My postconf -n (Postfix 2.6.6) is in the attachment.
>>
>> Why are you using obsolete software? 2.6.6 was released over 8 years
>> ago.
>> The last 2.6.x support release was 2.6.19, over 5 years ago.
>>
>> If this is generally how software on your system is maintained, it is
>> unsurprising that it has been taken over by a spammer.
>
> I assume you are using some bleeding edge distro,

I don't use any system vendor's version of any open source MTA on
production mail servers that are exposed to the Internet.

However, as Wietse noted, I wrote that a bit too hastily and harshly,
and I apologize for the tone and implication. I realize that this may
not even be a "production mail server" in the sense of being primarily
tasked with handling mail to and/or from the world at large, so it may
be entirely reasonable to stick with the base MTA.

> but for us using
> production servers, it makes more sense to use long supported distro
> with regular security patches. Like CentOS 6. I'm confident that
> CentOS
> security team does a good job providing latest security patches RedHat
> releases including those related to Postfix.

That would be an issue to debate in some other venue.
For here, it is generally a good idea to note if you're using a variant
vendor-custom version of Postfix. Debian, Apple, and RedHat have all
distributed customized versions.

[...]

>> Unmunged log entries and output from postconf -n and postconf -M
>> might help
>> us to help you make this easier to analyze but there is a very strong
>> possibility that the first step towards an actaul fix is to wipe the
>> system
>> clean and reinstall everything from the ground up (hopefuly in
>> non-obsolete
>> versions.)
>
> You probably know that there is no postconf -M on my ancient Postfix,
> so
> you're just pulling my leg, right?

No, just not editing myself well.

The alternative to 'postconf -M' would be 'grep -v '^#'
/etc/postfix/master.cf' and this MIGHT have been useful to discover if
you had something like amavisd configured, which could make it look like
mail is originating locally when it is not.

I'm glad that you've identified the specific problem script. I hope you
figure out its breakage well enough not actually need to do a full wipe
and reload.

--
Bill Cole
[hidden email] or [hidden email]
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steadier Work: https://linkedin.com/in/billcole
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

@lbutlr
In reply to this post by Proxy
On 03 Jun 2018, at 16:08, Proxy <[hidden email]> wrote:
> I'm confident that CentOS security team does a good job providing latest security patches RedHat releases including those related to Postfix.

Are you under the impression that CentOS is writing security patches for obsolete and unsupported versions of Postfix?

That is not the case.

There is a big difference between bleeding edge and obsolete, and you are firmly in the obsolete (as in not support, not patched, not secure) camp. The last update to 2.6 was over 5 years ago (Feb 2014) and that is a significantly newer version that you are running (Mar 2010).

--
'They were myths and they were real,' he said loudly. 'Both a wave and a
particle.' --Guards! Guards!

Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost [OT]

Mike Guelfi
Upstream RHEL, and therefore CentOS, don't update version numbers when
they roll security patches.

Latest release though:
2016-10-31 - Jaroslav Škarvada <[hidden email]> - 2:2.6.6-8
- Backported support for TLS 1.1, TLS 1.2

Not insanely old...

Quoting "@lbutlr" <[hidden email]>:

> On 03 Jun 2018, at 16:08, Proxy <[hidden email]> wrote:
>> I'm confident that CentOS security team does a good job providing
>> latest security patches RedHat releases including those related to
>> Postfix.
>
> Are you under the impression that CentOS is writing security patches
> for obsolete and unsupported versions of Postfix?
>
> That is not the case.
>
> There is a big difference between bleeding edge and obsolete, and
> you are firmly in the obsolete (as in not support, not patched, not
> secure) camp. The last update to 2.6 was over 5 years ago (Feb 2014)
> and that is a significantly newer version that you are running (Mar
> 2010).
>
> --
> 'They were myths and they were real,' he said loudly. 'Both a wave and a
> particle.' --Guards! Guards!


Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost [OT]

Stephen Satchell
On 06/03/2018 11:13 PM, Mike Guelfi wrote:
> Upstream RHEL, and therefore CentOS, don't update version numbers when
> they roll security patches.
>
> Latest release though:
> 2016-10-31 - Jaroslav Škarvada <[hidden email]> - 2:2.6.6-8
> - Backported support for TLS 1.1, TLS 1.2

From Centos 7.5:
[satch@c7-i5 ~]$ postconf -d | grep mail_version
mail_version = 2.10.1

By the way, Red Hat offers this file on its systems:
/usr/share/doc/postfix-2.10.1/README-Postfix-SASL-RedHat.txt
Reply | Threaded
Open this post in threaded view
|

Re: Emails from localhost

Matus UHLAR - fantomas
In reply to this post by Proxy
On 03.06.18 13:25, Proxy wrote:
>Jun  3 06:12:04 mail postfix/smtpd[26186]: connect from mail.example.com[DD.DDD.DD.DDD]
>Jun  3 06:12:04 mail postfix/smtpd[26186]: setting up TLS connection from mail.example.com[DD.DDD.DD.DDD]
>Jun  3 06:12:04 mail postfix/smtpd[26186]: Anonymous TLS connection established from mail.example.com[DD.DDD.DD.DDD]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)
>Jun  3 06:12:04 mail postfix/smtpd[26186]: NOQUEUE: reject: RCPT from mail.example.com[DD.DDD.DD.DDD]: 550 5.1.1 <[hidden email]>: Recipient address rejected: User unknown in virtual mailbox table; from=<[hidden email]> to=<[hidden email]> proto=ESMTP helo=<localhost.localdomain>
>Jun  3 06:12:04 mail postfix/smtpd[26186]: lost connection after RCPT from mail.example.com[DD.DDD.DD.DDD]
>Jun  3 06:12:04 mail postfix/smtpd[26186]: disconnect from mail.example.com[DD.DDD.DD.DDD]

>On 2018-Jun-03 11:43, Wietse Venema wrote:
>> To find out which processes have a connection to or from port 25,
>>
>>     # lsof -Pi | grep :25 (must run as root to see all processes)

On 03.06.18 21:23, Proxy wrote:
>Thanks Wietse, actualy I needed to grep :587 as this is mail sent after
>authentication and I got pid that I searched with ps command and got to
>see full command including script responsible for mail sending.

now I wonder, 587 should generate logs like "postfix/submission/smtpd".
Did you modify master.cf or just omitted
   -o syslog_name=postfix/submission\
there?

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
The only substitute for good manners is fast reflexes.