Gateway to Internal Server random "Connection refused"

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Gateway to Internal Server random "Connection refused"

Nikolaos Milas
Hello,

We are building a new mail gateway server (on CentOS 8), which is
running postfix (v3.5.8) from the GhettoForge Repo.

I noticed the following problem: Sometimes the server seems to not be
able to connect to our main mail server (Postfix v3.2.5) to verify
recipients; for example:

Jan 12 15:54:24 mailgw1 postfix/postscreen[2535]: CONNECT from
[2.228.155.xxx]:64067 to [83.212.5.xxx]:25
Jan 12 15:54:24 mailgw1 postfix/postscreen[2535]: PASS OLD
[2.228.155.xxx]:64067
Jan 12 15:54:24 mailgw1 postfix/smtpd[2540]: connect from
mailserver.comune.avellino.xxx[2.228.155.xxx]
Jan 12 15:54:24 mailgw1 postfix/smtpd[2540]: Anonymous TLS connection
established from mailserver.comune.avellino.xxx[2.228.155.xxx]: TLSv1
with cipher ECDHE-RSA-AES256-SHA (256/256 bits)
Jan 12 15:54:24 mailgw1 postfix/smtpd[2540]: NOQUEUE: reject: RCPT from
mailserver.comune.avellino.xxx[2.228.155.xxx]: 450 4.1.1
<[hidden email]>: Recipient address rejected: unverified address:
connect to vmail2.noa.xxx[2001:648:2011:15::xxx]:25: Connection refused;
from=<[hidden email]> to=<[hidden email]>
proto=ESMTP helo=<mail.comune.avellino.xxx>
Jan 12 15:54:24 mailgw1 postfix/smtpd[2540]: disconnect from
mailserver.comune.avellino.xxx[2.228.155.xxx] ehlo=2 starttls=1 mail=1
rcpt=0/1 quit=1 commands=5/6

or:

Jan 12 17:01:30 mailgw1 postfix/postscreen[3644]: CONNECT from
[2620:109:c006:104::170]:37882 to [2001:648:2ffc:1115::xxx]:25
Jan 12 17:01:31 mailgw1 postfix/postscreen[3644]: PASS OLD
[2620:109:c006:104::170]:37882
Jan 12 17:01:31 mailgw1 postfix/smtpd[3650]: connect from
mailc-bb.linkedin.com[2620:109:c006:104::170]
Jan 12 17:01:31 mailgw1 postfix/smtpd[3650]: Anonymous TLS connection
established from mailc-bb.linkedin.com[2620:109:c006:104::170]: TLSv1.2
with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Jan 12 17:01:32 mailgw1 postfix/cleanup[3655]: 4DFYgc0b5BzLm7T:
message-id=<[hidden email]>
Jan 12 17:01:32 mailgw1 postfix/qmgr[1263]: 4DFYgc0b5BzLm7T:
from=<[hidden email]>, size=229, nrcpt=1 (queue active)
Jan 12 17:01:32 mailgw1 postfix/smtp[3656]: 4DFYgc0b5BzLm7T:
to=<[hidden email]>, relay=vmail2.noa.xxx[194.177.195.xxx]:25,
delay=0.08, delays=0/0.02/0.05/0.01, dsn=2.1.5, status=deliverable (250
2.1.5 Ok)
Jan 12 17:01:32 mailgw1 postfix/qmgr[1263]: 4DFYgc0b5BzLm7T: removed
Jan 12 17:01:32 mailgw1 postfix/smtpd[3650]: NOQUEUE: reject: RCPT from
mailc-bb.linkedin.com[2620:109:c006:104::170]: 450 4.1.1
<[hidden email]>: Recipient address rejected: unverified address:
connect to vmail2.noa.xxx[2001:648:2011:15::xxx]:25: Connection refused;
from=<[hidden email]>
to=<[hidden email]> proto=ESMTP helo=<mailc-bb.linkedin.com>
Jan 12 17:01:37 mailgw1 postfix/smtpd[3650]: disconnect from
mailc-bb.linkedin.com[2620:109:c006:104::170] ehlo=2 starttls=1 mail=1
rcpt=0/1 quit=1 commands=5/6

You can see that: Recipient address rejected: unverified address:
connect to vmail2.noa.xxx[2001:648:2011:15::xxx]:25: Connection refused

This happens in many cases, for various recipients.

This does not happen to our other mail gateway server (MX),
mailgw3.noa.xxx, which has in essence the same setup (but older software
running on CentOS 6).

I cannot see anything related logged in the mail log of the main server
(vmail2.noa.xxx) which is responsible for user verification.

Can you please guess / trace / give me a hint about the possible cause
of the problem?

I would appreciate your help!

Follows postconf -n output for your reference:

==================================================================================================================

# postconf -n
allowed_list1 = reject
allowed_list2 = reject
command_directory = /usr/sbin
compatibility_level = 2
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
xxgdb $daemon_directory/$process_name $process_id & sleep 5
default_process_limit = 20
disable_vrfy_command = yes
enable_long_queue_ids = yes
header_checks = pcre:/etc/postfix/blacklisted_maillists
html_directory = no
inet_interfaces = all
inet_protocols = ipv4, ipv6
local_recipient_maps =
local_transport = error:local mail delivery is disabled
mail_name = NOA MAIL ICXC-NIKA
mail_owner = postfix
mailq_path = /usr/bin/mailq.postfix
manpage_directory = /usr/share/man
message_size_limit = 15728640
meta_directory = /etc/postfix
mydestination =
mynetworks = 127.0.0.1/32 [::1]/128
myorigin = $mydomain
newaliases_path = /usr/bin/newaliases.postfix
postscreen_access_list = permit_mynetworks,
cidr:/etc/postfix/postscreen_exceptions.cidr
postscreen_blacklist_action = enforce
postscreen_dnsbl_action = enforce
postscreen_dnsbl_reply_map =
texthash:/etc/postfix/postscreen_dnsbl_reply_map
postscreen_dnsbl_sites =
c4279dedc01b71e07b5b763485249bcc.combined.mail.abusix.zone,
b.barracudacentral.org, zen.spamhaus.org*2, psbl.surriel.com
postscreen_dnsbl_threshold = 2
postscreen_greet_action = enforce
queue_directory = /var/spool/postfix
rbl_reply_maps = texthash:/etc/postfix/rbl_reply_map
readme_directory = /usr/share/doc/postfix3-3.5.8/README_FILES
relay_domains = $transport_maps
relay_recipient_maps =
sample_directory = /usr/share/doc/postfix3-3.5.8/samples
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = postdrop
shlib_directory = /usr/lib/postfix
smtp_tls_security_level = may
smtpd_helo_required = yes
smtpd_milters = unix:/run/amavisd/amavisd-milter.sock
smtpd_recipient_restrictions = check_sender_access
hash:/etc/postfix/blacklisted_senders check_sender_access
pcre:/etc/postfix/blacklisted_maillists reject_unverified_recipient
reject_unauth_destination check_recipient_access
hash:/etc/postfix/protected_destinations permit_mynetworks
reject_invalid_hostname reject_unauth_pipelining reject_non_fqdn_sender
reject_unknown_sender_domain reject_non_fqdn_recipient
reject_unknown_recipient_domain permit_dnswl_client
c4279dedc01b71e07b5b763485249bcc.white.mail.abusix.zone
reject_rbl_client b.barracudacentral.org reject_rbl_client
zen.spamhaus.org reject_rbl_client psbl.surriel.com reject_rbl_client
bl.spamcop.net reject_rhsbl_client
c4279dedc01b71e07b5b763485249bcc.dblack.mail.abusix.zone
reject_rhsbl_client dbl.spamhaus.org reject_rhsbl_sender
c4279dedc01b71e07b5b763485249bcc.dblack.mail.abusix.zone
reject_rhsbl_sender dbl.spamhaus.org reject_rhsbl_helo
c4279dedc01b71e07b5b763485249bcc.dblack.mail.abusix.zone
reject_rhsbl_helo dbl.spamhaus.org permit
smtpd_restriction_classes = allowed_list1,allowed_list2
smtpd_tls_CAfile = /etc/pki/tls/certs/DigiCertCA.crt
smtpd_tls_cert_file = /etc/pki/tls/certs/star_noa_xxx-15410534.crt
smtpd_tls_exclude_ciphers = DES,3DES,MD5,aNULL,AES128,CAMELLIA128
smtpd_tls_key_file = /etc/pki/tls/private/star_noa_xxx-1243437.key
smtpd_tls_loglevel = 1
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1
smtpd_tls_security_level = may
smtpd_tls_session_cache_timeout = 3600s
transport_maps = hash:/etc/postfix/transportmap
unknown_local_recipient_reject_code = 550
unverified_recipient_reject_code = 550
unverified_sender_reject_code = 550
virtual_alias_maps = hash:/etc/postfix/virtualmap

==================================================================================================================

Thanks in advance,
Nick


Reply | Threaded
Open this post in threaded view
|

Re: Gateway to Internal Server random "Connection refused"

Wietse Venema
Nikolaos Milas:
> connect to vmail2.noa.xxx[2001:648:2011:15::xxx]:25: Connection refused;

This is a TCP-level problem.  The remote server's TCP/IP stack, or
some middle box, is refusing connections at the TCP level.

All Postfix can do is make fewer connections.

To reduce connection concurrency:

- a reduced SMTP client process limit in master.cf,
  (http://www.postfix.org/master.5.html)

- a reduced transport_destination_concurrency_limit in main.cf,
  (http://www.postfix.org/postconf.5.html#transport_destination_concurrency_limit)

To reduce the connection rate, if things get really desperate:

- a non-zero transport_destination_rate_delay in main.cf.
  (http://www.postfix.org/postconf.5.html#transport_destination_rate_delay)

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: Gateway to Internal Server random "Connection refused"

Nikolaos Milas
On 13/1/2021 4:41 μ.μ., Wietse Venema wrote:

> This is a TCP-level problem.  The remote server's TCP/IP stack, or
> some middle box, is refusing connections at the TCP level.

Thanks Wietse,

We'll be checking this out. Although I have done some connectivity
testing with repeated netcat (nc) polling to port 25 over IPv6 between
the two servers, I have not been able to find any issues, at least not yet.

Continuous ping6 shows stable connection at around 7ms; for example:

    rtt min/avg/max/mdev = 6.887/7.152/7.432/0.130 ms

In the last 12 hours I have noticed only one occurrence of the problem,
after I whitelisted all our mail servers (2 mail gateway servers and the
main one with the mailboxes) on our Cisco ASA firewall (which is the
main suspect I can think of for disrupting the communication). This only
occurrence did NOT happen at a time with high network load.

It seems to me difficult to troubleshoot since it is a so infrequent
event. The problem is how I can isolate and examine closely the TCP/IP
state at the moment when the problem occurs. Should I tcpdump IPv6 smtp
communication between two mail servers over many hours?

In any case, our mail traffic is low, so I think that reducing postfix
connection limit might not be needed.

If you or anyone else can provide suggestions/hints/thoughts for
troubleshooting at the TCP/IP layer, esp. IPv6, I would appreciate that.

A next move could be closer examination of dropped packets by our
firewall. I found this article for a start:

https://networklessons.com/cisco/asa-firewall/cisco-asa-packet-drop-troubleshooting

Thanks again,
Nick

Reply | Threaded
Open this post in threaded view
|

Re: Gateway to Internal Server random "Connection refused"

Wietse Venema
Nikolaos Milas:
> A next move could be closer examination of dropped packets by our
> firewall. I found this article for a start:
>
> https://networklessons.com/cisco/asa-firewall/cisco-asa-packet-drop-troubleshooting
>

I mentioned this, emphasis added:

> This is a TCP-level problem.  The remote server's TCP/IP stack, OR
> SOME MIDDLE BOX, is refusing connections at the TCP level.

Look no further. CISCO PIX or ASA has a long history of screwing
up connections with protocol inspection or other features.

        Wietse