syn flood generated by a postfix transaction

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

syn flood generated by a postfix transaction

Gabriele Bulfon
Hello,
 
we recently had some situation of full bandwidth usage through our firewall, and by investigation we discovered that this was caused by a specific mail to a specific destination sent via Postifx 3.1.6.
We found an ACK SYN DUP flood during the problem, many many packets sent, and this on the postfix log regarding that transaction:
 
enabling PIX workarounds: disable_esmtp delay_dotcrlf for...
conversation with ... timed out while sending message body
 
We had to block that destination on the firewall and remove the message on the queue, or it would restart the problem when retried by the queue.
 
The same mail sent via Postfix 2.7.1 on a different data center with the same firewall was sent without problems and without the postfix logs above.
 
I can reproduce the problem by sending again the same email.
Other different smaller mails to the same desintation are sent correctly.
 
What may be happening?

Gabriele

Reply | Threaded
Open this post in threaded view
|

Re: syn flood generated by a postfix transaction

Wietse Venema
Gabriele Bulfon:
> Hello,
> we recently had some situation of full bandwidth usage through our
> firewall, and by investigation we discovered that this was caused
> by a specific mail to a specific destination sent via Postifx 3.1.6.
> We found an ACK SYN DUP flood during the problem, many many packets
> sent, and this on the postfix log regarding that transaction:

FYI, Postfix does not send SYN packets. SYN packets are sent by the
network stack in your kernel when a program opens a TCP connection.
The remote server is then expected to respond with a SYN|ACK packet.

client   SYN ->   server
client <- SYN|ACK server
client   ACK ->   server

A properly working network stack will send a SYN packet for each
new outbound TCP connection, and will retransmit the SYN packet
after several seconds until it gives up. A repeated SYN packet will
result in a SYN|ACK response (with perhaps a different server initial
sequence number).

With delays of seconds it is hard to achieve a SYN flood.

Now, it is possible that you have configured Postfix with huge
limits on process counts and destination concurrency. In that case
Postfix will attempt to make a huge number of connections to the
same site, which effectively results in large numbers of SYN and
SYN|ACK packets.

That would be a problem of your own making.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: syn flood generated by a postfix transaction

Gabriele Bulfon
Thanks so much for the deep explanation :)
 
Kernel is illumos, I exclude it can be a bug in the kernel stack.
I will check Postfix config, but I don't think there is any huge limit as you suggested.
 
What about the "PIX workaround"? Can it be something causing this? It did not happen on a previous version on another site with same firewall, same message.
Or, we also find answers from the destination host saying "write error" on their side: can it be an unhandled situation somewhere?
Also keep in mind it happens only with this message (or maybe some others I don't know) any time I retry a send, not with any other simple message I try.
 
Gabriele




----------------------------------------------------------------------------------

Da: Wietse Venema <[hidden email]>
A: Postfix users <[hidden email]>
Cc: [hidden email] [hidden email]
Data: 10 settembre 2020 15.52.39 CEST
Oggetto: Re: syn flood generated by a postfix transaction

Gabriele Bulfon:
> Hello,
> we recently had some situation of full bandwidth usage through our
> firewall, and by investigation we discovered that this was caused
> by a specific mail to a specific destination sent via Postifx 3.1.6.
> We found an ACK SYN DUP flood during the problem, many many packets
> sent, and this on the postfix log regarding that transaction:

FYI, Postfix does not send SYN packets. SYN packets are sent by the
network stack in your kernel when a program opens a TCP connection.
The remote server is then expected to respond with a SYN|ACK packet.

client SYN -> server
client client ACK -> server

A properly working network stack will send a SYN packet for each
new outbound TCP connection, and will retransmit the SYN packet
after several seconds until it gives up. A repeated SYN packet will
result in a SYN|ACK response (with perhaps a different server initial
sequence number).

With delays of seconds it is hard to achieve a SYN flood.

Now, it is possible that you have configured Postfix with huge
limits on process counts and destination concurrency. In that case
Postfix will attempt to make a huge number of connections to the
same site, which effectively results in large numbers of SYN and
SYN|ACK packets.

That would be a problem of your own making.

Wietse


Reply | Threaded
Open this post in threaded view
|

Re: syn flood generated by a postfix transaction

Wietse Venema
Gabriele Bulfon:
> Thanks so much for the deep explanation :)

Considering the questions asked, I must keep the conversation at a
basic level.

> Kernel is illumos, I exclude it can be a bug in the kernel stack.
> I will check Postfix config, but I don't think there is any huge
> limit as you suggested.

Don't overlook the firewall (yours or theirs). or other boxes
in the middle that may manipulate TCP packets.

> What about the "PIX workaround"?

That causes Postfix to speak SMTP slower. If the data sent over an
established TCP connection can trigger a TCP SYN flood, then SOMETHING
AT THE NETWORK LEVEL is massively screwed up.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: syn flood generated by a postfix transaction

Gabriele Bulfon
Hi, just for the curious, we could solve it by using a rule on the bsd firewall lan to pass TCP using "synproxy".
The mail  transaction passes and goes on quickly without generating the syn/ack flood.
 
Hope this is enough for any situation...
 
Gabriele





----------------------------------------------------------------------------------

Da: Wietse Venema <[hidden email]>
A: Postfix users <[hidden email]>
Data: 11 settembre 2020 2.08.01 CEST
Oggetto: Re: syn flood generated by a postfix transaction

Gabriele Bulfon:
> Thanks so much for the deep explanation :)

Considering the questions asked, I must keep the conversation at a
basic level.

> Kernel is illumos, I exclude it can be a bug in the kernel stack.
> I will check Postfix config, but I don't think there is any huge
> limit as you suggested.

Don't overlook the firewall (yours or theirs). or other boxes
in the middle that may manipulate TCP packets.

> What about the "PIX workaround"?

That causes Postfix to speak SMTP slower. If the data sent over an
established TCP connection can trigger a TCP SYN flood, then SOMETHING
AT THE NETWORK LEVEL is massively screwed up.

Wietse