smtpd_upstream_proxy_protocol + smtpd_tls_wrappermode

[Lukas Erlacher]

Hello,

I am trying to put haproxy in front of postfix and utilise the proxy protocol to get accurate client IPs.

This works fine for all unencrypted / starttls based listeners, but not for tls-wrapped listeners using smtpd_tls_wrappermode.

This is the haproxy configuration:

frontend ft_smtps
         bind 0.0.0.0
         timeout client 1m
         log global
         option tcplog
         default_backend bk_postfix_smtps

backend bk_postfix_smtps
         option smtpchk HELO localhost
         log global
         option tcplog
         timeout server 1m
         timeout connect 5s
         server mailbackend mail:10464 send-proxy

And this is the postfix master.cf configuration:

10464     inet  n       -       -       -       -       smtpd
   -o smtpd_tls_wrappermode=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_upstream_proxy_protocol=haproxy

I am testing this using openssl s_client -connect localhost:465 and expect to get a 220 message from the postfix, but the connection just hangs until I close it.

Something goes wrong with establishing the SSL session:

Aug 31 09:52:47 mail postfix-from-user/smtpd[2416]: connect from a-mua.informatik.tu-muenchen.de[xxx.xxx.42.153]
Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: SSL_accept error from mailclient[xxx.xxx.42.153]: lost connection
Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: lost connection after CONNECT from mailclient[xxx.xxx.42.153]
Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: disconnect from mailclient[xxx.xxx.42.153]

Is this implemented in postfix? If it is, what is the right configuration to make it work?

[Wietse Venema]

Lukas Erlacher:
> Something goes wrong with establishing the SSL session:
>
> Aug 31 09:52:47 mail postfix-from-user/smtpd[2416]: connect from a-mua.informatik.tu-muenchen.de[xxx.xxx.42.153]

The HaProxy hand-over succeeds: Postfix gets a client name and address.

> Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: SSL_accept error from mailclient[xxx.xxx.42.153]: lost connection
> Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: lost connection after CONNECT from mailclient[xxx.xxx.42.153]
> Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: disconnect from mailclient[xxx.xxx.42.153]

I suspect that when the TLS hello becomes appended to the HaProxy server data,
the Postfix HaProxy client reads part of the TLS hello.
Fixing that would require setting the input read buffer size to 1:

in the HaProxy client before reading input:

    vstream_control(state->client,
            VSTREAM_CTL_BUFSIZE, 1,
            VSTREAM_CTL_END);

And before returning:

    vstream_control(state->client,
            VSTREAM_CTL_BUFSIZE, VSTREAM_BUFSIZE,
            VSTREAM_CTL_END);

        Wietse

[Wietse Venema]

Wietse Venema:

> Lukas Erlacher:
> > Something goes wrong with establishing the SSL session:
> >
> > Aug 31 09:52:47 mail postfix-from-user/smtpd[2416]: connect from a-mua.informatik.tu-muenchen.de[xxx.xxx.42.153]
>
> The HaProxy hand-over succeeds: Postfix gets a client name and address.
>
> > Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: SSL_accept error from mailclient[xxx.xxx.42.153]: lost connection
> > Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: lost connection after CONNECT from mailclient[xxx.xxx.42.153]
> > Aug 31 09:52:49 mail postfix-from-user/smtpd[2416]: disconnect from mailclient[xxx.xxx.42.153]
>
> I suspect that when the TLS hello becomes appended to the HaProxy server data,
> the Postfix HaProxy client reads part of the TLS hello.
> Fixing that would require setting the input read buffer size to 1:

Please try this.

        Wietse

*** ./src/smtpd/smtpd_haproxy.c- 2012-06-30 17:12:00.000000000 -0400
--- ./src/smtpd/smtpd_haproxy.c 2015-09-23 16:57:02.000000000 -0400
***************
*** 103,108 ****
--- 103,116 ----
       */
  #define ENABLE_DEADLINE 1
 
+     /*
+      * Don't buffer beyond the end-of-line. Setting a small buffer is
+      * meaningful only before the first I/O operation happens.
+      */
+     vstream_control(state->client,
+    VSTREAM_CTL_BUFSIZE, 1,
+    VSTREAM_CTL_END);
+
      smtp_stream_setup(state->client, var_smtpd_uproxy_tmout, ENABLE_DEADLINE);
      switch (io_err = vstream_setjmp(state->client)) {
      default:
***************
*** 139,144 ****
--- 147,159 ----
  state->port = mystrdup(smtp_client_port.buf);
 
  /*
+ * Enable normal buffering.
+ */
+ vstream_control(state->client,
+ VSTREAM_CTL_BUFSIZE, VSTREAM_BUFSIZE,
+ VSTREAM_CTL_END);
+
+ /*
  * Avoid surprises in the Dovecot authentication server.
  */
  state->dest_addr = mystrdup(smtp_server_addr.buf);

[Lukas Erlacher]

Thanks, I will try that!

Best,
Luke

[Lukas Erlacher]

Hi,

> Please try this.
>
> Wietse
>
> [patch]

Works like a charm! I couldn't just patch our live server of course but I grabbed the ubuntu 14.04 postfix 2.11.0 source package on a VM, and the haproxy1.5 from trusty-backports and it works.

Thanks for the prompt support! Will you be merging this?

Best,
Luke

[Wietse Venema]

Lukas Erlacher:

> Hi,
>
> > Please try this.
> >
> > Wietse
> >
> > [patch]
>
> Works like a charm! I couldn't just patch our live server of course
> but I grabbed the ubuntu 14.04 postfix 2.11.0 source package on a
> VM, and the haproxy1.5 from trusty-backports and it works.
>
> Thanks for the prompt support! Will you be merging this?

In the next 3.1 development release, and in a month or so, in the next
stable releases (2.9 .. 3.0).

        Wietse

[Lukas Erlacher]

>>
>> Thanks for the prompt support! Will you be merging this?
>
> In the next 3.1 development release, and in a month or so, in the next
> stable releases (2.9 .. 3.0).
>
> Wietse
>

That's great to hear!

Best,
Luke