TLS library problems

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

TLS library problems

shdjsahwkjq ehwq kehwkq h
Hello, I am seeing a lot of these in my syslog logs.  I am not sure  
what they mean, google did not yield a lot other than people calling  
out an incorrctly named cert/key

system.log:Jul 10 00:07:57 trex postfix/smtpd[45598]: warning: TLS  
library problem: 45598:error:140760FC:SSL  
routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:571:

The reason I am looking, is I am having trouble with a proxy that will  
sit in front of postfix.  The basic workflow for the one domain I am  
testing the proxy in:
Internet -> Proxy -> Postfix

Proxy is geographically separate from postfix, not same subnet.  I  
would like MTA to MTA crypto.  Sometimes it works, and mail is  
delivered, and other times it is not.

An email will hit the proxy on port 25, which will talk to postfix  
also on port 25.  STARTTLS is issued.  At that point, the proxy will  
either make the crypto connection, and deliver the mail off to  
postfix, or, it will drop the connection.
--
Scott * If you contact me off list replace talklists@ with scott@ *

Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

Sahil Tandon
On Fri, 10 Jul 2009, Scott Haneda wrote:

> system.log:Jul 10 00:07:57 trex postfix/smtpd[45598]: warning: TLS  
> library problem: 45598:error:140760FC:SSL  
> routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:571:

Debug the proxy.  What is it?  Not Postfix, I'd guess.

--
Sahil Tandon <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

Wietse Venema
In reply to this post by shdjsahwkjq ehwq kehwkq h
Scott Haneda:
> Hello, I am seeing a lot of these in my syslog logs.  I am not sure  
> what they mean, google did not yield a lot other than people calling  
> out an incorrctly named cert/key
>
> system.log:Jul 10 00:07:57 trex postfix/smtpd[45598]: warning: TLS  
> library problem: 45598:error:140760FC:SSL  
> routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:571:

This is openssl's way of saying that the client sent garbage.

        Wietse
Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

Barney Desmond
2009/7/11 Wietse Venema <[hidden email]>:
>> system.log:Jul 10 00:07:57 trex postfix/smtpd[45598]: warning: TLS
>> library problem: 45598:error:140760FC:SSL
>> routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:571:
>
> This is openssl's way of saying that the client sent garbage.

To expand on that, I imagine it means the client tried to talk
plaintext when Postfix was expecting crypto.

Can you clarify exactly how this is meant to work? You said you want
MTA-to-MTA crypto, I assume in this particular case you mean
Proxy->Postfix crypto. Depending on how much control you have over the
configuration, you could use a "dumb" method like an stunnel pipe, or
something smarter like STARTTLS in-band.

It sounds like you're trying to do the latter, but you say "STARTTLS
is issued.  At that point, the proxy will either make the crypto
connection, and deliver the mail off to postfix, or, it will drop the
connection.". Why should the proxy drop the connection? In any case, I
think the proxy needs debugging. You might also try adding the proxy
as a verbose peer in Postfix, it might make the client's mistakes
quickly evident.
Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

shdjsahwkjq ehwq kehwkq h
In reply to this post by Sahil Tandon
On Jul 10, 2009, at 4:42 PM, Sahil Tandon wrote:

> On Fri, 10 Jul 2009, Scott Haneda wrote:
>
>> system.log:Jul 10 00:07:57 trex postfix/smtpd[45598]: warning: TLS
>> library problem: 45598:error:140760FC:SSL
>> routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:571:
>
> Debug the proxy.  What is it?  Not Postfix, I'd guess.


The proxy is ASSP.  Not many people are doing TLS with this, I suspect  
it will be a challenge for me to pin down, since I know very little  
about crypto stuff with regard to TLS.
--
Scott * If you contact me off list replace talklists@ with scott@ *

Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

shdjsahwkjq ehwq kehwkq h
In reply to this post by Barney Desmond
On Jul 11, 2009, at 6:40 PM, Barney Desmond wrote:

> 2009/7/11 Wietse Venema <[hidden email]>:
>>> system.log:Jul 10 00:07:57 trex postfix/smtpd[45598]: warning: TLS
>>> library problem: 45598:error:140760FC:SSL
>>> routines:SSL23_GET_CLIENT_HELLO:unknown protocol:s23_srvr.c:571:
>>
>> This is openssl's way of saying that the client sent garbage.
>
> To expand on that, I imagine it means the client tried to talk
> plaintext when Postfix was expecting crypto.

Thanks for the estimation.  Comparing a working transaction with one  
that does not work, shows no difference.  The one part I need even  
more debug log data, only states "start tls" and then "failure".  I  
somehow need to get to the data that happens between those two log  
lines.

It is good to finally know this is more than likely the proxy though.

> Can you clarify exactly how this is meant to work? You said you want
> MTA-to-MTA crypto, I assume in this particular case you mean
> Proxy->Postfix crypto. Depending on how much control you have over the
> configuration, you could use a "dumb" method like an stunnel pipe, or
> something smarter like STARTTLS in-band.

I am trying to avoid stunnel, because this is supposed to be built  
into the proxy, and I have invested a lot of time into a package for  
the proxy.  I have invested about as much time into testing and trying  
to debug this issue.

My basic setup is Internet -> proxy - postfix
Where postfix is a working MTA that has worked for months on end as a  
rock solid MTA.

The basics are, an email comes in on port 25, from anywhere, it could  
be the local machine or inbound from any host.  Connect to port 25 on  
the proxy, which is then connected up to the remote postfix machine.

STARTTLS is issued, and a secured connection from the proxy to postfix  
is made.  The majority of the time, emails do make it, and are  
secured.  Some times they do not.  I have found some hosts that simply  
never make it, others that will make it in many hours time.

I have found in 99% of the cases, a machine on the local subnet to the  
proxy, will fail, but can eventually deliver a few hours later.  They  
just sit in that local machines postfix queue and are tried later.  
This is a convenient way for me to test.

For what it is worth, turning off STARTTLS on port 25 in postfix, and  
I am back to 100% reliability.

> It sounds like you're trying to do the latter,

Correct.

> but you say "STARTTLS
> is issued.  At that point, the proxy will either make the crypto
> connection, and deliver the mail off to postfix, or, it will drop the
> connection.".

Dropped connection.  What is more odd, is telnet prxoy.example.com 25  
then the ehlo, mail from, rcpt to, data dance works.  Where it fails,  
is when I use `mail [hidden email]` on the command line.

openssl client to the remote postfix, and the proxy, connect up fine  
as well. But maybe I just am not testing it enough it hit a failure.

> Why should the proxy drop the connection? In any case, I
> think the proxy needs debugging.

I agree.

> You might also try adding the proxy
> as a verbose peer in Postfix, it might make the client's mistakes
> quickly evident.

Doing a search on that turns up this very thread :)
Can you point me to docs on verbose peer, as well as an other  
suggestion you may have now that you know a little more.

If there is a kind soul out there that knows this stuff well, and  
could ever allow me to point an MX at them, and add an account, so I  
could point the proxy to them, allowing a little help with debugging  
this, I would be most appreciative.

Thank Barney for the suggestions.
--
Scott * If you contact me off list replace talklists@ with scott@ *

Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

Wietse Venema
Scott Haneda:
> Thanks for the estimation.  Comparing a working transaction with one  
> that does not work, shows no difference.  The one part I need even  
> more debug log data, only states "start tls" and then "failure".  I  
> somehow need to get to the data that happens between those two log  
> lines.

OpenSSL does not like what the proxy sends. To find out where the
proxy errs, you will need to go beyond logfiles, and look at the
data that is actually sent over the wire.

As Tsutomu once said, tcpdump is your friend (*).

For example one mistake is to send STARTTLS in a network packet
that also contains the first portion of the TLS handshake. The
proxy should send STARTTLS, wait for a positives server reply, and
then it should send the TLS handshake.

If you can't figure out what OpenSSL does not like about what the
proxy sends, then you will have to find someone to do it for you.
I won't.

        Wietse

(*) And as Dan Farmer replied, "No: tcpdump is YOUR friend", meaning
    that not everyone has the stomach to dissect packet dumps.
Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

shdjsahwkjq ehwq kehwkq h
On Jul 12, 2009, at 1:07 PM, Wietse Venema wrote:

> Scott Haneda:
>> Thanks for the estimation.  Comparing a working transaction with one
>> that does not work, shows no difference.  The one part I need even
>> more debug log data, only states "start tls" and then "failure".  I
>> somehow need to get to the data that happens between those two log
>> lines.
>
> OpenSSL does not like what the proxy sends. To find out where the
> proxy errs, you will need to go beyond logfiles, and look at the
> data that is actually sent over the wire.
>
> As Tsutomu once said, tcpdump is your friend (*).

Where is the best place to run tcpdump from, the proxy machine, or the  
postfix machine?  Could you suggest a tcpdump command that would help  
me with this?  I imagine, as long as tcpdump is instructed to send out  
something that is human readable, I can compare a packet dump of a  
working case, and a failing case, and look for the differences.

> For example one mistake is to send STARTTLS in a network packet
> that also contains the first portion of the TLS handshake. The
> proxy should send STARTTLS, wait for a positives server reply, and
> then it should send the TLS handshake.

Thanks.  Can you make any estimations as to why some sending servers  
have no issue, and others fail?

> If you can't figure out what OpenSSL does not like about what the
> proxy sends, then you will have to find someone to do it for you.
> I won't.

Thanks for your help, I will not continue this thread since I now know  
that it has nothing to do with postfix.  I will look to debug the proxy.

--
Scott * If you contact me off list replace talklists@ with scott@ *

Reply | Threaded
Open this post in threaded view
|

Re: TLS library problems

Wietse Venema
Scott Haneda:

> On Jul 12, 2009, at 1:07 PM, Wietse Venema wrote:
>
> > Scott Haneda:
> >> Thanks for the estimation.  Comparing a working transaction with one
> >> that does not work, shows no difference.  The one part I need even
> >> more debug log data, only states "start tls" and then "failure".  I
> >> somehow need to get to the data that happens between those two log
> >> lines.
> >
> > OpenSSL does not like what the proxy sends. To find out where the
> > proxy errs, you will need to go beyond logfiles, and look at the
> > data that is actually sent over the wire.
> >
> > As Tsutomu once said, tcpdump is your friend (*).
>
> Where is the best place to run tcpdump from, the proxy machine, or the  
> postfix machine?

OpenSSL on YOUR machine complains, so you need to find out what
OpenSSL on YOUR machine receives.

> Could you suggest a tcpdump command that would help  
> me with this?  I imagine, as long as tcpdump is instructed to send out  
> something that is human readable, I can compare a packet dump of a  
> working case, and a failing case, and look for the differences.

http://www.postfix.org/DEBUG_README.html has suggestions. And no,
the output is not human-readable, as tcpdump has limited understanding,
if any, of SMTP and TLS.

> > For example one mistake is to send STARTTLS in a network packet
> > that also contains the first portion of the TLS handshake. The
> > proxy should send STARTTLS, wait for a positives server reply, and
> > then it should send the TLS handshake.
>
> Thanks.  Can you make any estimations as to why some sending servers  
> have no issue, and others fail?

That is exactly how buggy systems work. There is a region called
"normal" where things appear to work error-free, and then there is
a much larger region called "not normal" where things break in
unexpected ways.

        Wietse