Javamail connection

[James B. Byrne]

On Mon, December 21, 2020 20:07, Viktor Dukhovni wrote:

> It clearly supports STARTTLS, since it is actually performing the TLS
> handshake, and abandons it after receiving certificates it is not happy
> with.
>

Which confuses me, because I can use java to successfully negotiate a
certificate exchange with Postfix using the same keystore that the application
is using.  I cannot see Postfix sending a different server certificate on port
465 from that it presents on ports 25 or 465. And if the certificate on 465
passes the keystore validation on the client then what would prevent it from
passing on post 25?

I know that the service is different on both ports, but the certificate
acceptability should be the same.  And it is the client that is causing the
problem.

--
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
   Unencrypted messages have no legal claim to privacy
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: [phone]
Hamilton, Ontario             fax: [phone]
Canada  L8E 3C3

[James B. Byrne]

I have been looking at the Postfix logs and wonder if this is significant:

Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS read client
hello
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write
server hello
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write
change cipher spec
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 write
encrypted extensions
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write
certificate
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 write server
certificate verify
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write finished
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 early data
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL3 alert
read:fatal:certificate unknown
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:error in error
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept error from
accounting-2.internal.harte-lyne.ca[192.168.216.88]: -1
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: warning: TLS library problem:
error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate
unknown:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number
46:


It appears to me that the client is insisting on SSLv3 but that Postfix is
looking for or replying with TLSv1.3.  Would that cause a problem with the
certificate being recognised by the client?

--
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
   Unencrypted messages have no legal claim to privacy
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: [phone]
Hamilton, Ontario             fax: [phone]
Canada  L8E 3C3

[Viktor Dukhovni]

On Tue, Dec 22, 2020 at 09:24:27AM -0500, James B. Byrne wrote:

> > It clearly supports STARTTLS, since it is actually performing the TLS
> > handshake, and abandons it after receiving certificates it is not happy
> > with.
>
> Which confuses me, because I can use java to successfully negotiate a
> certificate exchange with Postfix using the same keystore that the application
> is using.

Perhaps you're confusing the Java keystore with the Java trust store,
they are separate entities.  Java is a programming language, not a
single application.  Each Java program gets to do its own thing.
Yes, they often share some common libraries and file formats, ...

> I cannot see Postfix sending a different server certificate on port
> 465 from that it presents on ports 25 or 465. And if the certificate
> on 465 passes the keystore validation on the client then what would
> prevent it from passing on post 25?

Well, you're running tests with completely different Java client
applications and likely in fact different configuration settings.  All
that's in common is that both are Java, which is something, but not
much...

Note that certificates don't pass "keystore" validation, they pass
"trust store" validation.  Java keystores are for your own private
key and cert (akin to PKCS12 containers, or a PEM with key + cert
chain), while trust stores are akin to a cert bundle with lots of
CA certs.

> I know that the service is different on both ports, but the
> certificate acceptability should be the same.  And it is the client
> that is causing the problem.

You could hypothetically have different certificate settings for
the different ports in master.cf, but if you don't then indeed
the server side TLS behaviour is likely the same across the board.

--
    Viktor.

[Viktor Dukhovni]

On Tue, Dec 22, 2020 at 10:57:48AM -0500, James B. Byrne wrote:

> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS read client hello
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write server hello
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write change cipher spec
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 write encrypted extensions
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write certificate
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 write server certificate verify
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write finished
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 early data
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL3 alert read:fatal:certificate unknown
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:error in error
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept error from accounting-2.internal.harte-lyne.ca[192.168.216.88]: -1
> Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: warning: TLS library problem: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate
> unknown:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46:
>
> It appears to me that the client is insisting on SSLv3 but that Postfix is
> looking for or replying with TLSv1.3.  Would that cause a problem with the
> certificate being recognised by the client?

As I already explained, OpenSSL logs all TLS protocol versions that are
SSLv3 or later via code that handles these protocols *generically*, so
there is fact no SSLv3 going on here.  Neither the client nor the server
are actually attempting to do SSLv3 as such.  In fact the client offers
TLSv1.3 and the server reciprocates.

The *ONLY* problem is that the client is not configured to trust the
server's certificates.  The only way to solve the problem is to accept
this as a fact, and not get distracted looking for unrelated issues.

This means that the client's Java trust store does not include the
issuer CA of the server certificate, and/or perhaps the server's
chain is incomplete (missing intermediate certificates).

It is not productive to get distracted by protocol versions, ...

--
    Viktor.

[James B. Byrne]

On Tue, December 22, 2020 12:46, Viktor Dukhovni wrote:

> On Tue, Dec 22, 2020 at 09:24:27AM -0500, James B. Byrne wrote:
>
>> > It clearly supports STARTTLS, since it is actually performing the TLS
>> > handshake, and abandons it after receiving certificates it is not happy
>> > with.
>>
>> Which confuses me, because I can use java to successfully negotiate a
>> certificate exchange with Postfix using the same keystore that the
>> application
>> is using.
>
> Perhaps you're confusing the Java keystore with the Java trust store,
> they are separate entities.  Java is a programming language, not a
> single application.  Each Java program gets to do its own thing.
> Yes, they often share some common libraries and file formats, ...

In this case, from what I can see in the application configuration files, the
keystore and the truststore are the same file:

<!-- ============================================================= -->
<!-- SSL ContextFactory configuration                              -->
<!-- ============================================================= -->
. . .
  <Set name="KeyStorePath"><Property name="jetty.base" default="." />/<Property
name="jetty.sslContext.keyStorePath" deprecated="jetty.keystore"
default="etc/keystore"/></Set>
. . .
  <Set name="TrustStorePath"><Property name="jetty.base" default="."
/>/<Property name="jetty.sslContext.trustStorePath"
deprecated="jetty.truststore" default="etc/keystore"/></Set>
. . .

>
> Well, you're running tests with completely different Java client
> applications and likely in fact different configuration settings.  All
> that's in common is that both are Java, which is something, but not
> much...

Granted, but short of writing my own mail agent in Java, a language that I have
next to no experience with, I have to use the tools that I can find and which
are simple enough for me to use.  SSLPoke is just using the Java SSL libraaries
to exercise connectivity, and for a very long time that was the issue.  I am
now past that stage.

>
> Note that certificates don't pass "keystore" validation, they pass
> "trust store" validation.  Java keystores are for your own private
> key and cert (akin to PKCS12 containers, or a PEM with key + cert
> chain), while trust stores are akin to a cert bundle with lots of
> CA certs.

As shown above, in this case the application configured to use the same file as
both keystore and truststore.  In fact, in the SSLPoke tests I was only passing
the truststore arguments. But if I do use both key and truststore arguments
then I get the same result:

JAVA_VERSION="12" java \
-Djavax.net.ssl.trustStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore
\
-Djavax.net.ssl.keyStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore \
-Djavax.net.ssl.trustStorePassword=testing \
-Djavax.net.ssl.keyStorePassword=testing \
SSLPoke 192.168.216.32 465
Successfully connected

>
>> I know that the service is different on both ports, but the
>> certificate acceptability should be the same.  And it is the client
>> that is causing the problem.
>
> You could hypothetically have different certificate settings for
> the different ports in master.cf, but if you don't then indeed
> the server side TLS behaviour is likely the same across the board.
>

I do not.  And, I believe we are past the phase where the truststore
verification is the issue.  I may be wrong but the evidence from SSLPoke and
KeyStore Explorer tells against verification failing.

I am suspicious of the SSLv3 / TLSv1.3 contratemps shown in the Postfix log
files.  But I have no idea how that would make a certificate unacceptable to
the client.  Unless it could not read it?




--
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
   Unencrypted messages have no legal claim to privacy
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: [phone]
Hamilton, Ontario             fax: [phone]
Canada  L8E 3C3

[Viktor Dukhovni]

On Tue, Dec 22, 2020 at 01:21:06PM -0500, James B. Byrne wrote:

> > You could hypothetically have different certificate settings for
> > the different ports in master.cf, but if you don't then indeed
> > the server side TLS behaviour is likely the same across the board.
>
> I do not.  And, I believe we are past the phase where the truststore
> verification is the issue.  I may be wrong but the evidence from SSLPoke and
> KeyStore Explorer tells against verification failing.
>
> I am suspicious of the SSLv3 / TLSv1.3 contratemps shown in the Postfix log
> files.  But I have no idea how that would make a certificate unacceptable to
> the client.  Unless it could not read it?

Your suspicions are unfounded.  The client is rejecting the server's
certificate chain with a fatal certificate unknown alert.  That's the
issue to fix.  All else is distraction.

--
    Viktor.

[James B. Byrne]

On Tue, December 22, 2020 13:27, Viktor Dukhovni wrote:

>
> Your suspicions are unfounded.  The client is rejecting the server's
> certificate chain with a fatal certificate unknown alert.  That's the
> issue to fix.  All else is distraction.
>

After reviewing Postix logs with smtpd_tls_logging turned up to 3 I arrived at
the same conclusion a little while ago.  I am just bereft of ideas as to how to
proceed at the moment.


--
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
   Unencrypted messages have no legal claim to privacy
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: [phone]
Hamilton, Ontario             fax: [phone]
Canada  L8E 3C3

[Wietse Venema]

James B. Byrne:
[ Charset ISO-8859-1 converted... ]

>
>
> On Tue, December 22, 2020 13:27, Viktor Dukhovni wrote:
>
> >
> > Your suspicions are unfounded.  The client is rejecting the server's
> > certificate chain with a fatal certificate unknown alert.  That's the
> > issue to fix.  All else is distraction.
> >
>
> After reviewing Postix logs with smtpd_tls_logging turned up to 3
> I arrived at the same conclusion a little while ago.  I am just
> bereft of ideas as to how to proceed at the moment.

Here is a suggestion:

- You see a different behaviors with the Javamail app on port 25
and some other Java tool on port 465. One rejects the server
certficate, while other does not.

- The Postfix services on port 465 and 25 use the same TLS settings
(apart from tls_wrappermode which is enabled on port 465).

- The difference must then be on the client side, in the way that
the client verifies the server cerificate.

Therefore I suggest that you move your attention to the client side.

        Wietse

[Viktor Dukhovni]

On Tue, Dec 22, 2020 at 02:26:08PM -0500, James B. Byrne wrote:

> > Your suspicions are unfounded.  The client is rejecting the server's
> > certificate chain with a fatal certificate unknown alert.  That's the
> > issue to fix.  All else is distraction.
>
> After reviewing Postix logs with smtpd_tls_logging turned up to 3 I arrived at
> the same conclusion a little while ago.  I am just bereft of ideas as to how to
> proceed at the moment.

There's nothing further you can learn on the Postfix side of things, the
only actionable information available is that the client is unable to
verify the server's certificate chain.

Assuming the server is returning a sufficiently complete chain
(including all requisite intermediate certs), all further debugging is
entirely a client-side problem, with whatever diagnostic tools and
logging are available there.

There is nothing further that Postfix can do for you.

Though mostly unlikely, one last thing to consider is whether the
client in question has (possibly broken) DANE support?  If the server in
question has DANE TLSA records like your public MX hosts:

    harte-lyne.ca. IN MX 70 mx31.harte-lyne.ca.
    harte-lyne.ca. IN MX 80 mx32.harte-lyne.ca.
    harte-lyne.ca. IN MX 90 mx131.harte-lyne.ca.
    harte-lyne.ca. IN MX 100 mx132.harte-lyne.ca.
    harte-lyne.ca. IN MX 200 mx118.harte-lyne.ca.
    harte-lyne.ca. IN MX 200 mx119.harte-lyne.ca.
    mx31.harte-lyne.ca. IN A 216.185.71.31
    mx31.harte-lyne.ca. IN AAAA ?
    _25._tcp.mx31.harte-lyne.ca. IN CNAME _tlsa._dane.trust.harte-lyne.ca.
    _tlsa._dane.trust.harte-lyne.ca. IN TLSA 2 0 2 67274b355428905895c6b581950e0ed4f7d043f31f7e7020b716b7faa06776b6aadd33e127624b6e8c75c520a01d9cad3bd29f18fa7dcb3d5fd3917510e6722a
    _tlsa._dane.trust.harte-lyne.ca. IN TLSA 2 1 2 380259229e21a1946b38cfc594cbc993b61bc93762b7b6c6637b3eef9c5a2bb70c589b91beb73bd1304eac11b3917e33819e2b47d25d4966435a2a3e83c1f80f
    _tlsa._dane.trust.harte-lyne.ca. IN TLSA 2 1 2 c26e0ec16a46a97386e8f31f8ecc971f2d73136aa377dfdaac2b2b00f7cab4bb29b17d913c82093b41fd0d9e40b66a68361c126f1f4017f9ce60eabc5adba90e
    mx32.harte-lyne.ca. IN A 216.185.71.32
    mx32.harte-lyne.ca. IN AAAA ?
    _25._tcp.mx32.harte-lyne.ca. IN CNAME _tlsa._dane.trust.harte-lyne.ca.
    mx131.harte-lyne.ca. IN A 216.185.71.131
    mx131.harte-lyne.ca. IN AAAA ?
    _25._tcp.mx131.harte-lyne.ca. IN CNAME _tlsa._dane.trust.harte-lyne.ca.
    mx132.harte-lyne.ca. IN A 216.185.71.132
    mx132.harte-lyne.ca. IN AAAA ?
    _25._tcp.mx132.harte-lyne.ca. IN CNAME _tlsa._dane.trust.harte-lyne.ca.
    mx118.harte-lyne.ca. IN A 216.185.71.118
    mx118.harte-lyne.ca. IN AAAA ?
    _25._tcp.mx118.harte-lyne.ca. IN CNAME _tlsa._dane.trust.harte-lyne.ca.
    mx119.harte-lyne.ca. IN A 216.185.71.119
    mx119.harte-lyne.ca. IN AAAA ?
    _25._tcp.mx119.harte-lyne.ca. IN CNAME _tlsa._dane.trust.harte-lyne.ca.

Perhaps the client is failing to verify those.  I'm not aware of any
mainstream Java Mail implementations that support DANE (correctly or
otherwise), but perhaps some are now becoming available.

--
    Viktor.

On 22 Dec 2020, at 07:19, James B. Byrne <[hidden email]> wrote:
> it appears to me that
> connecting to port 25 with STARTTLS is the accepted practice.

The recommendation for years has been to use port 587 (submission) for STARTTLS and to never use port 25 for anything but server-to-server communication.

--
When the stars threw down their spears And watered heaven with their
        tears, Did He smile his work to see? Did He who made the Lamb
        make thee?

[Jaroslaw Rafa]

Dnia 23.12.2020 o godz. 09:32:31 @lbutlr pisze:
>
> The recommendation for years has been to use port 587 (submission) for
> STARTTLS and to never use port 25 for anything but server-to-server
> communication.

And server-to-server communication nowadays uses STARTTLS, doesn't it?
--
Regards,
   Jaroslaw Rafa
   [hidden email]
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

[Wietse Venema]

Jaroslaw Rafa:
> Dnia 23.12.2020 o godz. 09:32:31 @lbutlr pisze:
> >
> > The recommendation for years has been to use port 587 (submission) for
> > STARTTLS and to never use port 25 for anything but server-to-server
> > communication.
>
> And server-to-server communication nowadays uses STARTTLS, doesn't it?

Indeed.

smtp/25         MTA to MTA, STARTTLS usually optional
smtps/465       MUA to MTA, TLS wrappermode (mandatory TLS)
submission/587  MUA to MTA, STARTTLS usually required

        Wietse

[John Stoffel-2]

>>>>> "James" == James B Byrne <[hidden email]> writes:

James> On Mon, December 21, 2020 20:00, Jaroslaw Rafa wrote:
>>
>> If you are able to connect via 465, then maybe the application just isn't
>> designed to use "inline" TLS, but rather uses only SMTP-over-TLS? The latter
>> is supported on port 465, while submission via port 587 requires first
>> plaintext connection and then dynamic in-session switchover to TLS, using
>> STARTTLS command. Maybe your application just does not support that?

James> That is a possibility.  However, having looked at the example
James> configurations and discussions on the application support
James> groups, it appears to me that connecting to port 25 with
James> STARTTLS is the accepted practice.

James> The problem I had with the certificate negotiation is not
James> uncommon with this application, due to Java's rather
James> idiosyncratic PKI certificate handling. I had reason to believe
James> that when I finally solved that problem then everything else
James> would just work.  The SSLv3 problem was a surprise, but was
James> easily compensated for.  But, I still cannot get this
James> application to send email so there must be something else that
James> I have done, or not done, which is preventing this from
James> working.

James> And it seems that I am past the SSL problems and into SMTP, I hope.

Why don't you setup a local only postfix instance on the same host as
the application, which only listed on 127.0.0.1:25, which the dumb
Java app can then send email through *without encryption*, then let
the local postfix instance do all the hard work of sending the email
to other servers using encryption?

It seems like the simpler setup to get working right, and gets most of
the pain in the Java app out of the way.

John

[James B. Byrne]

On Fri, December 25, 2020 12:43, John Stoffel wrote:

>
> Why don't you setup a local only postfix instance on the same host as
> the application, which only listed on 127.0.0.1:25, which the dumb
> Java app can then send email through *without encryption*, then let
> the local postfix instance do all the hard work of sending the email
> to other servers using encryption?
>
> It seems like the simpler setup to get working right, and gets most of
> the pain in the Java app out of the way.
>
> John
>

That is reasonable suggestion and is a possibility which I have considered
several times.  However, in this case the application is being evaluated for
its suitability to replace and existing one.  Bypassing this particular problem
with Postfix only exposes the next arising from the application's unwillingness
to accept our certificates.

It is established to my satisfaction that this is not a Postfix issue.
Therefore,this has to be a configuration issue within the application itself.
If I do not solve it now then at some point it is going to cause another
problem, likely when we do not have the luxury of time to figure it out.

Regards,

--
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
   Unencrypted messages have no legal claim to privacy
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:[hidden email]
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: [phone]
Hamilton, Ontario             fax: [phone]
Canada  L8E 3C3

[John Stoffel-2]

>>>>> "James" == James B Byrne <[hidden email]> writes:

James> On Fri, December 25, 2020 12:43, John Stoffel wrote:

>>
>> Why don't you setup a local only postfix instance on the same host as
>> the application, which only listed on 127.0.0.1:25, which the dumb
>> Java app can then send email through *without encryption*, then let
>> the local postfix instance do all the hard work of sending the email
>> to other servers using encryption?
>>
>> It seems like the simpler setup to get working right, and gets most of
>> the pain in the Java app out of the way.
>>
>> John
>>

James> That is reasonable suggestion and is a possibility which I have
James> considered several times.  However, in this case the
James> application is being evaluated for its suitability to replace
James> and existing one.  Bypassing this particular problem with
James> Postfix only exposes the next arising from the application's
James> unwillingness to accept our certificates.

Sure, then the application is pretty much failing your acceptance
tests.  *grin*.

James> It is established to my satisfaction that this is not a Postfix
James> issue.  Therefore,this has to be a configuration issue within
James> the application itself.  If I do not solve it now then at some
James> point it is going to cause another problem, likely when we do
James> not have the luxury of time to figure it out.

Good luck then!