springsslopenssltls1.3ubuntu-24.04

Ubuntu24.04 SSL handshake failed when use self-signed cert


There is a server(java spring) running in HTTP, I want to switch to HTTPS. So I use openssl to generate self-signed cert and key, and test them like below:

Curl verison:

curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.0.13 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.2.0 libpsl/0.21.2 (+libidn2/2.3.7) libssh/0.10.6/openssl/zlib nghttp2/1.59.0 librtmp/2.3 OpenLDAP/2.6.7
Release-Date: 2023-12-06, security patched: 8.5.0-2ubuntu10.4
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Openssl version:

OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

openssl s_server -accept 10001 -key server.key -cert server.crt Output:

Using default temp DH parameters
ACCEPT
-----BEGIN SSL SESSION PARAMETERS-----
MIGCAgEBAgIDBAQCEwIEICifjaMkuqnBMiNQB4qri/5IYwhr6Lnth70WiCRiFE7L
BDCPs1X5f168KC57bYp0dz1Mv4NJs/Hk04N1H1pBsXpZxS3EjeLqEi28XWyUvAsS
sDmhBgIEZzrmlaIEAgIcIKQGBAQBAAAArgYCBHCDhdKzAwIBHQ==
-----END SSL SESSION PARAMETERS-----
Shared ciphers:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA
Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Shared Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Supported groups: x25519:secp256r1:x448:secp521r1:secp384r1:ffdhe2048:ffdhe3072:ffdhe4096:ffdhe6144:ffdhe8192
Shared groups: x25519:secp256r1:x448:secp521r1:secp384r1:ffdhe2048:ffdhe3072:ffdhe4096:ffdhe6144:ffdhe8192
CIPHER is TLS_AES_256_GCM_SHA384
Secure Renegotiation IS supported

openssl s_client -connect localhost:10001 Output:

CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 C = CN, ST = Zhejiang, L = Hangzhou, O = NIC CERNET, OU = CERNET, CN = nic.edu.cn
verify return:1
depth=0 C = CN, ST = Zhejiang, L = Hangzhou, O = NIC CERNET, OU = CERNET, CN = nic.edu.cn
verify return:1
---
Certificate chain
 0 s:C = CN, ST = Zhejiang, L = Hangzhou, O = NIC CERNET, OU = CERNET, CN = nic.edu.cn
   i:C = CN, ST = Zhejiang, L = Hangzhou, O = NIC CERNET, OU = CERNET, CN = nic.edu.cn
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 18 03:04:57 2024 GMT; NotAfter: Nov 16 03:04:57 2034 GMT
 1 s:C = CN, ST = Zhejiang, L = Hangzhou, O = NIC CERNET, OU = CERNET, CN = nic.edu.cn
   i:C = CN, ST = Zhejiang, L = Hangzhou, O = NIC CERNET, OU = CERNET, CN = nic.edu.cn
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 18 03:02:33 2024 GMT; NotAfter: Nov 16 03:02:33 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIERjCCAy6gAwIBAgIULlOVrrGw65XJnmTFRnoH0ZhCdLowDQYJKoZIhvcNAQEL
BQAwbjELMAkGA1UEBhMCQ04xETAPBgNVBAgMCFpoZWppYW5nMREwDwYDVQQHDAhI
YW5nemhvdTETMBEGA1UECgwKTklDIENFUk5FVDEPMA0GA1UECwwGQ0VSTkVUMRMw
...

They work fine. However, I take error when I'm trying to config server with them: openssl s_client -connect localhost:43521 -tls1_3

CONNECTED(00000003)
Can't use SSL_get_servername
4057DC39227F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1599:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 243 bytes and written 225 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

more details(acutally it's the log when i use cert of let's encrypt(also a "fake" cert which was used for my another website<xxx.xx.xx>, i just edit /etc/hosts to resolve <xxx.xx.xx> to this server's ip), but it's same with the case i use self-signed cert):

openssl s_client -connect 0.0.0.0:34413 -status -msg -debug

CONNECTED(00000003)
>>> TLS 1.0, RecordHeader [length 0005]
    16 03 01 01 29
>>> TLS 1.3, Handshake [length 0129], ClientHello
    01 00 01 25 03 03 2f 4c d4 4e 7e 35 01 8c 58 61
    07 51 7a 8f 9d 1c 81 60 59 75 e5 ea 46 2b 23 62
    80 e6 fd da 1d 6d 20 d6 a3 61 dc de c5 6d 43 90
    4b f2 42 9e f4 a4 81 72 ea 96 1e 19 74 4b 1a 12
    66 55 ad 2b 11 88 6e 00 3e 13 02 13 03 13 01 c0
    2c c0 30 00 9f cc a9 cc a8 cc aa c0 2b c0 2f 00
    9e c0 24 c0 28 00 6b c0 23 c0 27 00 67 c0 0a c0
    14 00 39 c0 09 c0 13 00 33 00 9d 00 9c 00 3d 00
    3c 00 35 00 2f 00 ff 01 00 00 9e 00 0b 00 04 03
    00 01 02 00 0a 00 16 00 14 00 1d 00 17 00 1e 00
    19 00 18 01 00 01 01 01 02 01 03 01 04 00 23 00
    00 00 05 00 05 01 00 00 00 00 00 16 00 00 00 17
    00 00 00 0d 00 2a 00 28 04 03 05 03 06 03 08 07
    08 08 08 09 08 0a 08 0b 08 04 08 05 08 06 04 01
    05 01 06 01 03 03 03 01 03 02 04 02 05 02 06 02
    00 2b 00 05 04 03 04 03 03 00 2d 00 02 01 01 00
    33 00 26 00 24 00 1d 00 20 c3 25 0e 3b 83 b8 ec
    70 4e df 28 91 75 87 b0 4b 2a 76 ed 50 2b b1 c5
    e0 3a 6a a7 d3 d2 a7 29 5e
write to 0x5d39b41639d0 [0x5d39b4250f30] (302 bytes => 302 (0x12E))
0000 - 16 03 01 01 29 01 00 01-25 03 03 2f 4c d4 4e 7e   ....)...%../L.N~
0010 - 35 01 8c 58 61 07 51 7a-8f 9d 1c 81 60 59 75 e5   5..Xa.Qz....`Yu.
0020 - ea 46 2b 23 62 80 e6 fd-da 1d 6d 20 d6 a3 61 dc   .F+#b.....m ..a.
0030 - de c5 6d 43 90 4b f2 42-9e f4 a4 81 72 ea 96 1e   ..mC.K.B....r...
0040 - 19 74 4b 1a 12 66 55 ad-2b 11 88 6e 00 3e 13 02   .tK..fU.+..n.>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa   .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27   .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d   .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 9e   ...=.<.5./......
0090 - 00 0b 00 04 03 00 01 02-00 0a 00 16 00 14 00 1d   ................
00a0 - 00 17 00 1e 00 19 00 18-01 00 01 01 01 02 01 03   ................
00b0 - 01 04 00 23 00 00 00 05-00 05 01 00 00 00 00 00   ...#............
00c0 - 16 00 00 00 17 00 00 00-0d 00 2a 00 28 04 03 05   ..........*.(...
00d0 - 03 06 03 08 07 08 08 08-09 08 0a 08 0b 08 04 08   ................
00e0 - 05 08 06 04 01 05 01 06-01 03 03 03 01 03 02 04   ................
00f0 - 02 05 02 06 02 00 2b 00-05 04 03 04 03 03 00 2d   ......+........-
0100 - 00 02 01 01 00 33 00 26-00 24 00 1d 00 20 c3 25   .....3.&.$... .%
0110 - 0e 3b 83 b8 ec 70 4e df-28 91 75 87 b0 4b 2a 76   .;...pN.(.u..K*v
0120 - ed 50 2b b1 c5 e0 3a 6a-a7 d3 d2 a7 29 5e         .P+...:j....)^
read from 0x5d39b41639d0 [0x5d39b4247d03] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 7a                                    ....z
<<< TLS 1.2, RecordHeader [length 0005]
    16 03 03 00 7a
read from 0x5d39b41639d0 [0x5d39b4247d08] (122 bytes => 122 (0x7A))
0000 - 02 00 00 76 03 03 93 c3-c8 0e 77 d3 79 72 55 d3   ...v......w.yrU.
0010 - 46 b6 d3 94 2b 70 db ad-fb 53 67 b3 75 c1 90 0b   F...+p...Sg.u...
0020 - 0c 9a f7 9e 60 b5 20 d6-a3 61 dc de c5 6d 43 90   ....`. ..a...mC.
0030 - 4b f2 42 9e f4 a4 81 72-ea 96 1e 19 74 4b 1a 12   K.B....r....tK..
0040 - 66 55 ad 2b 11 88 6e 13-02 00 00 2e 00 2b 00 02   fU.+..n......+..
0050 - 03 04 00 33 00 24 00 1d-00 20 35 10 ab 7b b7 81   ...3.$... 5..{..
0060 - 1b 2e 4f de bb 18 53 1a-36 43 75 57 0a f9 86 4e   ..O...S.6CuW...N
0070 - 7f 43 55 ad 48 f3 5a 18-e3 11                     .CU.H.Z...
<<< TLS 1.3, Handshake [length 007a], ServerHello
    02 00 00 76 03 03 93 c3 c8 0e 77 d3 79 72 55 d3
    46 b6 d3 94 2b 70 db ad fb 53 67 b3 75 c1 90 0b
    0c 9a f7 9e 60 b5 20 d6 a3 61 dc de c5 6d 43 90
    4b f2 42 9e f4 a4 81 72 ea 96 1e 19 74 4b 1a 12
    66 55 ad 2b 11 88 6e 13 02 00 00 2e 00 2b 00 02
    03 04 00 33 00 24 00 1d 00 20 35 10 ab 7b b7 81
    1b 2e 4f de bb 18 53 1a 36 43 75 57 0a f9 86 4e
    7f 43 55 ad 48 f3 5a 18 e3 11
read from 0x5d39b41639d0 [0x5d39b4247d03] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01                                    .....
<<< TLS 1.2, RecordHeader [length 0005]
    14 03 03 00 01
read from 0x5d39b41639d0 [0x5d39b4247d08] (1 bytes => 1 (0x1))
0000 - 01                                                .
read from 0x5d39b41639d0 [0x5d39b4247d03] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 41                                    ....A
<<< TLS 1.2, RecordHeader [length 0005]
    17 03 03 00 41
read from 0x5d39b41639d0 [0x5d39b4247d08] (65 bytes => 65 (0x41))
0000 - 87 70 1b 6b dd 31 ac 68-d7 c0 cf 3a bb 8b a6 e2   .p.k.1.h...:....
0010 - 69 2b ff d0 23 37 33 82-bb 6d ff b3 5d 6f 09 40   i+..#73..m..]o.@
0020 - 2d 15 76 4b 88 03 76 ea-98 91 1a 94 26 5f fa 02   -.vK..v.....&_..
0030 - 2c 1d 99 79 59 5a 34 15-10 05 e2 d4 3c 98 80 c4   ,..yYZ4.....<...
0040 - f1                                                .
<<< TLS 1.3, InnerContent [length 0001]
    16
<<< TLS 1.3, Handshake [length 0020], EncryptedExtensions
    08 00 00 1c 00 1a 00 0a 00 16 00 14 00 1d 00 17
    00 18 00 19 00 1e 01 00 01 01 01 02 01 03 01 04
Can't use SSL_get_servername
read from 0x5d39b41639d0 [0x5d39b4247d03] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 23                                    ....#
<<< TLS 1.2, RecordHeader [length 0005]
    17 03 03 00 23
read from 0x5d39b41639d0 [0x5d39b4247d08] (35 bytes => 35 (0x23))
0000 - 1e b6 32 85 6e 95 a4 42-39 70 3a ce c0 fa 89 41   ..2.n..B9p:....A
0010 - ff de 16 15 b3 01 f2 f3-f7 d3 26 dd 61 98 5d 8f   ..........&.a.].
0020 - d1 93 48                                          ..H
<<< TLS 1.3, InnerContent [length 0001]
    15
<<< TLS 1.3, Alert [length 0002], fatal handshake_failure
    02 28
40E7B8B150760000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1599:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 243 bytes and written 302 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x5d39b41639d0 [0x5d39b411f320] (8192 bytes => 0)

Solution

  • Resolved, it was caused by a parsing error with my server private key.