I'm trying to connect x11vnc server through VncViewer (TigerVnc). I'm giving following command to start vnc server with -ssl option
$ x11vnc -ssl -rfbauth /home/root/.vnc/passwd
When I'm trying to connect the vnc server, I'm getting TLS handshake failed error.
Following is the vncserver log
07/05/2020 13:25:58 SSL: accept_openssl(OPENSSL_VNC)
07/05/2020 13:25:58 SSL: spawning helper process to handle: 172.21.33.47:64181
07/05/2020 13:25:58 SSL: helper for peerport 64181 is pid 629:
07/05/2020 13:25:58 connect_tcp: trying: 127.0.0.1 20000
07/05/2020 13:25:59 check_vnc_tls_mode: waited: 1.422183 / 1.40 input: (future) RFB Handshake
07/05/2020 13:26:00 check_vnc_tls_mode: version: 3.8
07/05/2020 13:26:00 check_vnc_tls_mode: reply: 19 (VeNCrypt)
07/05/2020 13:26:00 vencrypt: received 0.2 client version.
07/05/2020 13:26:00 vencrypt: client selected sub-type: 257 (rfbVencryptTlsNone)
07/05/2020 13:26:00 Using Anonymous Diffie-Hellman mode.
07/05/2020 13:26:00 WARNING: Anonymous Diffie-Hellman uses encryption but is
07/05/2020 13:26:00 WARNING: susceptible to a Man-In-The-Middle attack.
07/05/2020 13:26:00 loaded Diffie Hellman 1024 bits, 0.001s
07/05/2020 13:26:00 SSL: ssl_init[629]: 6/6 initialization timeout: 20 secs.
07/05/2020 13:26:01 SSL: ssl_helper[629]: SSL_accept() *FATAL: -1 SSL FAILED
07/05/2020 13:26:01 SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher
07/05/2020 13:26:01 SSL: ssl_helper[629]: Proto: unknown
07/05/2020 13:26:01 SSL: accept_openssl: cookie from ssl_helper[629] FAILED. 0
This was working properly without error. Only change is openssl version, It changed from OpenSSL 1.0.2p 14 Aug 2018 to OpenSSL 1.1.1d 10 Sep 2019.
I checked that OpenSSL 1.1.1d support TLSv1.3 & TigerVNC supports TLSv1.2, Is the error because of version mismatch in SSL version?
I found the solution for my question
07/05/2020 13:26:01 SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher
This means, the client and server have failed to find a common set of ciphersuites that they both support.
07/05/2020 13:26:00 Using Anonymous Diffie-Hellman mode.
07/05/2020 13:26:00 WARNING: Anonymous Diffie-Hellman uses encryption but is
07/05/2020 13:26:00 WARNING: susceptible to a Man-In-The-Middle attack.
This suggests that I am using ADH based ciphersuites in x11vnc server which are not available in the default security level in 1.1.1. This was not the case in 1.0.2.
Solution for this problem is lower the security level in use to not complain about the use of ADH.
I lowered the security level of x11vnc server from ADH:@STRENGHT to ADH:@SECLEVEL=0 & it worked.