I'm running the popular open-source mediasoup WebRTC server on a standalone box with no firewalls, fresh SSL certs installed and literally nothing else running on the machine (no web server, no database server, nothing). When my video chat app tries to stream video through this server, to a client using Firefox, mediasoup emits an SSL error (2nd line below):
mediasoup:WARN:mediasoup-worker [id:gxzingwt#3] RTC::DtlsTransport::OnSslInfo() | received DTLS fatal alert: bad certificate +0ms
mediasoup:ERROR:mediasoup-worker [id:gxzingwt#3] RTC::DtlsTransport::CheckStatus() | OpenSSL error [desc:'SSL status: SSL_ERROR_SSL', error:'error:14102412:SSL routines:dtls1_read_bytes:sslv3 alert bad certificate'] +0ms
mediasoup:WARN:mediasoup-worker [id:gxzingwt#3] RTC::DtlsTransport::CheckStatus() | connection failed +1ms
mediasoup:WARN:mediasoup-worker [id:gxzingwt#3] RTC::DtlsTransport::Reset() | resetting DTLS transport +1ms
mediasoup:WARN:mediasoup-worker [id:gxzingwt#3] RTC::WebRtcTransport::OnDtlsFailed() | DTLS failed +0ms
Now here's what weird, this ONLY happens with Firefox browser. For Safari, Chrome, and Safari iOS (phone/iPad), no problems at all. If I have 5 people broadcasting in my video chat room, I can open those and see/stream the video with no problems from the other browsers. But Firefox (the latest version, both developer edition and regular Firefox) cause mediasoup to throw that SSL error and the stream can't resolve and show the video to the user.
mediasoup is running on port 8443 and I've also tried it on port 443. I've checked the debug logs and can't find any other problems-- this SSL issue is it.
I thought perhaps my Let's Encrypt certs were bad, so I have revoked, deleted, and re-issued the SSL certs for the domain. No problems there or anything strange.
I've run openssl s_client -connect [my domain running mediasoup].com:8443 -showcerts and everything looks great, certs are fine-- the format matches the same output as my valid certs on my web server machine as well the same output from as a friend's WebRTC server. I also tried openssl s_client -host [my domain].com -port 8443 -cert /etc/letsencrypt/live/[my domain].com/cert.pem -key /etc/letsencrypt/live/[my domain].com/privkey.pem on the machine itself, all good. So I don't think it's the certs.
Since it's only Firefox giving these problems, I thought maybe Firefox was caching an old/invalid certificate somehow, so I deleted and cleaned Firefox's cache, and also looked in my Mac's Keychain Access app to see if there were any certs in there for the domain where mediasoup is running (there weren't).
What is going on?! ;-) Why does Firefox-- and only Firefox!-- cause mediasoup to throw this error, and prevent my users with Firefox from using the video chat? Again: with Chrome, Safari, iPad/iPhone, everything works perfectly.
(Or, how can I get that mediasoup-worker process to tell me more about EXACTLY why it doesn't like the SSL cert?)
(Also, why would it only be Firefox that is having problems? Could it be a certificate that is stored in Firefox that it sends to the mediasoup server to talk to it? I know the WebRTC protocol is complex and has a lot of negotiations, perhaps that's part of it... like maybe it's referring to a different certificate than the one on the server domain?)
the mediasoup forum is here: https://mediasoup.discourse.group
BTW, as Philipp says, mediasoup "is not listening in port 8443 or 443". That's the WebSocket server of the mediasoup-demo project. mediasoup does not listen by itself for HTTP or WebSocket at all.
If you don't provide your TLS certificate to mediasoup.createWorker() then mediasoup will create its own random certificates for DTLS. And it's in DTLS where you see that error.
BTW I've tested in latest Firefox stable (68.0.2) and Nightly (70.0a1 (2019-08-14)) in OSX and I don't get any SSL error in DTLS, so honestly no idea.
PS: If you can provide more info (for example which mediasoup version you are using), please open an issue in mediasoup Discourse Group in the "mediasoup-demo" category.