I am writing a small WebRTC client as a hobby project , right now I am stuck in DTLS implementation and cannot figure out what is going wrong with the DTLS handshake. I am trying to make a p2p WebRTC connection from My WebRTC implementation and Gstreamer Implementation
I send the client hello as a DTLS client to gstreamer client it responds with server hello certificate key ..... Then I respond with the FLIGHT 4 messages as displayed in below diagram but I do not get a response back for flight 5 from Gstreamer.
(i have attached wireshark pcap and Gstreamer logs ) There Might be some thing wrong with flight 4 that I am sending but I cannot figure out the problem , I have verified all the messages are correct by manually calculating every thing from premaster secret ,master secret , RSA AES ....
I have implemented TLS_RSA_WITH_AES_128_CBC_SHA
premaster secret : FEFD9E38B76CF5780B6C58CBB55309C1C08EE1B5DF7D4507159115A8D459C35936FEEC2344698CF4A7F0689714C0BA19
client random : 39623565396565382d333032332d343865642d383064392d6432346165616538
server random : a9dbe285e2ef980972812e45c86f6aaff1d486212bbb2d93744d2bae14813ae3
SENDING encrypted PREMSATER KEY TO SERVER :
non-encrypted padded premaster key
0225E7E44EF4694A39D4F4C458DC604DFBD806F2DB0FEF4136FC14FBC827541525900DACB6CEACE1679886270CEF2D9DEEEB05BB773A579B98718DD798DE2A1BE049C869201E9480C521691B25629D1D76C08BE92A7F99718720BBB2CC9BFA151E060BEFDF9BDDAC22B601557FC97E65A583D889A65A28EB2C22717AC2FEC6D82277D6F9E6FFD7F7A216EC4783E4F38E8C0EA393B4080D1FC73D8ED526EBC7466173563CC2B668E12104D85210246CE50A90D689238260FD7ACF9342DC4DDEC0BBDC7884FAB521DEE2DABF8DBAF300FEFD369DD3FE67938055892D6CF4C910C6A06F68524FAF557A087C22CEAAFD53038168EF713C58591157CD6F943FE6E
server public key modulus and exponent
modulus:
8F207DA9E4D246C1C5D002C38C5983CDB0A8C86C8DDE6E7D2249BB1EC85F15CAEA4A0E4EC2EFEBF45A6CCCB1733E208D2852A986F458D60A14E386B6D35717B8C770957B9AD92ED44E19DA73F7626DA8E3329785137044AB8EE89D34F0A3BDE27E2E892BC78AAD904C038F6BCD1AAFA96E7808B0950423B0FDD26720EF2561F779C8309DDA5A6980823E0B1C03B3171E7C63997B606DB1E5BF64F6ABB72D2BBFCE1694CEA9B31D976163ECB3EDB4AB471D187C69FCAA0194A94F969DDD9BA2E59A1618FF1DCBC2C71062FF95DDD6CDEAEC0581D3FA685542DE68DA13E51B38EB8C74A69DADCD94287DC422ABF5FB7488E8122FC3FEC227BE0BA5A4A5874DA84D
exponent : 010001
encrypted premaster key :
0194DF7D60F9992D633617A3D3E66F828D9D18ACEC75778808066A17A6F2625761901FF3C88970C588060358551F7B9F65DD6DC510525487C10A85FFCE33962C719113B18FF1D197CF96889E2A8348A0135939E139CEB5BCD6D9FF9D707604B455119AA97C4ADB1EBEF84101D7B1B9860241FF601DE4FDC79DB4901A1F43E71942E5044828D5106862B1FEDE79F1F928B05C55EDD796AD0F5BC6C56BC8386FA4F331B8292A4C5FEB9D98D087D2DC34AD1CF608FE40BDACAB8B679CEE40863E65E8687C90F73BB8C61530CD19CD58C63A1ABFAB7F2E9BA400210C22EB16F3BA57291EF53B2BC28B33B9817AB5CDE7559BB957470252B29E2434DBB264EB39117A
I have encrypted the above message with my server's certificate modulus and exponent and send it in key exchange message
DERIVING MASTER SECRET FROM PREMASTER SECRET: prf result for master secret
991FE0D757F2E3422CB36B287974B8F1CFEA6EA464674D8CCCCA9111AF817BA0934F07F7F376B5ED1F507F7575A7CA03
key expansion for deriving keying material
b2693e4352ac4a9669d92b4ba2a3153a9556b3ff82a347a85713bb4864e8d3cfbed57e8c60f6965cfc10b7c85a88934fa83fe883ee02f1a4fd6392045b87d38775c5e059f75613fefa3208b78d0b61f0245fcc6e10d68c56ab661da1641d132cb0a71ce0fd074cf6149826dbdd70bfc8d6139fccbdbe3182ad80bba13254339d
derived keys:
AES CLIENT key : 88 04 d7 e0 41 c1 5a b4 3a f7 7f 40 d9 fd 0d 13
AES CLIENT IV : 3a f3 3d 4c 30 57 14 6a 22 50 c7 56 97 c3 75 f7
I am then using the above keys to encrypt the finish message with aes cbc and send it.
I can confirm that the certificate verify message is correct because when I send wrong data in certificate verify message then gstreamer responds with bad signature .
non-encrypted Finish message with padding
14 00 00 0c 00 00 00 00 00 00 00 0c ad f8 6f 17 16 a9 3f df 10 b9 c1 9b ca d1 f7 d2 21 58 97 26 72 08 15 bb d8 f0 61 81 49 4e 79 69 03 03 03 03
encrypted finish message
(first 16 bytes are client IV in the message)
3af33d4c3057146a2250c75697c375f710d8335f318be6ee18f3d49be4300e77bbb4aab65eb9caa35e983bad7101bf16920f46cb7822bc67b1801dbf0e0a1ea7
I have verified the encrypted message is correctly encrypted using AES CBC by using this online tool to decrypt and verify that it gets me the original message.
(first 16 bytes is IV so skip them when decrypting ) Gstreamer Logs Wireshark pcap
UPDATE : I was able to decrypt the DTLS packets using master secret Wireshark
Wireshark Pcap file
Wireshark decryption log file
wireshark decryption premaster secret file
while inspecting the log file I can see that the expanded keys in wireshark are same as mine and finish message is decrypted correctly and mac is verified correctly and also I know that Gstreamer will return bad signature error when wrong sign is sent but right now I do not get such an error in the log that means that Client Verify is correct as well , then that leaves my suspicion to only Client key exchange message.
@nneonneo was correct The main problem was the sequence number of the "Finished" message in my implementation. Initially, I was comparing my packet sequence numbers against a packet capture that I downloaded from the internet. In that packet capture, the "Change Cipher Spec" message was counted as an increment in the sequence number.
I applied the same logic in my implementation, incrementing the sequence number for the "Change Cipher Spec" message. However, this approach led to the problem I was facing. After some investigation, I realized that the "Change Cipher Spec" message should not trigger a sequence number increment.
When I adjusted my code to not increment the sequence number when sending the "Change Cipher Spec" message, the issue was resolved.