google-chrome webrtc audio-streaming opus surround

How to fix "m=audio 0 UDP/TLS/RTP/SAVPF 0" issue to enabling surround sound streaming (5.1 and 7.1 channel)

I am creating a streaming application with surround sound audio support, currently stereo is supported and streaming is working. Codec - Opus, Browser - Google Chrome

When I give offer SDP to browser it return me with m=audio 0 UDP/TLS/RTP/SAVPF 0 then I munge this resulting SDP from browser with following changes (also this is what I pass in offer sdp from server to browser).

For 7.1 channel

m=audio 9 UDP/TLS/RTP/SAVPF 111\r\n
a=rtpmap:111 multiopus/48000/8\r\n
a=rtcp-fb:111 transport-cc\r\n
a=fmtp:111 channel_mapping=0,6,1,2,3,4,5,7;num_streams=8;coupled_streams=0;maxaveragebitrate=510000;minptime=10;useinbandfec=1\r\n

For 5.1 channel

m=audio 9 UDP/TLS/RTP/SAVPF 111\r\n
a=rtpmap:111 multiopus/48000/6\r\n
a=rtcp-fb:111 transport-cc\r\n
a=fmtp:111 a=fmtp:111 channel_mapping=0,4,1,2,3,5;num_streams=4;coupled_streams=2;maxaveragebitrate=510000;minptime=10;useinbandfec=1\r\n

But at the end audio streaming doesn't work !!!

Offer Sdp from server

a=group:BUNDLE 0 1 2
m=audio 47998 UDP/TLS/RTP/SAVPF 111
c=IN IP4 0.0.0.0
a=rtcp:47998 IN IP4 0.0.0.0
a=candidate:1 1 udp 2122260223 10.40.102.152 47998 typ host
a=mid:0
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=sendonly
a=msid:second_stream_id audio
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 multiopus/48000/8
a=rtcp-fb:111 transport-cc
a=fmtp:111 channel_mapping=0,6,1,2,3,4,5,7;num_streams=8;coupled_streams=0;maxaveragebitrate=510000;minptime=10;useinbandfec=1
a=ptime:10
a=ssrc:1 cname:odrerir
m=video 47998 UDP/TLS/RTP/SAVPF 96 97 101 102 98
c=IN IP4 0.0.0.0
a=rtcp:47998 IN IP4 0.0.0.0
a=candidate:1 1 udp 2122260223 10.40.102.152 47998 typ host
a=mid:1
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=sendonly
a=msid:stream_id fbc-video-0
a=rtcp-mux
a=rtcp-rsize
...

Processed Sdp from browser after it pass through function setRemoteDescription

a=group:BUNDLE 1 2
m=audio 0 UDP/TLS/RTP/SAVPF 0
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:HSRB
a=ice-pwd:xzh/4T2ENxA4tO4mUMlTN42s
a=ice-options:trickle
a=fingerprint:sha-256 5A:71:54:A8:A4:82:58:AD:1E:73:7B:65:2E:FC:D6:E0:CA:47:76:D1:FA:B9:33:F4:90:FE:77:74:64:C5:08:02
a=setup:active
a=mid:0
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=recvonly
a=rtcp-mux
a=ssrc:3852651172 cname:4sZE5ap0JbPq4Is1
m=video 9 UDP/TLS/RTP/SAVPF 96 97 101 102 98
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Z8bB
a=ice-pwd:hOiAOMEYMZlPVLZYrcvQCIH6
a=ice-options:trickle
a=fingerprint:sha-256 5A:71:54:A8:A4:82:58:AD:1E:73:7B:65:2E:FC:D6:E0:CA:47:76:D1:FA:B9:33:F4:90:FE:77:74:64:C5:08:02
a=setup:active
a=mid:1
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=recvonly
a=rtcp-mux
a=rtcp-rsize
...

Munge Sdp that we send again to browser

a=group:BUNDLE 0 1 2
m=audio 9 UDP/TLS/RTP/SAVPF 111
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Z8bB
a=ice-pwd:hOiAOMEYMZlPVLZYrcvQCIH6
a=ice-options:trickle
a=fingerprint:sha-256 5A:71:54:A8:A4:82:58:AD:1E:73:7B:65:2E:FC:D6:E0:CA:47:76:D1:FA:B9:33:F4:90:FE:77:74:64:C5:08:02
a=setup:active
a=mid:0
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=recvonly
a=rtcp-mux
a=rtpmap:111 multiopus/48000/8
a=rtcp-fb:111 transport-cc
a=fmtp:111 channel_mapping=0,6,1,2,3,4,5,7;num_streams=8;coupled_streams=0;maxaveragebitrate=510000;minptime=10;useinbandfec=1
a=ssrc:3852651172 cname:4sZE5ap0JbPq4Is1
m=video 9 UDP/TLS/RTP/SAVPF 96 97 101 102 98
c=IN IP4 0.0.0.0
b=AS:19440
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Z8bB
a=ice-pwd:hOiAOMEYMZlPVLZYrcvQCIH6
a=ice-options:trickle
a=fingerprint:sha-256 5A:71:54:A8:A4:82:58:AD:1E:73:7B:65:2E:FC:D6:E0:CA:47:76:D1:FA:B9:33:F4:90:FE:77:74:64:C5:08:02
a=setup:active
a=mid:1
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=recvonly
a=rtcp-mux
a=rtcp-rsize
...

I did notice that on setRemoteDesription callback passed sdp has additional ice-username/password

a=ice-ufrag:Z8bB
a=ice-pwd:hOiAOMEYMZlPVLZYrcvQCIH6

which are not the ones I got from server.

Solution

The group id bundle doesn't contain the mid of audio stream after processing from browser. Thus the surround streaming doesn't works.

Processed Sdp from browser
a=group:BUNDLE 1 2        <--- Real issue

## A little explanation :-

Chrome browser is expected to reject the audio stream as multiopus is not supported, and when you go through the chrome_debug logs it states that only supported audio format is opus/48000/2
Thus chrome browser return with port 0 in case of audio(RFC 3264)
But when we again munge the sdp then browser follows the multiopus path which is an extension to libwebrtc and streaming works.

Q. How I arrive at group id solution?

Chrome debug log help me that something with mid 0 is arrived but it is not able to demux those packets
Failed to demux RTP packet: PT=111 SSRC=1 (Reference)
All the streams in a webrtc transport are bundled and to identify each stream we pass on mid and inside group bundle we pass high level information related to what streams we will be sending

Please fill in the missing information, as there are very least resources around surround sound streaming. This may help developers working in the similar domain :)