javascriptpython-3.xwebrtchtml5-videoaiortc

Unable to fully establish ICE connection between two Python aiortc scripts


I'm working on putting together a prototype which streams video from one Python script to another using WebRTC via aiortc.

I'm riffing on the webcam example from the Python aiortc repository (using the --play-from flag to stream from a video file instead of the webcam) and attempting to port the client.js script to Python. I'm mostly there but the ICE connection never (successfully) completes and goes from IN_PROGRESS to FAILED after ~30s. I receive and can subscribe to the track in the client script but never start receiving the stream from the webcam.py script.

The webcam.py script remains unchanged and this is my client.py script:

import asyncio
import json
import requests
import time

from aiortc import (
    MediaStreamTrack,
    RTCConfiguration,
    RTCIceServer,
    RTCPeerConnection,
    RTCSessionDescription,
)
from aiortc.contrib.media import MediaPlayer, MediaRelay
from loguru import logger

relay = MediaRelay()


class VideoTransformTrack(MediaStreamTrack):
    """
    A video stream track that transforms frames from an another track.
    """

    kind = "video"

    def __init__(self, track):
        if track.kind != "video":
            raise Exception("Unsupported kind: %s" % (track.kind))

        super().__init__()  # don't forget this!
        logger.info("track kind: %s" % track.kind)
        self.track = track

    async def recv(self):

        logger.info("Attempt to receive frame ...")

        frame = await self.track.recv()

        logger.info("Received frame: %s" % (frame))


async def run():
    # config = RTCConfiguration(
    #     iceServers=[
    #         RTCIceServer(
    #             urls=[
    #                 "stun:stun.l.google.com:19302",
    #                 "stun:stun1.l.google.com:19302",
    #                 "stun:stun2.l.google.com:19302",
    #                 "stun:stun3.l.google.com:19302",
    #                 "stun:stun4.l.google.com:19302",
    #             ]
    #         ),
    #     ]
    # )

    pc = RTCPeerConnection()

    pc.addTransceiver("video", direction="recvonly")

    @pc.on("track")
    def on_track(track):
        """
        Track has been received from client.
        """

        logger.info("Track %s received" % track)

        pc.addTrack(
            VideoTransformTrack(
                relay.subscribe(track),
            )
        )

        @track.on("ended")
        async def on_ended():
            logger.info("Track ended: %s" % track)

    offer = await pc.createOffer()
    await pc.setLocalDescription(offer)

    while True:
        logger.info("icegatheringstate: %s" % pc.iceGatheringState)
        if pc.iceGatheringState == "complete":
            break

    offer2 = pc.localDescription

    data = {
        "sdp": offer2.sdp,
        "type": offer2.type,
    }

    response = requests.post(
        "http://localhost:8080/offer",
        headers={"Content-Type": "application/json"},
        json=data,
    )

    response_data = response.json()

    await pc.setRemoteDescription(
        RTCSessionDescription(sdp=response_data["sdp"], type=response_data["type"])
    )

    while True:
        time.sleep(2)
        logger.info("pc.iceGatheringState: %s" % pc.iceGatheringState)
        logger.info("pc.connectionState : %s" % pc.connectionState)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(run())

The relevant portions of the (fully functional) client.js script from the example repository:

var config = {
    sdpSemantics: 'unified-plan'
};

pc = new RTCPeerConnection(config);

negotiate();

function negotiate() {
    pc.addTransceiver('video', {direction: 'recvonly'});
    pc.addTransceiver('audio', {direction: 'recvonly'});
    return pc.createOffer().then(function(offer) {
        return pc.setLocalDescription(offer);
    }).then(function() {
        // wait for ICE gathering to complete
        return new Promise(function(resolve) {
            if (pc.iceGatheringState === 'complete') {
                resolve();
            } else {
                function checkState() {
                    if (pc.iceGatheringState === 'complete') {
                        pc.removeEventListener('icegatheringstatechange', checkState);
                        resolve();
                    }
                }
                pc.addEventListener('icegatheringstatechange', checkState);
            }
        });
    }).then(function() {
        var offer = pc.localDescription;
        return fetch('/offer', {
            body: JSON.stringify({
                sdp: offer.sdp,
                type: offer.type,
            }),
            headers: {
                'Content-Type': 'application/json'
            },
            method: 'POST'
        });
    }).then(function(response) {
        return response.json();
    }).then(function(answer) {
        return pc.setRemoteDescription(answer);
    }).catch(function(e) {
        alert(e);
    });
}

enter image description here

Am I missing something obvious in my translation or is there some additional step I need to take to satisfy the aiortc side of the equation? Also, how do people debug these sorts of problems? I haven't used it much but, thus far, WebRTC strikes me as being a bit of a black box.

Note: I have tried explicitly configuring STUN servers on both sides but that doesn't seem to have any effect and my understanding is that the aiortc uses Google STUN servers by default.

Environment:

UPDATE: I may have made some progress debugging this by comparing the offer generated by Chrome using client.js and by Python using client.py. There seems to be some slight differences in the contents of the sdp: client.js makes reference to my ISP-provided IP and client.py references 0.0.0.0. I've tried using explicit IPs in all cases and that seems to break things. This may be a red herring but it seems plausible that misconfiguration here could be the root problem.


Solution

  • time.sleep in your asyncio run routine blocks the main thread and freezes the event loop.

    You should use await asyncio.sleep(2) instead.

    Test

    A quick test with your example code: your client.py code then prints:

    2022-08-07 14:39:42.291 | INFO | __main__:run:105 - pc.iceGatheringState: complete
    2022-08-07 14:39:42.291 | INFO | __main__:run:106 - pc.connectionState : connected
    

    Likewise, webcam.py then prints to the debug console:

    INFO:aioice.ice:connection(0) ICE completed
    Connection state is connected