swifttwilioscenekitiosurface

Unsupported IOSurface format: 0x26424741 using twilio video in scenekit


I am using twilio to send a video and use that video in a scenekit as a texture. But the problem is it works fine with iPhone X, but it gave this error Unsupported IOSurface format: 0x26424741 on iPhone XR and XS.

this is what I am doing:

Get Video:

func subscribed(to videoTrack: TVIRemoteVideoTrack, publication: TVIRemoteVideoTrackPublication, for participant: TVIRemoteParticipant) {
    print("Participant \(participant.identity) added a video track.")
    let remoteView = TVIVideoView.init(frame: UIWindow().frame,
                                       delegate:self)
    videoTrack.addRenderer(remoteView!)
    delegate.participantAdded(with: remoteView!)
}

delegate:

func participantAdded(with videoView: UIView) {
    sceneView.addVideo(with: videoView)
}

and add video to plane:

func addVideo(with view: UIView){
    videoPlane.geometry?.firstMaterial?.diffuse.contents = view
}

Solution

  • The problem was actually with renderingType of remoteView. For older devices using metal was fine but newer devices it needed openGLES. I dont know why but it was the fix.

    I used this solution to find out the device type.

    Next I determined which renderingType to use

    var renderingType: VideoView.RenderingType {
        get{
            let device = UIDevice()
            switch device.type{
            case .iPhoneXS:
                return .openGLES
            case .iPhoneXR:
                return .openGLES
            case .iPhoneXSMax:
                return .openGLES
            default:
                return .metal
            }
        }
    }
    

    And used it to initialize remoteView

    func didSubscribeToVideoTrack(videoTrack: RemoteVideoTrack, publication: RemoteVideoTrackPublication, participant: RemoteParticipant) {
        print("Participant \(participant.identity) added a video track.")
        let remoteView = VideoView.init(frame: UIWindow().frame,
                                        delegate:self,
                                        renderingType: renderingType)
        videoTrack.addRenderer(remoteView!)
        delegate.participantAddedVideo(for: participant.identity, with: remoteView!)
    }