Capturing ARSCNView with virtual objects - iOS

I have an ARSCNView with virtual objects drawn. The virtual objects are drawn on the user's face. The session has the following configuration:

let configuration = ARFaceTrackingConfiguration()
configuration.worldAlignment = .gravityAndHeading

sceneView.session.run(configuration)

This ARSCNView is part of a video call. If we send back the pixel buffer like below,

 public func session(_ session: ARSession, didUpdate frame: ARFrame) {
    videoSource.sendBuffer(frame.capturedImage, timestamp: frame.timestamp)
 }

The virtual objects are not shown to my caller.

One of the things I tried is, to not rely on ARSessionDelegate's callback but use DispatchSourceTimer to send events.

func startCaptureView() {  
  // Timer with 0.1 second interval
  timer.schedule(deadline: .now(), repeating: .milliseconds(100))
  timer.setEventHandler { [weak self] in
    // Turn sceneView data into UIImage
    guard let sceneImage: CGImage = self?.sceneView.snapshot().cgImage else {
      return
    }

    self?.videoSourceQueue.async { [weak self] in
       if let buffer: CVPixelBuffer = ImageProcessor.pixelBuffer(forImage: sceneImage) {
             self?.videoSource.sendBuffer(buffer, timestamp: Double(mach_absolute_time()))
        }
    }
  }

  timer.resume()
}

The caller receives the data slowly with a choppy video experience and the images are not of the right size.

Any suggestions on how to send data about the virtual object along with the captured frame?

Reference: https://medium.com/agora-io/augmented-reality-video-conference-6845c001aec0

Solution

The reason the Virtual objects are not appearing is because ARKit provides only the raw image, so frame.capturedImage is the image captured by the camera, without any of the SceneKit rendering. To pass the rendered video you will need to implement an offscreen SCNRenderer and pass the pixel buffer to Agora's SDK.

I would recommend you check out the Open Source framework AgoraARKit. I wrote the framework and it implements Agora.io Video SDK and ARVideoKit as dependancies. ARVideoKit is a popular library that implements an off-screen renderer and provides the rendered pixel buffer.

The library implements WorldTracking by default. If you want to extend the ARBroadcaster class to implement faceTracking you could use this code:

import ARKit

class FaceBroadcaster : ARBroadcaster {

    // placements dictionary
    var faceNodes: [UUID:SCNNode] = [:]           // Dictionary of faces

    override func viewDidLoad() {
        super.viewDidLoad() 
    }

    override func setARConfiguration() {
        print("setARConfiguration")        // Configure ARKit Session
        let configuration = ARFaceTrackingConfiguration()
        configuration.isLightEstimationEnabled = true
        // run the config to start the ARSession
        self.sceneView.session.run(configuration)
        self.arvkRenderer?.prepare(configuration)
    }

    // anchor detection
    override func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
        super.renderer(renderer, didAdd: node, for: anchor)
        guard let sceneView = renderer as? ARSCNView, anchor is ARFaceAnchor else { return }
        /*
         Write depth but not color and render before other objects.
         This causes the geometry to occlude other SceneKit content
         while showing the camera view beneath, creating the illusion
         that real-world faces are obscuring virtual 3D objects.
         */
        let faceGeometry = ARSCNFaceGeometry(device: sceneView.device!)!
        faceGeometry.firstMaterial!.colorBufferWriteMask = []
        let occlusionNode = SCNNode(geometry: faceGeometry)
        occlusionNode.renderingOrder = -1

        let contentNode = SCNNode()
        contentNode.addChildNode(occlusionNode)
        node.addChildNode(contentNode)
        faceNodes[anchor.identifier] = node
    }
}