swiftuirealitykitperspectivecameravisionos

How to display a RealityKit Perspective Camera View in a visionOS SwiftUI 2D window?


I am developing an immersive visionOS app based on RealityKit and SwiftUI.
This app has ModelEntities that have a PerspectiveCamera entity as child.
I want to display the camera view in a 2D window in visionOS.

I am creating the camera, and add it to the entity with

let cameraEntity = PerspectiveCamera()
cameraEntity.camera.far = 10000
cameraEntity.camera.fieldOfViewInDegrees = 60
cameraEntity.camera.near = 0.01

entity.addChild(cameraEntity)

There are some post on SO, like this one, that apparently show such camera views as part of an arView. However my app is not AR. The immersive view is programmatically generated.

My question is:
How can I show the camera view in a SwiftUI 2D window?


Solution

  • I contacted Apple, and they confirmed that this is a bug in visionOS 1.0 … 2.0 beta: One can currently define a PerspectiveCamera, but cannot use its output. They suggested to use a RealityRenderer instead. Here is the suggested code that I did not check until now.

    @Observable
    @MainActor
    final class OffscreenRenderModel {
            
        private let renderer: RealityRenderer
        
        private let colorTexture: MTLTexture
            
        init(scene: Entity) throws {
            renderer = try RealityRenderer()
            
            renderer.entities.append(scene)
            
            let camera = PerspectiveCamera()
            renderer.activeCamera = camera
            renderer.entities.append(camera)
            
            let textureDesc = MTLTextureDescriptor()
            textureDesc.pixelFormat = .rgba8Unorm
            textureDesc.width = 512
            textureDesc.height = 512
            textureDesc.usage = [.renderTarget, .shaderRead]
            
            let device = MTLCreateSystemDefaultDevice()!
            colorTexture = device.makeTexture(descriptor: textureDesc)!
        }
        
        func render() throws {
            
            let cameraOutputDesc = RealityRenderer.CameraOutput.Descriptor.singleProjection(colorTexture: colorTexture)
            
            let cameraOutput = try RealityRenderer.CameraOutput(cameraOutputDesc)
    
            try renderer.updateAndRender(deltaTime: 0.1, cameraOutput: cameraOutput, onComplete: { renderer in
                
                guard let colorTexture = cameraOutput.colorTextures.first else { fatalError() }
                
                // The colorTexture holds the rendered scene.
            })
        }
    }