The setup:
My visionOS app shows a programmatically created immersive space with some entities.
It does not show the user's environment, i.e. it does not use an ARView, only a RealityView with its RealityViewContent.
Some of the entities have a primitive camera with a pixel matrix of 10 x 10 pixels.
To compute the image of this camera, I have to casts a ray from each pixel sensor through a hole (the lens) to the immersive space where all entities are located.
The point where this ray hits another entity defines the output of the pixel sensor, e.g. a color.
The problem:
I did not find a possibility within RealityKit to do this ray casting without an ARView.
The docu on an Entity shows that only an ARView has a Scene property, and func raycast(from:to:query:mask:relativeTo:) is a function of Scene, see the docu.
There are many posts and tutorials how to do raycasting in RealityKit, e.g. this one, but they are use an ARView with a Scene.
My question:
How can I do a ray cast within a RealityViewContent from a certain coordinate in the immersive space in a given direction, and get the entity hit, and the local coordinate of this hit on this entity?
EDIT 1:
By now, I found this post. Here, an Apple DTS Engineer shows raycasting code for visionOS. However, my entities have as scene property nil, so it does not work for me.
EDIT 2:
By now I realized, that one can use ARView with cameraMode .nonAR, see here (although the docu is surely wrong: It probably should say, that option .nonAR does not use the device camera, but a virtual camera).
However, it is no longer possible (Xcode 15.3, Xcode 16.0 beta) to use ARView in visionOS. One gets a compiler error
'ARView' is unavailable in visionOS
You can retrieve the RealityKit scene in your RealityView using environment, like so:
@Environment(\.realityKitScene) var scene: RealityKit.Scene?
Another way would be to add a subscription on the scene content for the didAddEntity event (https://developer.apple.com/documentation/realitykit/sceneevents/didaddentity) once that fires for your entity the entity.scene property should be set.
Example: GitHub Gist