The setup:
My visionOS app shows a programmatically created immersive space with some entities.
It does not show the user's environment, i.e. it does not use an ARView
, only a RealityView
with its RealityViewContent
.
Some of the entities have a primitive camera with a pixel matrix of 10 x 10 pixels.
To compute the image of this camera, I have to casts a ray from each pixel sensor through a hole (the lens) to the immersive space where all entities are located.
The point where this ray hits another entity defines the output of the pixel sensor, e.g. a color.
The problem:
I did not find a possibility within RealityKit
to do this ray casting without an ARView
.
The docu on an Entity
shows that only an ARView
has a Scene
property, and func raycast(from:to:query:mask:relativeTo:)
is a function of Scene
, see the docu.
There are many posts and tutorials how to do raycasting in RealityKit
, e.g. this one, but they are use an ARView
with a Scene
.
My question:
How can I do a ray cast within a RealityViewContent
from a certain coordinate in the immersive space in a given direction, and get the entity hit, and the local coordinate of this hit on this entity?
EDIT 1:
By now, I found this post. Here, an Apple DTS Engineer shows raycasting code for visionOS. However, my entities have as scene
property nil
, so it does not work for me.
EDIT 2:
By now I realized, that one can use ARView
with cameraMode .nonAR
, see here (although the docu is surely wrong: It probably should say, that option .nonAR
does not use the device camera, but a virtual camera).
However, it is no longer possible (Xcode 15.3, Xcode 16.0 beta) to use ARView
in visionOS. One gets a compiler error
'ARView' is unavailable in visionOS
You can retrieve the RealityKit scene in your RealityView using environment, like so:
@Environment(\.realityKitScene) var scene: RealityKit.Scene?
Another way would be to add a subscription on the scene content for the didAddEntity event (https://developer.apple.com/documentation/realitykit/sceneevents/didaddentity) once that fires for your entity the entity.scene property should be set.
Example: GitHub Gist