entityrealitykitcoordinate-transformationvisionos

Don't understand coordinate conversion of visionOS RealityKit


I want to convert a world coordinate to the local coordinate of a RealityKit entity. According to the docs, this should be possible using convert(position:from:), called on the entity. Since it did not work as expected, I wrote a little test project. I used Apple's template for a visionOS immersive app, and modified it slightly. This is my immersive view:

struct ImmersiveView: View {
    var body: some View {
        RealityView { content in
            var defaultMaterial = UnlitMaterial()
            defaultMaterial.color.tint = .brown
            let mesh = MeshResource.generateBox(size: 1)
            let boxEntity = ModelEntity(mesh: mesh, materials: [defaultMaterial])
            let anchorEntity = AnchorEntity(world: SIMD3<Float>(0, 0, -4))
            anchorEntity.addChild(boxEntity)
            content.add(anchorEntity)
            
            let coordinate = SIMD3<Float>(0.1, 0.2, 0.3)
            let coordinateFrom = boxEntity.convert(position: coordinate, from: nil)
            let coordinateTo = boxEntity.convert(position: coordinate, to: nil)
            print("coordinateFrom: \(coordinateFrom)")
            print("coordinateTo: \(coordinateTo)")
        }
    }
}

It contains a box visible in front of the main window. The box is not at the origin (z = -4).

Unexpectedly, coordinateFrom as well as coordinateTo are equal to coordinate. I expected that coordinateFrom is different, because the box is not at the origin.
What am I missing?


Solution

  • A friend of mine found a solution:
    Apparently, a view has to be rendered, before the coordinate conversion functions work.
    Here is the extended code that demonstrates the conversion:

    struct ImmersiveView: View {
        @State private var coordinateFrom = SIMD3<Float>(0, 0, 0)
        @State private var coordinateTo = SIMD3<Float>(0, 0, 0)
        @State private var anchorPosition = SIMD3<Float>(0, 0.0, -4)
    
        var body: some View {
          ZStack {
              RealityView { content in
                  var defaultMaterial = UnlitMaterial()
                  defaultMaterial.color.tint = .brown
                  let mesh = MeshResource.generateBox(size: 1)
                  let boxEntity = ModelEntity(mesh: mesh, materials: [defaultMaterial])
                  boxEntity.name = "box"
                  let anchorEntity = AnchorEntity(world: anchorPosition)
                  anchorEntity.name = "anchor"
                  anchorEntity.addChild(boxEntity)
                  content.add(anchorEntity)
    
                  print("---- not yet rendered coords")
                  updateCoordinates(boxEntity: boxEntity)
              }
              update: { content in
                  if let anchorEntity = content.entities.first?.findEntity(named: "anchor") as? AnchorEntity {
                      anchorEntity.setPosition(anchorPosition, relativeTo: nil)
                  }
                  if let boxEntity = content.entities.first?.findEntity(named: "box") as? ModelEntity {
                      updateCoordinates(boxEntity: boxEntity)
                  }
              }
              VStack {
                  Text("Coordinate From: \(coordinateString(coordinateFrom))")
                  Text("Coordinate To: \(coordinateString(coordinateTo))")
                  Button(action: {
                      anchorPosition += SIMD3<Float>(Float.random(in: -1..<1), 0, 0)
                  }, label: { Text("Move") })
              }
              .padding()
              .background(Color.black.opacity(0.5))
              .cornerRadius(10)
              .foregroundColor(.white)
          }
        }
        private func updateCoordinates(boxEntity: ModelEntity) {
            let coordinate = SIMD3<Float>(0.1, 0.2, 0.3)
            let coordinateFrom = boxEntity.convert(position: coordinate, from: nil)
            let coordinateTo = boxEntity.convert(position: coordinate, to: nil)
            DispatchQueue.main.async {
                self.coordinateFrom = coordinateFrom
                self.coordinateTo = coordinateTo
            }
            print("coordinateFrom: \(coordinateFrom)")
            print("coordinateTo: \(coordinateTo)")
        }
        private func coordinateString(_ coordinate: SIMD3<Float>) -> String {
            String(format: "(%.2f, %.2f, %.2f)", coordinate.x, coordinate.y, coordinate.z)
        }
    }