augmented-realityarkitrealitykitvisionos

How to place a mesh entity on a tracked image in ARKit for VisionOS


Im attempting to place a mesh sphere on a tracked image using the ImageTrackingProvider from ARKit for VisionOS. The following code is the image tracking view model that I have set up to initialize an ARKit session and update anchors for any detected images. When I run it on device, the console prints out the tracking origin results as expected, so I can confirm the tracking is working. However I cannot get the sphere to appear. Is there another more simple way to use the printed origin transform as the anchor point for a 3d object? My current method is based on apples example here

import RealityKit
import SwiftUI
import ARKit
import RealityKitContent

@MainActor var imageAnchors: [UUID: ImageAnchor] = [:]
@MainActor var entityMap: [UUID: Entity] = [:]

@MainActor
class ImageTrackingViewModel: ObservableObject {
  private let session = ARKitSession()
  private var rootEntity = Entity()
  private let imageInfo = ImageTrackingProvider(
      referenceImages: ReferenceImage.loadReferenceImages(inGroupNamed: "albumCovers")
  )

  func start() async {
      do {
          if ImageTrackingProvider.isSupported {
              print("ARKitSession starting.")
              Task {
                  try await session.run([imageInfo])
                  for await update in imageInfo.anchorUpdates {
                      updateImage(update.anchor, rootEntity: rootEntity)
                  }
              }
          }
      }
  }

  func updateImage(_ anchor: ImageAnchor, rootEntity: Entity) {
      
          // Add a new entity to represent this image.
          let entity = ModelEntity(mesh: .generateSphere(radius: 0.1), materials: [SimpleMaterial(color: .systemBlue, isMetallic: false)])
          entityMap[anchor.id] = entity
          rootEntity.addChild(entity)
      

      if anchor.isTracked {
          print("Anchor Transform:")
          print(anchor.originFromAnchorTransform)
          entityMap[anchor.id]?.transform = Transform(matrix: anchor.originFromAnchorTransform)
      }
  }
}

I have tried setting up an entityMap array and attempted to generate a new entity on the tracked image as suggested by the apple example linked above, but it does not result in a visible sphere mesh placed on the generated anchor position


Solution

  • Turns out I was managing the root entity incorrectly. I solved it by adding the following in the RealityView configuration:

    var body: some View {
        RealityView { content in
            content.add(model.setupContentEntity())
        }
    }
    

    This calls a new function in the view model:

    func setupContentEntity() -> Entity {
        return contentEntity
    }
    

    This allowed me to ensure the root entity (which I renamed here to contentEntity) is created when the session begins, so I can add child objects to it. My thanks to reddit user Time_Concert_1751 for their help solving this.