iosswiftswiftuiarkitrealitykit

How to display the name of a specific part od model when clicking on that part


In RealityKit, how can I display a popup with the name of a specific part of a model when clicking on that part?

I am developing an iOS AR application, and I have imported a USDZ model of a tooth. I want to implement a feature where, when a user taps on a specific part of the tooth model, a small popup appears with some descriptive text about that part.

The main challenge I am facing is identifying which part of the model has been clicked. How can I achieve this? Could you please guide me through the steps and the interfaces that I need to call to implement this functionality? Thank you very much for your help!

Here are my code now:

import ARKit
import UIKit
import RealityKit

class ARViewController_plane: UIViewController {
    
    let arView = ARView()
    
    var modelEntity: ModelEntity?
    
    override func loadView() {
        super.loadView()
        
        view = arView
        
        // create model
        modelEntity = createModel()
        
        // 放置模型
        placeModel(model: modelEntity!)
        
        // 设置手势
        installestures(on: modelEntity!)
        
        // 配置
        let configuration = ARWorldTrackingConfiguration()
        configuration.planeDetection = .horizontal
        
        // 运行
        arView.session.run(configuration)
        
    }
    
    func createModel() -> ModelEntity {
        let modelEntity = try! ModelEntity.loadModel(named: "tsmile1.usdz")
        return modelEntity
    }
    
    func placeModel(model:ModelEntity) {
        let planeAnchor = AnchorEntity(world: SIMD3(0, 0, 0))//(.plane(.horizontal, classification: .any, minimumBounds: SIMD2<Float>(0.02, 0.02)))
        planeAnchor.addChild(modelEntity!)
        arView.scene.addAnchor(planeAnchor)
    }
    
    func installestures(on object: ModelEntity) {
        // scale
        object.generateCollisionShapes(recursive: true)
        arView.installGestures([.scale], for: object)
        // my rotation
        let rotationGesture = UIPanGestureRecognizer(target: self, action: #selector(handleRotationGesture(_:)))
        arView.addGestureRecognizer(rotationGesture)
    }
    @objc func handleRotationGesture(_ gesture: UIPanGestureRecognizer) {
        let translation = gesture.translation(in: arView)
        // 控制旋转速度
        let rotationAngle = Float(translation.x) * 0.001
        
        // 获取当前模型的变换
        var currentTransform = modelEntity!.transform

        // 创建新的四元数旋转(围绕模型的Z轴旋转)
        let newRotation = simd_quatf(angle: rotationAngle, axis: [0, 0, 1])

        // 将新的旋转与现有的旋转叠加
        currentTransform.rotation = simd_mul(currentTransform.rotation, newRotation)

        // 应用新的变换
        modelEntity!.transform = currentTransform
    }
}




Solution

  • Creating tappable components of the tooth model

    If your tooth model actually consists of several separate parts, and is not a so called combined mesh (where all the visible parts are just textures), then all you need to do is to assign a CollisionComponent to each individual part of the tooth model (enamel, dentine, pulp, etc). The evident drawback of this idea is that collision shapes may overlap each other, which will inevitably lead us to incorrect responses. In order to make a simple test model of a tooth consisting of different parts, I used Maya.

    The magic wand for such a scenario is generateConvex(from:) method.

    Here's a code:

    import SwiftUI
    import RealityKit
    
    struct ContentView : View {
        let arView = ARView(frame: .zero)
    
        var body: some View {
            ARToothView(arView: arView)
                .ignoresSafeArea()
                .onTapGesture {
                    let name = (arView.entity(at: $0) as? ModelEntity)?.name ?? ""
                    print(name)
                }
        }
    }
    

    struct ARToothView : UIViewRepresentable {
        let arView: ARView
        let parts = ["enamel", "pulp", "dentine"]
        let anchor = AnchorEntity()
        // load tooth as Entity to preserve Maya's separate parts
        let tooth = try! Entity.load(named: "tooth")
                            
        func makeUIView(context: Context) -> ARView {
            tooth.scale *= 9
            tooth.position.z = -1.0
            tooth.orientation = .init(angle: .pi/2, axis: [1,0,0])
            anchor.addChild(tooth)
            
            for i in 0 ..< parts.count {
                let model = tooth.findEntity(named: parts[i]) as! ModelEntity
                let mesh = model.model?.mesh
                let shape = ShapeResource.generateConvex(from: mesh!) // convex shape
                let collision = CollisionComponent(shapes: [shape])
                model.components.set(collision)
                arView.installGestures([.translation], for: model)
            }
            print(tooth)  // full hierarchy with CollisionComponents
            
            arView.scene.anchors.append(anchor)
            return arView
        }
        func updateUIView(_ view: ARView, context: Context) { }
    }
    

    Xcode 16.0 console:

    A 3D cross-sectional model of a tooth could look like this.