iosswiftuiscenekitrealitykitrealityview

RealityView iOS 18 How to Move Entity Accurately using Touch Drag Gesture


With iOS 18 RealityView is so convenient replacing SceneKit creating 3d But when it comes to moving 3d Entities using 2d touch and drag I face a problem, I could not make an accurate move which like attaching the 3d Entity to my finger while drag it on the screen, it was much easier in SceneKit using project and unproject and Also in ARKit/ARView using raycast also it is very successful in VisionOS cause it is 3D Touch and I can use 3d location in gesture value but in 2d it simply not working

here is the closest I get to and it do move the 3d entity but not responding well to the location of the finger.

import SwiftUI
import RealityKit

struct ContentView: View {
  @State var box = Entity()

  @State var lastPanTouchPosition: CGPoint = .zero

  var body: some View {
    RealityView{ content in
        
        let item = ModelEntity(mesh: .generateBox(size: .init(0.25, 0.25, 0.25)), materials: [SimpleMaterial(color: .blue, isMetallic: true)])
        box.addChild(item)
        content.add(box)
    }
    .gesture(dragThis)
  }

  var dragThis: some Gesture {
    DragGesture()
        .onChanged { value in
            print(value.translation)
            print(value.location)
            
            let dragRatio: Float = 0.0001

            box.position.x += Float(value.translation.width) * dragRatio

            box.position.y -= Float(value.translation.height) * dragRatio
        }
        .onEnded{_ in 
            box.position = [0, 0, 0]
        }
    }
}

enter image description here


Solution

  • The beauty of moving AR/VR models in 3D space of iOS RealityView using a standard 2D gesture is that you don't need to turn on a CollisionComponent, which means that heavy models are much more responsive. The disadvantage is also obvious: you need to find a balance between the speed of moving your finger on the screen and the speed of model's movement, which may be located at a considerable distance along the Z axis from the camera. This code works well (tested on 12.9" iPad):

    import SwiftUI
    import RealityKit
    
    struct ContentView : View {
        @State var position: UnitPoint = .zero
        let box = ModelEntity(mesh: .generateBox(size: 0.25))
        let ratio: Float = 0.002
        let group = Entity()
        
        var drag: some Gesture {
            DragGesture(minimumDistance: 15, coordinateSpace: .global)
                .onChanged {
                    group.position.x = Float($0.translation.width + position.x) * ratio
                    group.position.y = Float($0.translation.height + position.y) * -ratio
                }
                .onEnded {
                    position.x += $0.translation.width
                    position.y += $0.translation.height
                }
        }
        var body: some View {
            RealityView { rvc in
                group.addChild(box)
                rvc.add(group)
            }
            .gesture(drag)
        }
    }