swiftanimationblenderrealitykitskeleton

RealityKit Character Skeleton animation weapons are not animating


I am experiencing an issue with a character animation using RealityKit. I have a file created in Blender that contains the rigged Character, a sword, and a shield. The sword and the shield have bones connected to the character's hands so they can follow the character's animation. When I run the animation in Blender and preview the exported USDZ file on Mac, I can see the sword and shield attached to the hands and the animation is fine. But when I add the USDZ model in RealityKit and play the animation, only the character is animating, the sword and the shield are not moving at all.

This is the code I use to animate the character:

private func loadAnimations() {

    let unifiedAnimations = children[0].availableAnimations.first!.definition
    let animationResource = try! AnimationResource.generate(with: unifiedAnimations)

    self.children[0].playAnimation(
        animationResource.repeat()
    )
}

Here is a link with a Demo Xcode project, the Blender file, and a video: https://www.dropbox.com/scl/fi/ypq2iwxc5f9dwzjggsvin/AppleTest.zip?rlkey=wiag3rg44urhjdh2wal8cdx2u&st=vbpf7x11&dl=0

STEPS TO REPRODUCE I have created a demo project that displays the character on a horizontal surface, and the animation starts playing.

  1. Run the App. The ARView will be set up, and a yellow square will appear in the middle of the screen.
  2. When a horizontal surface is detected the yellow Square will change indicating that the surface is found.
  3. Tap on the screen to load the USDZ model and position it in the yellow square's position.
  4. The Animation will start playing and you can see that the character is animating, but the sword and shield remain still.

Thanks

I have tried to find a way to get skeleton bone realtime position so i can sync with the weapon, but not able to make it.

Also tried to export from blender with different techniques, not worked again.


Solution

  • RealityView

    There's no need in going down the hierarchy. This code can be used to play the .usdz animation in RealityView (this is iOS 18+ version). Please note that in this example the model may be placed in the scene without an anchor.

    import SwiftUI
    import RealityKit
    
    struct ContentView : View {
        var body: some View {
            RealityView { rvc in
                rvc.camera = .virtual
                let knight = try! await Entity(named: "Knight")
                knight.position = [0.0,-0.7,-2.0]
                let animatedKnight = animated(knight)
                rvc.add(animatedKnight)
            }
            .ignoresSafeArea()
            .background(.black)
            .realityViewCameraControls(.dolly)
        }
        
        private func animated(_ knight: Entity) -> Entity {            
            let resource = knight.availableAnimations[0]
            knight.playAnimation(resource.repeat())
            return knight
        }
    }
    

    ARView

    The code below is suitable for enabling .usdz animation in UIKit's ARView. As you can know, the anchor is mandatory here.

    import SwiftUI
    import RealityKit
    
    struct ContentView : View {
        var body: some View {
            ARContainer().ignoresSafeArea()
        }
    }
    

    struct ARContainer : UIViewRepresentable {
        func makeUIView(context: Context) -> ARView {
            let arView = ARView(frame: .zero)
            arView.environment.background = .color(.black)
            let knight = try! Entity.load(named: "Knight")
            let animatedKnight = animated(knight)
            
            let anchor = AnchorEntity(world: [0.0,-0.7,-2.0])
            anchor.addChild(animatedKnight)
            arView.scene.anchors.append(anchor)
            return arView
        }
        func updateUIView(_ view: ARView, context: Context) { }
        
        private func animated(_ knight: Entity) -> Entity {
            let resource = knight.availableAnimations[0]
            knight.playAnimation(resource.repeat())
            return knight
        }
    }
    

    enter image description here

    Loading as Entity vs loading as ModelEntity

    Pay attention that by using Entity.loadModel(named:) instead of Entity.load(named:) you are flattening the .usdz model's hierarchy down to the "root" ModelEntity component only. So now your animation is broken into pieces because it only contains the skeletal animation of the character (and doesn't contain the "shield" and "sword" components at all).

    let knight = try! Entity.loadModel(named: "Knight")
    print(knight)
    

    enter image description here

    let knight = try! Entity.load(named: "Knight")
    print(knight)
    

    In this case, the full .usdz hierarchy of the scene is printed to the Xcode console. It's worth noting that if the sword and shield were an integral part of the character, including skeletal system, then the result of playing the animation using either the load(named:) or loadModel(named:) instance methods would be the same.