Using Three.js, and Rapier physics: Why does this object have a jitter on moving?

I have a simple sphere which has an impulse added to it on a tap, but as it moves it seems to jitter a bit as the mesh doesn't quite keep up with the physical model. Is there a solution to this? The code for the ball is:

import { Sphere } from "@react-three/drei"
import { useFrame } from "@react-three/fiber"
import { RapierRigidBody, RigidBody } from "@react-three/rapier"
import { useRef } from "react"
import { Vector3 } from "three"

export default function Roller({pos}:{pos:Vector3}) {
  const ball = useRef<RapierRigidBody>(null)

  useFrame((state) => {
    if (!ball.current) return
    const t = ball.current.translation()
    let pos:Vector3 = new Vector3(t.x, t.y, t.z)
    state.camera.position.copy(new Vector3(0, 30, -30).add(pos));
    state.camera.lookAt(pos);
  })

  function click() {
    if (!ball.current) return
    ball.current.applyImpulse({x:100, y:0, z:100}, true)
  }

  return <RigidBody ref={ball} position={pos} colliders={"ball"}> 
    <Sphere args={[1]} onClick={click}>
      <meshStandardMaterial color={'red'} />
    </Sphere>
  </RigidBody>
}

However, I'm not sure that's where the issue lies. Here's a shot of the issue.

That depends (probably in your case) on the frame rate. You need to adapt the physics engine to the varying frame rate.

If your frame rate is high and you use a fixed timestep (default 1/60 seconds/60Hz), the physics engine may produce problems in the rendering, leading to jitter for example.

When you use timeStep="vary", the physics engine adapts to each frame.

Try set

<Physics timeStep="vary">
   {/* ... */}
</Physics>

Remove timeStep then you will see what I am talking about