unity-game-enginehololensphotonwindows-mixed-reality

Realtime Sharing of Pointer/Cursor of Each Player in Game Room Using PUN2/MRTK2 on Hololens 2

Is there a way to share each player's raycast pointer/cursor with the other players in realtime? I'm using PUN2 and MRTK2 and Hololens 2.

Approaches I've tried so far:

  1. I tried a naïve approach of modifying the MRTK provided ShellHandRayPointer to contain a PhotonView and then using that on the MixedRealityToolkit object within the scene, but that seems to have no effect.
  2. I've tried creating a cursor prefab based off of MRTK's CursorFocus in which I add a Surface Magnetism component (tracking the hands) and then instantiate this prefab for each player in PUN2's OnJoinedRoom callback. After the instantiation call, I add the object to a non-rendered layer for the player with the goal of hiding it for the local player but allowing it show up for other players. This seems to hide the object as expected when only one player is in the room, but when a second player joins, the first player then sees a cursor show up that tracks with their hand movement, which seems unexpected to me (of note is that I'm using one Hololens 2 headset with a computer acting as the second player). Though perhaps this "crossed" behavior is due to the Surface Magnetism component?

Thanks!

Step-by-step images of how I modified the ShellHandRayPointer with a PhotonView and then reattached to the MRTK system:

scene: scene

MRTK system: MRTK system part 1

MRTK system part 2: reference to cloned ShellHandRayPointer: MRTK system part 2: reference to cloned ShellHandRayPointer

my cloned ShellHandRayPointer part 1: my cloned ShellHandRayPointer part 1

PhotonView components expanded on the cloned ShellHandRayPointer: PhotonView components expanded on the cloned ShellHandRayPointer

Solution

  • Regarding how to share objects in Photon in real time, as far as I know, the real-time shared objects in Photon need to be instantiated by PhotonNetwork.Instantiate, but ShellHandRayPointer in MRTK is instantiated by Input system.

    You can customize a copy of ShellHandRayPointer, map the position and rotation of ShellHandRayPointer to the copy at runtime and share this copy in Photon in real time.

    The position and rotation of ShellHandRayPointer can be obtained in MixedRealityToolkit.InputSystem.DetectedInputSources. Or you can use Unity's methods to get this Game Object directly.

    For the cursor, you can use the same method above to create a copy of the cursor and map its position and rotation.