Network Client use wrong input devices when using Unity.Netcode with Starter Asset Controller (New input system)

The server is always using Keyboard and Mouse just fine.

The client however always use "xbox controller" instead of Keyboard & Mouse:

The following is the inspector view as a client:

The Start Asset input action are unchanged,

This is what I tried but client is still being assigned to controller:

private void Start()
{
    if (!IsOwner)
    {
        Destroy(GetComponent<PlayerInput>());
    }
}

How could I fix this? Other than hard coding (PlayerInput)map.SwitchCurrentControlScheme("KeyboardAndMouse");?

This issue is fixed by disabling Player Input script.

And only enable it on Network Spawn.

    public override void OnNetworkSpawn()
    {
        base.OnNetworkSpawn();
        if (IsOwner)
        {
            _playerInput = GetComponent<PlayerInput>();
            _playerInput.enabled = true;
        }
    }