When I tried to patch assignments to a task it only added on to existing ones. For example a task has assignments as {A, B}.
When I did patch assignments {A, C}, the assignments became {A, B, C} while I was expecting it to be updated to {A, C}. When I did patch assignments {} it didn't change anything while I wanted to remove all assignments.
I have not tried it myself but I am certain, if you patch with A's user Id with the null value to task endpit PATCH /planner/tasks/{id} should do the Magic
{ "A's user Guid Id": null, }
Don't forget the etag