Remove AD Membership From user A if user B is not in the group

I am very new to Powershell scripting, even newer to powershell scripting for AD. I am trying to basically replicate memberOf from one AD user to another I have gotten a copy script to work, but in the cases like department transfers, I would also like to remove any access the user will no longer need. I have tried the script I have included below. It runs (as in does not throw an error), but does not make any changes.

$Source = Get-ADUser *ShortName* -prop MemberOf
$Destination = Get-ADUser *ShortName* -prop MemberOf
$Source.MemberOf | Where{($Destination.MemberOf -contains $_) -and ($Source.MemberOf -notcontains $_)} | Remove-ADGroupMember $Destination

I have seen some documentation on the Compare-object command but do not understand it well enough to see a method of implementationn using that command. Any Help or hints would be appreciated

Check the code and let me know if it makes sense to you, I added comments to give you hints on the logic.

Of course, this is not the only way of doing this, but im using classic coding so it's easier for you to read and understand.

# Get the membership of sourceUser
$sourceUser = Get-ADUser sourceUser -Properties MemberOf

# Get the membership of destUser
$destUser = Get-ADUser destUser -Properties MemberOf

# You want to compare if the the membership of destUser is different from
# the membership of sourceUser, hence we use destUser's membership to loop

foreach($group in $destUser.MemberOf)
{
    # If this group is not in the membership of sourceUser ->
    # remove thisUser from thisGroup
    
    # Option 1:
    if(-not $sourceUser.MemberOf.Contains($group)) # This is harder to read but more efficient
    {
        Remove-ADGroupMember -Identity $group -Members $destUser.distinguishedName
    }

    # Option 2:
    if($group -notin $sourceUser.MemberOf) # This is easier to read but less efficient
    {
        Remove-ADGroupMember -Identity $group -Members $destUser.distinguishedName
    }
}

Edit: Adding the oneliners for reference.

# This is how the oneliner you were attempting should look
# I honestly don't like this for people new to Powershell

# Option 1:
$destUser.MemberOf|where{-not $sourceUser.MemberOf.Contains($_)}|Remove-ADGroupMember -Members $destUser.distinguishedName

# Option 2:
$destUser.MemberOf|where{$_ -notin $sourceUser.MemberOf}|Remove-ADGroupMember -Members $destUser.distinguishedName

# Option 3:
$destUser.MemberOf|where{-not $sourceUser.MemberOf.Contains($_)}|foreach{
    Remove-ADGroupMember -Identity $_ -Members $destUser.distinguishedName
}

# Option 4:
$destUser.MemberOf|where{$_ -notin $sourceUser.MemberOf}|foreach{
    Remove-ADGroupMember -Identity $_ -Members $destUser.distinguishedName
}