Exception calling InvokeSet, The directory property cannot be found in the cache

When running the below code I receive the error Exception calling "InvokeSet" with "2" argument(s): "The directory property cannot be found in the cache." Online this error exists because of an issue with a variable being null. After using Write-Host I have come to the conclusion that the $User = [ADSI]$LdapUser line is the one causing an issue.

#Set Remote Control Settings Permissions 
        $LdapUser = "LDAP://" + (Get-ADUser $username).distinguishedName
        Write-Host $LdapUser
        $User = [ADSI]$LdapUser
        Write-Host $User
        $User.InvokeSet("EnableRemoteControl",5)
        $User.InvokeSet("TerminalServicesHomeDrive","U:\")
        $User.setinfo()

The Write-Host command for $LdapUser displays "LDAP://CN=Test Taco,CN=Users,DC=blah,DC=org" which is correct however the Write-Host Command for $User displays System.DirectoryServices.DirectoryEntry Why does the $User variable display the wrong information and how do I fix it?

The issue with my code was that I was using the number 5 for enable remote control as well as using U:\ the correct code uses a 2 as well as no backslash so U:

#Set Remote Control Settings Permissions 
        $LdapUser = "LDAP://" + (Get-ADUser $username).distinguishedName
        $User = [ADSI]$LdapUser
        Write-Host $User
        $User.InvokeSet("EnableRemoteControl",2)
        $User.InvokeSet("TerminalServicesHomeDrive","U:")
        $User.setinfo()

pause

Thank you @Judd Davey for helping me through his answer to realize where I could be going wrong in my answer. I am not quite certain why it gave me such a weird error.