I have a MSI'd application I am installing, ignoring the %programfiles% environment variable. From a procmon trace, it seems to be using the value of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\ProgramFilesDir instead.
Why might it be doing this (I mean is it common practice to use this value, or is this the only value that is exposed to MSI builders)?
Why does this other registry value exist, when HKCU\Environment\ProgramFiles & HKLM\System\CurrentControlSet\Control\Session Manager\Environment\ProgramFiles are available for use?
The Windows Installer exposes the ProgramFilesFolder and ProgramFiles64Folder properties. The behavior is documented but the implementation is not. You should only rely on the properties and no other technique.
Officially, you need a 64bit MSI using the ProgramFiles64Folder property to go to C:\Program Files\ on a 64bit machine. Even if you use a custom action to explicitly set your directory to this path in a 32bit MSI, the installer will intercept the path and redirect it to what it thinks is the intended directory. This is for application compatibility purposes.
There is an undocumented way to subvert this behavior. Use a custom action to obtain the intended folder and convert it to a short path (C:\Progra~1). The installer does not detect this as invalid and will allow it through. Use this at your own risk.