I have recently run into a predicament involving DirectX11 and the calculation of a 3D coordinate. I wish to convert a mouse coordinate to this 3D coordinate. I wish to have it act similar to Maya or Unity when inserting a polygon model (default cube, etc.) or a mesh object into an empty space. What steps and maths do I require to calculate this coordinate?
This is the same math you use in ray tracing to convert a screen coordinate back to a ray in 3D space.
Here is the D3DXMath code from the legacy DirectX SDK sample "Pick10"
const D3DXMATRIX* pmatProj = g_Camera.GetProjMatrix();
POINT ptCursor;
GetCursorPos( &ptCursor );
ScreenToClient( DXUTGetHWND(), &ptCursor );
// Compute the vector of the pick ray in screen space
D3DXVECTOR3 v;
v.x = ( ( ( 2.0f * ptCursor.x ) / pd3dsdBackBuffer->Width ) - 1 ) / pmatProj->_11;
v.y = -( ( ( 2.0f * ptCursor.y ) / pd3dsdBackBuffer->Height ) - 1 ) / pmatProj->_22;
v.z = 1.0f;
// Get the inverse view matrix
const D3DXMATRIX matView = *g_Camera.GetViewMatrix();
const D3DXMATRIX matWorld = *g_Camera.GetWorldMatrix();
D3DXMATRIX mWorldView = matWorld * matView;
D3DXMATRIX m;
D3DXMatrixInverse( &m, NULL, &mWorldView );
// Transform the screen space pick ray into 3D space
vPickRayDir.x = v.x * m._11 + v.y * m._21 + v.z * m._31;
vPickRayDir.y = v.x * m._12 + v.y * m._22 + v.z * m._32;
vPickRayDir.z = v.x * m._13 + v.y * m._23 + v.z * m._33;
vPickRayOrig.x = m._41;
vPickRayOrig.y = m._42;
vPickRayOrig.z = m._43;
I've been meaning to convert it to DirectXMath and repost it to MSDN Code Gallery for some time, but it hasn't made it to the top of the stack yet. The code above assumes left-handed coordinates.
Remember that a 2D screen position plus the transformation matrix only provides 2 of the 3 degrees of freedom, so you have to make some assumption about the depth.
Also search for the terms "picking in 3D".