c++opencvwinapidirectxtexture2d

Win32 API convert ID3D11Texture2D to cv::Mat

So I am trying to convert ID3D11Texture2D to OpenCV cv::Mat, but my code distorts the image for some reason.

D3D11_TEXTURE2D_DESC capturedTextureDesc;
texture->GetDesc(&capturedTextureDesc);
capturedTextureDesc.Usage = D3D11_USAGE_STAGING;
capturedTextureDesc.BindFlags = 0;
capturedTextureDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
capturedTextureDesc.MiscFlags = 0;

winrt::com_ptr<ID3D11Texture2D> userTexture = nullptr;
winrt::check_hresult(d3dDevice->CreateTexture2D(&capturedTextureDesc, NULL, userTexture.put()));
d3dContext->CopyResource(userTexture.get(), texture.get());
D3D11_MAPPED_SUBRESOURCE resource;
winrt::check_hresult(d3dContext->Map(userTexture.get(), NULL, D3D11_MAP_READ, 0, &resource));

//https://arxiv.org/pdf/1702.02514
int scWidth = capturedTextureDesc.Width;
int scHeight = capturedTextureDesc.Height;
int CAMERA_CHANNELS = 4;
unsigned char* buffer = new unsigned char[(scWidth * scHeight * CAMERA_CHANNELS)];
unsigned char* mappedData = reinterpret_cast<unsigned char*>(resource.pData);
memcpy(buffer, mappedData, (scWidth * scHeight * CAMERA_CHANNELS));
d3dContext->Unmap(userTexture.get(), 0);
IplImage* frame = cvCreateImageHeader(cvSize(scWidth, scHeight),
    IPL_DEPTH_8U, CAMERA_CHANNELS);
frame->imageData = (char*)buffer;
cvSetData(frame, buffer, frame->widthStep);
// cv::Mat mt = cv::Mat(frame, true); // constructor dissapered
//https://stackoverflow.com/questions/15925084/conversion-from-iplimage-to-cvmat
cv::Mat mt = cv::cvarrToMat(frame);
return mt;

Here is the window captured with the cv::Mat output:

enter image description here

They have the same size , but the content is different.

Solution

  • The code doesn't seem to be referencing the RowPitch value in the D3D11_MAPPED_SUBRESOURCE. RowPitch is not always width*pixelSize - each row can be buffered such that the row width is a multiple of 16 or some other value.

    Instead of this:

    memcpy(buffer, mappedData, (scWidth * scHeight * CAMERA_CHANNELS));

    Copy row by row:

    for (int h = 0; h < scHeight; h++) {
    unsigned char* srcRow = mappedData + (resource.RowPitch * h);
    unsigned char* dstRow = buffer + scWidth * CAMERA_CHANNELS * h;
    memcpy(dstRow, srcRow, scWidth*CAMERA_CHANNELS);
}

    Or.... pass the rowPitch to OpenCV.

    buffer = new unsigned char[resource.RowPitch * scHeight * CAMERA_CHANNELS];
memcpy(buffer, mappedData, resource.RowPitch * scHeight * CAMERA_CHANNELS);

...

cvSetData(frame, buffer, resource.RowPitch);