I need to extract all textures from an application at the driver level, similar to GFXR and RenderDocs, but I am unclear on the main methodology. In Vulkan, texture data is transferred to GPU memory by the application rather than the driver. Based on my understanding and testing, the typical process appears to be:
Create a staging buffer.
The application copies pixel data to the staging buffer:
void* hostData;
vkMapMemory(device, stagingBufferMemory, 0, imageSize, 0, &data); // Map GPU memory to host and copy data
memcpy(data, srcPixels, static_cast<size_t>(imageSize));
vkUnmapMemory(device, stagingBufferMemory);
Create a VkImage (another VkDeviceMemory).
Transition the VkImage layout to a GPU optimal layout.
Copy the staging buffer (linear data) to the VkImage using:
vkCmdCopyBufferToImage(commandBuffer, stagingBuffer, VkImage, etc...)
One approach to obtaining the texture data is to dump it before calling vkUnmapMemory. However, there is no guarantee the application will call vkUnmapMemory as it may keep it mapped indefinitely. Another potential method is to intercept vkCmdCopyBufferToImage, but if the application does not use a staging buffer and has pixels in a GPU-friendly format, this approach would also fail, resulting in swizzled pixel data that is difficult to recompose.
Could anyone familiar with GFXR or RenderDoc explain their general methodology for this process?
Thank you for your time.
Best regards,
All of RenderDoc's interaction with Vulkan-accessing programs is done through Vulkan layers. A layer is a piece of code that can sit between the implementation and the application. Layers are often used during debugging, but implicit layers can register themselves in a way that they're always hooked in.
Exactly how this works is OS-specific, but the hooks are always there.
Once you have a layer, you can listen in on all interactions between the host and the Vulkan implementation. This gives you all the information you need to be able to extract whatever information you like.
But that doesn't mean that it's easy.
As you have seen, applications don't have to unmap memory immediately after copying to it. Indeed, this would cause poor performance, since mapping is not a trivial operation and keeping memory mapped has almost no downsides. So most real applications that allocate mappable memory will leave it mapped. And if that memory is allocated as coherent, they don't even have to call a function to make values they've written visible to Vulkan.
But that doesn't mean that there's no way to know when changes have happened.
Because Vulkan operations execute asynchronously relative to host operations, if the host writes data to a piece of memory, there must be some synchronization between the writes to that memory and any reads from it. vkQueueSubmit can handle this implicitly, so long as the host writes are themselves synchronized with the host call to vkQueueSubmit. But they don't have to be; the batch of work could have a wait on an event that gets set by the host. That event would have a memory barrier to make the bytes written by the host available to the GPU.
The point being that, while it is doable, it is also a very complicated process. RenderDoc is open-source, so one way to learn all of the details is to look at its implementation.