How to Create a Vulkan VkImage from an Android SurfaceTexture Bound to ExoPlayer?

I'm working on an Android app where I have a SurfaceTexture that's bound to an ExoPlayer for video playback. The texture uses the GL_TEXTURE_EXTERNAL_OES target. I need to create a Vulkan VkImage from this SurfaceTexture so that I can use Vulkan for further rendering.

How can I create a VkImage from the SurfaceTexture that is using GL_TEXTURE_EXTERNAL_OES? Are there specific Vulkan extensions (like VK_ANDROID_EXTERNAL_MEMORY_ANDROID_HARDWARE_BUFFER) that I need to utilize for this process?

To create a Vulkan VkImage from an Android SurfaceTexture using GL_TEXTURE_EXTERNAL_OES, you need to take advantage of Android's AHardwareBuffer and Vulkan extensions designed for interaction between Android and Vulkan. The general approach involves getting an AHardwareBuffer from a SurfaceTexture and then using the Vulkan extension VK_ANDROID_EXTERNAL_MEMORY_ANDROID_HARDWARE_BUFFER to create a VkImage from the buffer.

The instructions are as follows:

1. Get AHardwareBuffer from SurfaceTexture When using SurfaceTexture for animation from GL_TEXTURE_EXTERNAL_OES, the underlying data is usually handled by AHardwareBuffer. You can use the Android NDK to store the AHardwareBuffer associated with SurfaceTexture:

CPP

#include <android/hardware_buffer.h>
#include <android/native_window.h>

// Assume `surfaceTexture` is your SurfaceTexture instance
ANativeWindow* nativeWindow = ANativeWindow_fromSurface(env, 
surfaceTexture);
ANativeWindowBuffer* buffer;
nativeWindow->dequeueBuffer(nativeWindow, &buffer);

AHardwareBuffer* hardwareBuffer = buffer->handle;

2. Use the Vulkan extension VK_ANDROID_EXTERNAL_MEMORY_ANDROID_HARDWARE_BUFFER The VK_ANDROID_EXTERNAL_MEMORY_ANDROID_HARDWARE_BUFFER extension allows you to let Vulkan from AHcm understand that the rendered image needs to be connected externally.

CPP

VkExternalMemoryImageCreateInfo externalMemoryImageCreateInfo = {};
externalMemoryImageCreateInfo.sType =         
VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO;
externalMemoryImageCreateInfo.handleTypes =         
VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID;

Create a VkImageCreateInfo Structure:

You need to set up your VkImageCreateInfo to create a Vulkan image.

CPP

VkImageCreateInfo imageCreateInfo = {};
imageCreateInfo.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO;
imageCreateInfo.imageType = VK_IMAGE_TYPE_2D;
imageCreateInfo.extent.width = ...;  // Set width from your SurfaceTexture
imageCreateInfo.extent.height = ...; // Set height from your     
SurfaceTexture
imageCreateInfo.extent.depth = 1;
imageCreateInfo.mipLevels = 1;
imageCreateInfo.arrayLayers = 1;
imageCreateInfo.format = ...; // Appropriate format matching the     
`SurfaceTexture`
imageCreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
imageCreateInfo.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
imageCreateInfo.usage = VK_IMAGE_USAGE_SAMPLED_BIT |     
VK_IMAGE_USAGE_TRANSFER_DST_BIT;
imageCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
imageCreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
imageCreateInfo.flags = 0;
imageCreateInfo.pNext = &externalMemoryImageCreateInfo;

Create the VkImage:

Use the VkImageCreateInfo to create the VkImage.

CPP

VkImage vkImage;
VkResult result = vkCreateImage(device, &imageCreateInfo, nullptr,     
&vkImage);

Allocate and Bind Memory:

Allocate memory for the image using AHardwareBuffer. The VkImportAndroidHardwareBufferInfoANDROID structure is used to import the AHardwareBuffer.

CPP

VkMemoryRequirements memoryRequirements;
vkGetImageMemoryRequirements(device, vkImage, &memoryRequirements);

VkImportAndroidHardwareBufferInfoANDROID importHardwareBufferInfo = {};
importHardwareBufferInfo.sType =     
VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID;
importHardwareBufferInfo.buffer = hardwareBuffer;

VkMemoryAllocateInfo memoryAllocateInfo = {};
memoryAllocateInfo.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO;
memoryAllocateInfo.allocationSize = memoryRequirements.size;
memoryAllocateInfo.memoryTypeIndex = ...; // Find suitable memory type
memoryAllocateInfo.pNext = &importHardwareBufferInfo;

VkDeviceMemory deviceMemory;
result = vkAllocateMemory(device, &memoryAllocateInfo, nullptr,     
&deviceMemory);

vkBindImageMemory(device, vkImage, deviceMemory, 0);

3. Use the VkImage for Rendering After binding the VkImage to the memory, you can use it like any other Vulkan image. You might want to transition its layout, attach it to a framebuffer, or sample from it in a shader.

Key Extensions and Considerations: VK_ANDROID_EXTERNAL_MEMORY_ANDROID_HARDWARE_BUFFER: This is essential for creating a VkImage from an AHardwareBuffer. AHardwareBuffer: The underlying buffer that backs the SurfaceTexture and can be used for Vulkan interop. Synchronization: Ensure proper synchronization when accessing the buffer across different APIs (OpenGL ES and Vulkan). This approach allows you to create a Vulkan VkImage from the SurfaceTexture and use Vulkan for further rendering operations.