In DX10, it is common practice to compress normal maps by placing the Y value in the green channel, and the X value in the alpha, and apply DXT5 compression (the Z component is then reconstructed in the shader). What is the corresponding format in Vulkan? The closest i can find is VK_FORMAT_BC5_UNORM_BLOCK, which is unsigned RG channels with DXT5. Is this equivalent? If so, would you store it as R: X and G: Y or the other way around (does it matter)? Or are there newer compression formats that do a better job?
DXT5 was renamed to BC3 in DX10. Vulkan's equivalent compressed texture formats follow the DX10+ naming convention. So VK_FORMAT_BC3_UNORM_BLOCK is exactly what you're used to using.
That being said, using DXT5/BC3 for normal maps was outdated a decade ago, even in the DX10 days. The common way to handle that nowadays is to use BC5 to store the X and Y components, generating Z in the shader. You get better quality for the same texture size.
But the even more modern equivalent is to use BC7 with a 2-channel, separate encoding mode. Though this requires more advanced compressors.