In Vulkan, it's suggested to limit yourself to binding 4 descriptor sets at once because that will get you by on the lowest tier of hardware, even though the higher end devices will allow much more than 4. But since the concept of a descriptor set is just Vulkan smoke and mirrors, what is actually creating this limitation for the number of sets bound?
If I understand correctly, Direct3D descriptor tables would be the closest thing to a Vulkan descriptor set, but there is no similar limitation for the number of descriptor tables added to the root signature, so how is hardware determining a limit for Vulkan descriptor sets?
I may be completely wrong about some things in my assumptions also, so feel free to correct anything I'm misunderstanding.
But since the concept of a descriptor set is just Vulkan smoke and mirrors ...
This seems like an assumption that may be the case on some GPUs, but it's certainly not everywhere.
Descriptor sets are literally "tables of descriptors" that hardware needs to be able to address and index in to, so they are very much a hardware concept and not smoke and mirrors.
... what is actually creating this limitation for the number of sets bound?
Vulkan runs on a wide range of hardware, including mobile GPUs, and the first Vulkan implementations run on hardware designs that predate Vulkan. The exact limiting factor is not always public, although there are hardware limits and/or performance drop offs that make a larger number unviable on older hardware.
Arm documents some of the limiting factors on their older GPUs here, but they are not the only vendor with limits here:
If any of the four source descriptor sets have changed, then the driver rebuilds the internal table for a draw call. The first draw call, after a descriptor changes, has a higher CPU overhead than following draw calls that reuse the same descriptor set.
Most (all?) recent GPU hardware support a larger number (e.g. 7 has been common as the device minimum on most mobile hardware for a few years), so the practical real-world minimum is increasing as older hardware drops off.