I have a Mango Pi Mq Quad, which uses the following overlay: https://github.com/readysloth/mangopi-buildroot-overlay. I have successfully built with the help of this overlay. However, now it is interesting to me -- how did the buildroot choose what dts to use? The board support "package" does not mention anything related.. Thanks for hints/full answers!
P. S. dts and dtb are different things. Basically my question is how does buildroot select dts to compile so that we have dtb as a result.
In this setup, there are actually 3 device tree files to consider: the device tree used by ARM Trusted Firmware (TF-A), the device tree used by U-Boot, and the device tree used by the kernel.
Both TF-A and U-Boot normally have the device tree reference in the board configuration included in the source itself. For U-Boot, that's in the selected defconfig file, so in this case board/mangopi/configs/u-boot.config, which sets CONFIG_OF_LIST. As noted in the other answer, that is then overridden with the DEVICE_TREE=... make option but there's actually no reason to do that.
Note that Buildroot also allows to provide a U-Boot DTS from the board directory, by setting BR2_TARGET_UBOOT_CUSTOM_DTS_PATH.
For the kernel, you normally use either BR2_LINUX_KERNEL_INTREE_DTS_NAME or BR2_LINUX_KERNEL_CUSTOM_DTS_PATH to specify the device tree. In this case, nothing of the sort is set, so I guess the kernel uses the device tree from U-Boot.