I saw a few posts regarding this topic, but nothing really helped my cause, and so restating my problem here. 

I would like to build my own toolchain, U-boot and the mainline Linux kernel to be ported on a particular processor. Since I currently am in possession of a Raspberry Pi 4B, I decided to use it to test my binaries stage-by-stage on it.

I performed the following so far:

1. Built a 64-bit toolchain using Crosstool-ng (v1.24.0) - Got \x-tools\aarch64-rpi3-linux-gnu
2. Cloned U-boot and built v2020.07 version thereby generating u-boot.bin + other files(Built using rpi_4_defconfig and rpi_arm64_defconfig)
3. Cloned Linux Kernel (v5.4.65 and v5.8.9), added bcm2711_defconfig in arch/arm64/configs from the Raspberry Pi repositories and built the Linux Kernel - generating vmlinux, Image, Image.gz, System.map, bcm2711-rpi-4-b.dtb + other files
4. Created two partitions boot (FAT32), rootfs (Ext4) on my 32-GB SD card
5. Copied u-boot.bin, Kernel images, start4.elf, fixup4.dat into the boot partition
6. Have this in config.txt//config.txt

Uncomment below to use your custom named kernel

kernel=u-boot.bin

Serial console output!

enable_uart=1 7. Tried connecting RPi to the monitor with HDMI cable

What I see:  The device seems to switch on, I see some quick green LED blinks, and then it is the red LED only that keeps steady. The monitor gets signal indicated by the fact that it brightens, but the screen is blank. I expected a command prompt with UART > or something, but nothing came. 

Questions:

1. Can someone comment what is missing on my side?
2. What are the absolute minimum stuff to get an own U-Boot, own Linux Kernel and a root FS running on a Raspberry Pi 4B? (with regards to .elf, fix*.dat, etc., + the settings on config.txt, cmdline.txt? )

I hope this answers the U-Boot side of the question:

The following worked for me to boot a 1 GiB Raspberry 4 using a 64bit Raspbian kernel:

Build u-boot v2020.10-rc5 (on Debian Bullseye with GCC 10).

make rpi_arm64_defconfig
CROSS_COMPILE=aarch64-linux-gnu- make -j4

Copy u-boot.bin to first partition of the RPi4 SD-card.

Edit config.txt and add these lines:

enable_uart=1
arm_64bit=1
kernel=u-boot.bin

On the serial console of U-Boot I entered the following commands. You could put them into a boot.scr file using mkimage.

U-Boot> setenv bootargs coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_headphones=1 smsc95xx.macaddr=DC:A6:32:05:53:B6 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1 root=PARTUUID=7f586f18-02 rootfstype=ext4 elevator=deadline rootwait
U-Boot> load mmc 0:1 $fdt_addr_r bcm2711-rpi-4-b.dtb                 
47471 bytes read in 26 ms (1.7 MiB/s)
U-Boot> load mmc 0:1 $kernel_addr_r kernel8.img
15483392 bytes read in 823 ms (17.9 MiB/s)
U-Boot> booti $kernel_addr_r - $fdt_addr_r

You will have to adjust the PARTUUID according to match your root partition and the MAC address.