Can't connect to a pod running on different node with k3s and virtualbox

Trying to create a k3s cluster with Vagrant and Virtualbox.

Nodes can connect to each other. DNS resolution works from the pods, pods can connect to the internet and can connect to different pods as long as the pod runs on the same host. Inter-host pod networking is not working.

This is my Vagrantfile to start the whole thing up.

Vagrant.configure("2") do |config|

  config.vm.provider "virtualbox" do |vb|
      vb.memory = "1536"
  end

  config.vm.define "server" do |server|
    server.vm.box = "ubuntu/jammy64"
    server.vm.hostname = "server"
    server.vm.network "forwarded_port", guest: 6443, host: 6443
    server.vm.network :private_network, ip: '172.20.20.10',  virtualbox__intnet: true
    server.vm.provision "shell", 
      inline: "curl -sfL https://get.k3s.io | sh -s - server --write-kubeconfig /vagrant/config  --write-kubeconfig-mode 666 --token=SECRET --tls-san 172.20.20.10 --flannel-iface enp0s8 --flannel-backend=wireguard-native --kube-apiserver-arg advertise-address=172.20.20.10 --node-external-ip=172.20.20.10 --node-ip=172.20.20.10"
  end

  config.vm.define "agent1" do |agent|
    agent.vm.box = "ubuntu/jammy64"
    agent.vm.hostname = "agent1"
    agent.vm.network :private_network, ip: '172.20.20.11',  virtualbox__intnet: true
    agent.vm.provision "shell", 
      inline: "curl -sfL https://get.k3s.io | K3S_URL=https://172.20.20.10:6443 K3S_TOKEN=SECRET sh -s -  --node-ip=172.20.20.11 --node-external-ip=172.20.20.11"
  end

  config.vm.define "agent2" do |agent|
    agent.vm.box = "ubuntu/jammy64"
    agent.vm.hostname = "agent2"
    agent.vm.network :private_network, ip: '172.20.20.12',  virtualbox__intnet: true
    agent.vm.provision "shell", 
      inline: "curl -sfL https://get.k3s.io | K3S_URL=https://172.20.20.10:6443 K3S_TOKEN=SECRET sh -s -  --node-ip=172.20.20.12 --node-external-ip=172.20.20.12"
  end

end

What am I missing here?

Initially, the DNS resolution wasn't working. Using wireguard-native as flannel backend seemed to solve that issue. Also, the private network described here was a host-only adapter so changed it to internal.

You are missing --flannel-external-ip flag for your server node, if you go to docs you will see

Use node external IP addresses as the destination for Flannel traffic, 
instead of internal IPs. Only applies when --node-external-ip is set on a node.

This is exactly your case as your have --node-external-ip specified for each node, as I test it works