dockerdocker-composedocker-swarmmacvlanvip

docker swarm macvlan network issues


The Goal


Run a single haproxy container in the swarm with a static IP on my LAN that will be recreated if a node fails. In essence, it would be akin to a VIP pointing to haproxy but without requiring an external load balancer outside of the swarm.


What works - local macvlan

Example: Creating the macvlan networks on two nodes, running the container on node1, stopping the container, then running the container on node2. The haproxy container is created with the same static IP and is accessible on the LAN.

What doesn't work - swarm macvlan

Example: Setting the macvlan network scope to swarm, then deploying the stack. Container is not visible to the network.


Files used in example

docker-compose.yml:

version: "3.2"
networks:
  vlan0:
    external: true
services:
  haproxy:
    image: haproxy:2.3.2-alpine
    container_name: haproxy
    volumes:
      - ./data:/usr/local/etc/haproxy:ro
    environment:
      - TZ=America/Los_Angeles
    restart: always
    networks:
      vlan0:
        ipv4_address: 192.168.0.201

localnet.sh (script to stop stack / remove network / recreate network as local / run container local):

#!/bin/bash
docker service rm haproxy_haproxy
docker-compose down
docker network rm vlan0
docker network create -o parent=eth0 --subnet 192.168.0.0/24 --gateway 192.168.0.1 --driver macvlan --scope local vlan0
docker-compose up

swarmnet.sh (script to remove container and network / recreate network as swarm / run as swarm stack):

#!/bin/bash
docker service rm haproxy_haproxy
docker-compose down
docker network rm vlan0
docker network create -o parent=eth0 --subnet 192.168.0.0/24 --gateway 192.168.0.1 --driver macvlan --scope swarm vlan0
docker stack deploy -c docker-compose.yml haproxy

Is it possible to run a single container with a static macvlan IP in swarm mode?

Any help is appreciated.

Edit: I have been able to get macvlan addresses working, but docker swarm does not obey the ipv4_address field to static the container. I understand the reason for this (replicas, etc. not going on the same IP) but in this scenario, it will not happen due to it being a single container. I've found this issue discussed here: https://forums.docker.com/t/docker-swarm-1-13-static-ips-for-containers/28060/


Solution

  • I've figured out how to bind a single static ip address (with a major caveat below).

    Method

    To do this, create a config only network on each host with a single number ip range (32 bit mask). I've created a script here to auto-generate the commands.

        #!/bin/bash 
        echo 
        echo VIP swarm command generator
        echo 
        defint=$(ip route get 8.8.8.8 | head -n1 | awk '{print $5}')
        hostip=$(ip addr show dev $defint | grep "inet" | awk 'NR==1{print $2}' | cut -d'/' -f 1)
        hostsub=$(ip route | grep "src $hostip" | awk '{print $1}')
        hostgate=$(ip route show | grep default | awk '{print $3}')
        read -p "Subnet (default "$hostsub"): " sub
        sub=${sub:=$hostsub}
        read -p 'Gateway (default '$hostgate'): ' gate
        gate=${gate:=$hostgate}
        read -p 'Vip address: ' ip
        last=`echo $ip | cut -d . -f 4`
        begin=`echo $ip | cut -d"." -f1-3`
        read -p 'Ethernet interface (default '$defint'): ' eth
        eth=${eth:=$defint}
        read -p 'Docker Network Name: (default vip-'$last'): ' name
        name=${name:="vip-"$last}
        echo -e "Build the network on each node, \033[0;33mmake sure the physical parent interface (parent=) is set properly on each node if different)\033[0m:"
        echo -e "        \033[44mdocker network create --config-only --subnet $sub --gateway $gate --ip-range $ip/32 -o parent=$eth $name\033[0m"
        echo
        echo "Then create the swarm network from any manager node:"
        echo -e "        \033[44mdocker network create -d macvlan --scope swarm --config-from $name swarm-$name\033[0m"
    

    Example output:

    
    VIP swarm command generator
    
    Subnet (default 192.168.0.0/24):
    Gateway (default 192.168.0.1):
    Vip address: 192.168.0.201
    Ethernet interface (default eth0):
    Docker Network Name: (default vip-201):
    Build the network on each node, make sure the physical parent interface (parent=) is set properly on each node if different):
            docker network create --config-only --subnet 192.168.0.0/24 --gateway 192.168.0.1 --ip-range 192.168.0.201/32 -o parent=eth0 vip-201
    
    Then create the swarm network from any manager node:
            docker network create -d macvlan --scope swarm --config-from vip-201 swarm-vip-201
    

    So, on each node, I build the network config

    docker network create --config-only --subnet 192.168.0.0/24 --gateway 192.168.0.1 --ip-range 192.168.0.201/32 -o parent=eth0 vip-201

    Then I build the swarm interface

    docker network create -d macvlan --scope swarm --config-from vip-201 swarm-vip-201

    And in the applicable docker-compose.yaml file, add the appropriate lines:

    networks:
      swarm-vip-201:
        external: true
    services:
      haproxy:
        ...
        networks:
          swarm-vip-201:
    

    Results

    My container will now always be available at that single static IP address. If the node fails, it will restart on another node with the same IP, in essence creating a high-availability VIP for the swarm.

    Caveat (Swarm macvlan bug?)

    Docker swarm will throw errors trying to bind multiple macvlans to the same parent interface, like below:

    "network NETNAME is already using parent interface eth0"
    

    Described here:

    https://github.com/moby/libnetwork/issues/2384

    https://github.com/moby/libnetwork/issues/1743

    Conclusion

    It is promising that it is actually possible to create a static macvlan IP in the swarm, but until the bug is fixed with binding multiple networks to a single parent interface, it is very limited.

    If anyone knows how to reliably fix the problem above, or has a better method, please post.