I bundled Qugaa bgpd and nsm binaries into a docker based on Ubuntu 3.13.0-36-generic kernel. Now I run multiple of these dockers on a linux sever. These dockers are instantiated using docker-py.
I used pipework and created an Ethernet interface on each docker and assigned an 172.17.xx.xx kind of address. My BGP configuration is such that all the dockers are completely meshed with respect to BGP. i.e each BGP on a docker is connected to every other BGP running on other dockers.
BGP sessions get established and the BGP routes etc are absolutely fine. Now, when the number of dockers exceeds 30, i could never connect to bgp. "top" command doesn't show much cpu usage, memory is in limits, not much network activity and I also don't expect much processing within BGP process.
when i took tcpdump on the container, this is what it looks like.
9 2014-09-26 18:17:54.997861 0a:60:4a:3b:56:31 ARP 44 Who has 172.17.1.32? Tell 172.17.1.6
when I run 40 dockers, i see 40 such ARP requests as shown above followed by an one ARP reply.
however this continuously happens, and it approximately results in 1600 (40*40) such messages in a short span of time. I believe this is what is not allowing me to connect to a local bgp module using "telnet localhost bgpd" command.
I don't think this is anything specific to either Quagga or BGP. I suspect some thing to do with docker networking. Does anybody got such an issue or any idea how to fix this or what is the root cause for this ?
Finally I could find the root cause behind this and also fixed this. This is due to the combination of number of dockers instances, number of MAC entries that i am creating on each docker and the default ARP cache size specified on my linux server as 1024.
In my case I end up creating MAC entries as (number of dockers * number of ethernet interfaces i am creating through pipework). So the problem occurs when the number of MAC entries on each DOCKER exceeds the default size of 1024.
adding below lines at the end of /etc/sysctl.conf file or modifying as below if it already has these entries in /etc/sysctl.conf solved the issue.
net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 12288
net.ipv4.neigh.default.gc_thresh3 = 16384
After modifying, execute "sysctl -p" command