Minimize noise for benchmarking in docker

I am writing a benchmarking framework for compiler-like programs. For benchmarking, I use a docker container (for reproducibility). However, i still measure quite a bit of noise (up to 5%!). My current benchmarking process:

• Disable ASLR
• Put all CPUs in performance mode
• Start Docker Container that does the benchmarking
• Run each program 10 times with perf

Is there a way to prioritize the docker container in the host machine to further reduce noise? Are there any additional things i could do to improve the quality of the results?

Is there a way to prioritize the docker container in the host machine to further reduce noise?

There are two things you can do.

1. You can raise its CPU shares. This tells the kernel how much CPU to allocate to each process in the presence of contention.

   For example, the following option will double the CPU shares available compared to the default.

   docker run --cpu-shares 2048 ...
   

   Note that changing nice will not work. The nice level only affects priority of a process relative to other processes within the same cgroup. If your benchmarking program is the only thing running in that container, nice will have no effect.

2. You can move all processes except your benchmark program and perf off a pair of core, then pin that container to those cores.

   Here's an example. Run this command as root outside of the container. This command will move all processes, including kernel threads, off of a particular set of cores.

   cset shield -c N1,N2 -k on
   

   Replace N1,N2 with the index of the CPU cores you are allocating. (For example, if you wanted to allocate core 0 and 1, you would use 0,1.

   Then, when creating your docker image, you would use the option --cpuset-cpus:

   docker run --cpuset-cpus N1,N2 ...
   

Are there any additional things i could do to improve the quality of the results?

Things you could do that you haven't mentioned:

1. Measure the time within the container rather than outside the container. In other words, docker run image time command will be more accurate than time docker run image command. You want to avoid measuring the startup overhead of the container.
2. Disable hyperthreading. The thread running on an SMT sibling can use a variable amount of cache space, which can add noise to your benchmark. You can either do this in the BIOS, or there is a way suggested here about how to do this without rebooting.
3. Either drop caches or prewarm file cache. You mention that you want to benchmark compiler-like programs. Compilers tend to access a lot of files. Linux caches files after they are first accessed, so the first run you do is likely to be much slower than the rest, which will increase variance. You can adapt to this by either discarding the timing result from the first run, (a good idea for many reasons) or by dropping filesystem caches between each test run.