Why do I specify ipu4 and ipu4_ex both to use ipu device in docker like below command?
docker run --device=/dev/**ipu4**:/dev/ipu4 --device=/dev/**ipu4_ex**:/dev/ipu4_ex -ti graphcore/tools gc-inventory
The suggested way to launch docker images that require access to Graphcore IPUs is using the gc-docker command line tool that you can read more about here. This command line tool is available in the Poplar SDK and wraps the system installed docker command line so that you don't need to worry about passing in devices manually like you've shown above.
For interested users you can see what gc-docker is calling under the hood by using the --echo arg, and this is where you will see something similar to what you've posted:
docker run --device=/dev/ipu0:/dev/ipu0 --device=/dev/ipu0_ex:/dev/ipu0_ex --device=/dev/ipu0_mailbox:/dev/ipu0_mailbox --device=/dev/ipu0_mem:/dev/ipu0_mem -ti graphcore/tools gc-inventory
This is what the corresponding gc-docker call would look like:
gc-docker --device-id 0 -- -ti graphcore/tools gc-inventory
As you can see, each IPU device has 4 associated user space PCIe devices. This is because each Graphcore IPU device has 4 distinct memory regions (which you can see if you use lspci -v to list Graphcore PCI devices). Each memory region corresponds to a different functional part of the device (you can read more about why devices may want to have multiple distinct memory regions in this Stack Exchange post). These memory regions are the IPU config space, IPU exchange space, ICU mailbox, and the host exchange memory device.
The Graphcore PCIe driver bridges the IPU PCIe device memory regions to the 4 user space character devices that you see in the docker command. This mapping of memory regions into user space is required for applications to access them. If any of these devices aren't accessible from a docker container utilising IPUs then you'll run into issues...hence why it's much easier to use the gc-docker tool rather than remembering all the user space device names!