linux-kernelubuntu-16.04centos7cgroupscpuset

How to make cpuset.cpu_exclusive function of cpuset work correctly

I'm trying to use the kernel's cpuset to isolate my process. To obtain this, I follow the instructions(2.1 Basic Usage) from kernel doc cpusets, however, it didn't work in my environment.

I have tried in both my centos7 server and my ubuntu16.04 work pc, but neither did work.

  1. centos kernel version:
[root@node ~]# uname -r
3.10.0-327.el7.x86_64
  1. ubuntu kernel version:
4.15.0-46-generic

What I have tried is as follows.

root@Latitude:/sys/fs/cgroup/cpuset# pwd
/sys/fs/cgroup/cpuset
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.cpus
0-3
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.mems
0
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.cpu_exclusive
1
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.mem_exclusive
1
root@Latitude:/sys/fs/cgroup/cpuset# find . -name cpuset.cpu_excl
usive | xargs cat
0
0
0
0
0
1
root@Latitude:/sys/fs/cgroup/cpuset# mkdir my_cpuset
root@Latitude:/sys/fs/cgroup/cpuset# echo 1 > my_cpuset/cpuset.cpus
root@Latitude:/sys/fs/cgroup/cpuset# echo 0 > my_cpuset/cpuset.mems
root@Latitude:/sys/fs/cgroup/cpuset# echo 1 > my_cpuset/cpuset.cpu_exclusive
bash: echo: write error: Invalid argument
root@Latitude:/sys/fs/cgroup/cpuset#

It just printed the error bash: echo: write error: Invalid argument. Google it, however, I can't get the correct answers.

As I pasted above, before my operation, I confirmed that the cpuset root path have enabled the cpu_exclusive function and all the cpus are not been excluded by other sub-cpuset.

By using ps -o pid,psr,comm -p $PID, I can confirm that the cpus can be assigned to some process if I don't care cpu_exclusive. But I have also proved that if cpu_exclusive is not set, the same cpus can also be assigned to another processes.

I don't know if it is because some pre-setting are missed.

What I expected is "using cpuset to obtain exclusive use of cpus". Can anyboy give any clues?

Thanks very much.

Solution

  • i believe it is a mis-understanding of cpu_exclusive flag, as i did. Here is the doc https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt, quoting:

    If a cpuset is cpu or mem exclusive, no other cpuset, other than
a direct ancestor or descendant, may share any of the same CPUs or
Memory Nodes.

    so one possible reason you have bash: echo: write error: Invalid argument, is that you have some other cgroup cpuset enabled, and it conflicts with your operations of echo 1 > my_cpuset/cpuset.cpu_exclusive

    please run find . -name cpuset.cpus | xargs cat to list all your cgroup's target cpus.

    assume you have 12 cpus, if you want to set cpu_exclusive of my_cpuset, you need to carefully modify all the other cgroups to use cpus, eg. 0-7, then set cpus of my_cpuset to be 8-11. After all these cpus configurations , you can set cpu_exclusive to be 1.

    But still, other process can still use cpu 8-11. Only the tasks that belongs to the other cgroups will not use cpu 8-11

    for me, i had some docker container running, which prevents me from setting my cpuset cpu_exclusive

    with kernel doc, i do not think it is possible to use cpus exclusively by cgroup itself. One approach (i know this approach is running on production) is that we isolate cpus, and manage the cpu affinity/cpuset by ourselves