I wish to add support for CPU hot plugging in my application that has strict affinity rules. Each physical core has exactly one thread pinned to it, but the logic I use for pinning a thread to a CPU is fairly naive and fails if a CPU between 0..N is offline.
I have chosen an approach where I have an array of size MAX_CPU, where each CPU on the system maps to a slot by its identifier. For example, CPU0 -> threads[0], and CPU1 -> threads[1], and so on. The idea is to mirror the system's setup.
for (i = 0; i < N; ++i)
set_affinity(threads[i], i);
However, if an offline CPU is encountered anywhere but the end, it will fail.
To make things worse, when a CPU is taken offline during runtime, the pinned thread's affinity mask is reset without notice.
Ultimately, I hope to support complex setups like:
CPU0 CPU1 CPU2 CPU3
ONLINE OFFLINE ONLINE OFFLINE
How can I incorporate awareness of online and offline CPU:s into my application?
I'm avoiding /proc and /sys as I'm interested in porting to other platforms, specifically various BSD:s. I use x86_64 for now, so the cpuid instruction may be useful.
Turns out sched_getaffinity(2) is awesome and better suited for this case. The cpu_set_t it fills in is not a generalized mask like 0xffffffff... that would imply schedule anywhere, but actually a detailed and up-to-date mask of each online and permitted CPU. Using the CPU_* macros, it's possible to extract how many CPU:s are online and which.
cpu_set_t set;
int i, n;
if (sched_getaffinity(0, sizeof(cpu_set_t), &set))
return;
n = CPU_COUNT(&set);
/* pin a thread to each online and permitted cpu */
for (i = 0; n; ++i)
if (CPU_ISSET(i, &set)) {
/* spawn a thread and pin it to cpu identified by 'i' ... */
--n;
}
When notified of a hot plug event, a call to sched_getaffinity(2) from a non-pinned thread will give us an updated mask.
An added benefit is awareness of utilities like taskset(1).
FreeBSD has a cpuset_getaffinity(2) system call that probably operates in a similar fashion. I don't have access to try it out now.