I am hosting a computing service on Ubuntu 12.04 and I need a method to prevent users from forkbombing. I am currently using setrlimit(RLIMIT_NPROC) in Linux. However, this actually sets a global limit on the number of processes for a given UID. Because my service has many parallel jobs, all running with the same UID, this means that one process can DOS the entire service.
Instead I am looking for a way to set a restriction on e.g. the recursive number of child procs for the current process. I have been told to look into linux cgroups / libcgroup but I honestly don't really understand how this could solve my problem. Any tips?
The internet says that the only way to prevent (read: mitigate) fork bombs is by setting RLIMIT_NPROC. Probably because a user id is the only practical place to attach a "processes forked so far" call to.
Suggestion: Have your service run under a number of uids, and have each invocation of the service pick one. Then the ability of a rogue client to DOS via fork bombing is limited proportionately by the number of UIDs used.
You sould probably need some sort of SUID helper program to do this, unless the service's top-level process runs as root (generally considered a bad idea).