hadoop-yarnulimitcgroupsworkload

Hard Limits and Soft limits in Linux


I am trying to understand how these limits work. But I am not able to figure it out, why soft limit is needed in the first place when hard limit is present. Most of the sites I have gone through talk about what they are, but none mentioned about how do they work.

In context of processes, Can anyone explain it how do these work?

Thank you.


Solution

  • Let us consider a user abc whose number of processes are set as

    abc     soft    nproc   1024
    abc     hard    nproc   20000
    

    When this user logs in, the user has the effective soft limit of processes applied i.e., abc can run a maximum of 1024 processes. When this limit is used up, the user will not be able to run any more processes unless the soft limit is increased further.

    At this point, the user can optionally increase (but should not be greater than the hard limit) the process limit.
    If the user tries to increase it to 21000 (which is above the hard limit of 20000)

    ulimit -S -u 21000
    -bash: ulimit: max user processes: cannot modify limit: Operation not permitted
    

    But setting it to any value lesser or equal to the hard limit is accepted. Increasing hard limit can be done only by root.

    Can't we just use the hard limits?

    Yes, we can.

    abc     soft    nproc   20000
    abc     hard    nproc   20000
    

    Setting soft limit equal to hard limit would make the hard limit be the default limit for abc user. But it is not recommended as it could lead to a single user exhausting most of the processes (if the limit set is relatively high) that can be totally spawned (pid_max).