How does OS honor user limits configured in /etc/security/limits.conf since setrlimit is process based?

I noted that in /etc/security/limits.conf, the limits are configured on a per user basis (or per group basis), for example: @faculty hard nproc 50

I assume that it is setrlimit that does the work to set limits, but setrlimit works on a process basis, that is, it only sets resource limits on its calling process, so since it can't set limits on user, how can OS honor resource limits configured in limits.conf?

Another question, If a certain process exceeds its resource quota, will it be killed? If so, by what signal? Thank you.

You can set hard and soft limits, per user/group. To be able to change its own hard limit the process needs to be privileged (root). What happens when process attempts to exceed its soft limit depends on the resourse:

1. data limit - malloc and new would fail
2. open files - creation of file descriptor failes (open, creat, socket, accept, etc.)
3. core - core file would be truncated
4. file size - SIGXFSZ is delivered to offending thread.
5. stack - SIGSEGV delivered to offending thread
6. etc.

Look on setrlimit manpage for more information.

In limits.conf you assign limits per process for a specified user/group. So if you set 10MiB stack limit for user X it means that each and every process executed with user X credentials has 10MB stack limit set. It is not a limit that describes 'a sum of resources' for all processes owned by user X