As far as I know, trap is something that happens when special events occur. In case of a system call, program executes trap instruction and jumps to kernel mode. Then, trap handler jumps to desired handler (e.g. fork, exec, open).
When the execution of fork, exec, open, etc. is finished, the OS calls return-from-trap instruction and makes the program go back to user mode.
But what exactly is a trap handler? (Also, if you may, what is a trap table?)
The trap handler is the code that will run when the trap is triggered. In your example, the OS will have installed a handler (i.e. told the CPU a memory address of code to run when the trap happens), and the handler will execute the system call. It is NOT the program that jumps to kernel mode. The program is interrupted immediately after it triggers the trap. Execution resumes with the trap handler.
This way, the three layers (program that runs in protected mode, operating system that runs in privileged mode and CPU/hardware that enforces that currently executing code cannot break out of protected mode) can hand off control between each-other.
Also note that a) modern CPU have dedicated instructions for system calls -- a mechanism that is more efficient than trapping but conceptually works the same and that b) there are other traps/interrupts used for different purposes as well -- they provide the basic mechanism for stopping sequential program execution and do "something else" in reaction to some kind of event.