When calling CtrlP for the first time it cause quite a delay to build the cache. Is there a way to run it in the background so I can continue doing stuff in vim?
You cannot do that, in fact, vim isn't multi-threaded.
You can look over for NeoVim which is multi-threaded.
Alternativly, you could use the Unite plugin and the file_rec/async command :
Note: with large projects this may cause some performance problems. Normally it is recommended to use |unite-source-file_rec/async| source, which requires |vimproc|.