For a library I am working on, I have created a help file (Help 2 format). It would be very convenient if the other developpers of my team that use the library, could search the help from inside the Delphi IDE.
Is it possible to make Delphi XE2 search in a custom help file when I press F1 in the Delphi IDE? And if so, how should I do that?
Yes, it is, the same way third-party component vendors do. As you say, use the free Microsoft Help 2 compiler, Help 2 Workshop, available through MS.
Once you've written and compiled your help, you incorporate it into the IDE's help system via H2Reg.exe, which you can find in your $(BDSDIR)\Help\Doc folder. Read the comments in h2reg.ini (in that same folder) for details regarding how to do so.