I have tried all the register_callback defined in the list of plugin.def. Now I need to determine which plugin to use at which stage of compilation. The following are my questions:
1) May I know what are the source codes related to generating the RTL tree 2) Is it possible to have my plugin intercept between the stages of GIMPLE or RTL? Thanks. Thanks
You might consider coding your GCC extension with MELT which has some few documentation regarding your question. The MELT documentation page has several links to internal and external resources, and has some tutorials on how to use MELT to extend GCC. So it is also a roadmap (for several weeks of reading).
GCC run hundreds of "optimization passes". Most of them are transforming some form of Gimple into some other form of Gimple, and very probably you also want to do that (I probably don't recommend working at the RTL level). You should look at David Malcom's table of passes.
The question "where to insert my pass" is difficult to answer, and depends so much on why (and what for) do you want to customize GCC (intuitively, favor Gimple optimizations over RTL ones). Running some relevant example source code with -fdump-tree-all could help you understanding what is happening (caveat the numbering of dump files is meaningless).
Whatever approach you use in extending GCC (e.g. your GCC plugins written in C++, your GCC extensions coded for the GCC Python plugins, your GCC extensions coded in MELT), it is difficult because you need to understand a lot about GCC internal behavior. You'll probably need several weeks of work, even for a very simple thing.
FWIW, register allocation is very complex inside GCC, and has been rewritten several times. You'll need years to understand its details (and I certainly don't understand anything about the register allocator). Perhaps you just want to add some explicit reg vars in your pass.... (probably before gimplification, or on some high-level gimple).
You might consider your needs as writing some new backend for GCC (probably, several months, or perhaps a year, of work if you are a GCC newbie), leveraging on the existing register allocator. You'll better then patch your own GCC instead of just adding plugins. Then be sure to interact with the GCC community, e.g. describe much more your project on gcc@gcc.gnu.org and what you have tried so far. Publish your GCC fork (at least on github) while you are working on it (and budget several months of full time work for that, and probably more than a year).
Coding a competitive register allocator for GCC (designed to work for many target processors!) is nearly the work of a life time (and certainly needs many years of full-time work).