Is native code of browsers considered to be part of "Layout engine (Rendering engine)" or it is separate from that? In another words, let's say, if we take Mozilla Firefox, does the creation of DOM object is a process of GECKO or not?
The DOM is created by the DOM parser as the HTML is being processed. This step is completely independent of the layout. When the DOM is complete, the layout process (always implemented as native code for performance reasons) will add information to the existing DOM tree (like the position of every element, the colors it used, etc).
So the parts of Firefox which read/parse the HTML, and which render it, are separate (they are located in their own subtrees in the source code and they end). Both are implemented natively.