When I began programming (some 10+ years ago), three things amazed me:
Back then, I accepted all of them as facts of life. I was able to make my own special-purpose programs, but "programs that made my programs work", code editors and form editors were made by the Gods and there was no way I could mess with them.
Then I went to university, and took a course on formal language processing. After learning formal grammars, parsers, abstract syntax trees, etc.; all the magic about compilers, interpreters and code editors was soon gone. Compilers and interpreters could be written in sane and simple ways, and the only non-sane thing a syntax highlighting code editor could require were Windows API hacks.
However, to this day, form editors remain a mystery to me. Either I lack the technical knowledge required to make a form designer, or I have such knowledge, but cannot find a way to use it to implement a form designer.
Using Visual C++ and the MFC, I would like to implement a form designer inspired by the best form designer ever:
In particular, I would like to imitate its two features that I like the most:
The form being designed is inside a container. Thus, an arbitrarily large form may be designed without wasting too much screen real estate, by simply resizing the container to an appropriate size.
The "Align to Grid" option makes designing professional-looking user interfaces a lot less frustrating. In fact, I would go as far as saying creating professional-looking user interfaces using Visual Basic's form designer is actually easy, fun and enjoyable. Even for left-brained programmers like me.
So, I have the following questions:
How do I make a form designer, in which the form being designed is inside a container? Is the form being designed an actual window contained inside another window? Or is it just a mockup "manually" painted by the form designer?
Do the Windows API and/or the MFC contain functions, classes, whatever that make it easy to create "selectable" items (surrounded by little white or blue boxes when they are selected, resizable when they are "grabbed" by one of these "edges")?
How do I implement the "Align to Grid" functionality?
Both the answers here are good, but left out what I consider to be the really interesting bits (including a couple that you didn't ask directly, but you might find of interest anyhow), so here's my 2c:
Ideally, you just go ahead and create a regular instance of the control. You want something that looks like a button? Create a real button. The tricky thing is stopping it from behaving like a button: you want clicks to activate it for moving, not actually 'click' it.
One way of dealing with this - assuming that the controls in question are 'HWND-based' (eg. the standard windows set of button, edit, static, listbox, treeview, etc.) - is to create the control, and then subclass it - ie. override the wndproc with SetWindowLongPtr(GWLP_WNDPROC, ...), so that the designer code can intercept mouse and keyboard input and use it to initiate a move, for example, instead of having the mouse input go through to the actual button code, which would instead interpret it as a 'click' event.
An alternative approach to subclassing is to place an invisible window above the button to capture the input. Same idea of intercepting input, just different implementation.
The above applies to both managed (VB.Net, C#) and unmanaged (C/C++) controls; they're both essentially stock windows HWNDs; the managed versions just have a managed wrapper code handing off to the underlying unmanaged control.
The old (pre-managed code) ActiveX controls, as used in pre-.Net VB, were a whole different ball game. There's a fairly complex relationship between an ActiveX container and the ActiveX controls within it, with many COM interfaces handling things like negotiation of properties, events, painting, and so on. (There's event a set of interfaces that allows an ActiveX control to receive input and draw itself without having its own HWND.) One benefit you get from this complexity, however, is that ActiveX controls have an explicit 'design mode'; so a control knows to respond appropriately in that case and can cooperate with the whole procedure.
So basically the controls are just regular controls. So you'd expect the form itself to be a regular form? - Almost. As far as I know, its just another HWND-based window, that's a child of the designer (so it gets clipped, and can be scrolled within it); but I think the designer is doing a bit of 'cheating' here, because usually Windows only draws frames like - with titlebar and min/max buttons that for actual top-level windows. I don't know offhand the exact technique that they're using here, but some options could include: painting it manually to mimic the Windows look; using the Windows "theme" APIs, which allow you to access the graphic elements used for the bits and pieces of titlebars and paint them wherever you want to; or, perhaps less likely, setting the window up as a "MDI Child window" - this is one exception case where windows will draw a frame around a nested window.
Simplest approach here is for the designer to create eight small square title-bar-less popup windows that sit above all the other elements - which initiate the appropriate resize code when they are clicked. As the user clicks from control to control, just move the drag handle windows to the currently active control. (Note that in all the above, Windows itself is figuring out who's been clicked, you never have to actually compare mouse coords against element rectangle coordinates and work it out yourself.)
For plain windows system controls that are used by unmanaged C/C++, it's relatively easy: there's a well-known text-based file format - .rc - that describes the controls and locations. Have the designer spit out that (and likely a resource.h file also) and you're done: any C/C++ project can pick up those files and compile them in. Managed code (C#, VB.Net) has a somewhat more complex scheme, but it's still the same basic idea: write out a description in the style that the managed tools expect, and they'll happily compile it and use it.
(ActiveX controls are - you've guessed it - a whole 'nother story. There isn't a standard format that I'm aware of, so the form editor and runtime that consumes the data would be closely tied together - eg. the form editor from pre-.Net VB6 produces forms that only VB can use. - I think. It's been some time ago...)
As for recreating the form: if you have a .rc file, it gets compiled into a dialog resource, Windows has built in support to recreate those. Likewise, the managed code support libraries know how to recreate a form from its particular format. Both basically parse the description, and for each item, create elements of the appropriate classes, and set the appropriate style, text, and other properties as specified. It's not doing anything you can't do yourself, its just helper utility code.
For a collection of HWNDs in any container, whether in 'test' mode or actually running in the real app, and regardless of whether you let Windows or Winforms handle the form creation or whether you created each HWNDs yourself, you can add tabbing support by calling IsDialogMessage in your message loop: see the MSDN page remarks section for details. (While WinForms could do this, I think it actually does its own focus handling, so that it can have tab order independent from visual stacking Z-Order.)
Make friends with the Spy++ app (part of the SDK, installs with Visual Studio). If you're going to do anything with HWNDs, managed or unmanaged, it's a real good idea to know how to use this tool: you can point it at any piece of UI on Windows, and see how it's constructed out of a tree of different types of HWNDs. Point it at the VB designer and see what's really happening for yourself. (Click the 'binoculars' icon on the toolbar, then drag the crosshairs the the window you're interested in.)
Also take a look at the resource files that the designer spits out. Everything that you can tweak or move or edit in the forms designer corresponds to some item somewhere in one of those resource files. Make a copy of them, tweak some settings, and then file-compare the two sets, and see what's changed. Try changing some things in the files by hand (I think they're nearly all text), reload, and see if the designer picked up your changes.
Also, it's far easier if you limit the number of controls in the palette to a small finite set, at least at first. If you want to allow any control at all to be dragged in - eg. a 3rd party one, or one you've used in another project; you typically first need some way for that control to be 'registered' so that the designer knows that it's available in the first place. And you may also need some way to discover what icon it uses in the toolbar, what its name is, what properties it supports - and so on.
Have fun exploring!