I have a custom WiX extension (an extension for WiX itself).
https://github.com/nbelyh/VisioPanelAddinVSTO/tree/master/VisioWixExtension
This one is for building Office (Visio) related installations.
It has a custom table, scheduled actions, exec actions, etc. Lots of stuff.
I would really appreciate some pointers on how to migrate it to v4. A link to some document would be perfect. If there is no such document, would appreciate any other source of information (such as example of migrated extension). What should I look at in the first place, i.e. where to start?
Take a look at WiX's extensions. I'd start with a simpler one, like Firewall: https://github.com/wixtoolset/wix/tree/main/src/ext/Firewall