I was wondering if there is a way to define tests (something like unit test) for installers, specifically installers made in WiX and InstallShield.
You'll need to define exactly what functionality you think needs testing. It seems pointless to verify that an MSI file will install files correctly because that's just testing Microsoft functionality. That's like testing that Explorer will copy a bunch of files from A to B. Another example: MSI will install services correctly if you specify that data properly, but how do you know that what you've got in the MSI file is correct in the first place? Alsom keep in mind that an MSI install is transactional - it will all work or all fail, so if it finishes then it worked, and if it fails it all rolls back. There is no such thing as a partial install.
The issue is that almost all of the functionality of the install belongs to Microsoft, so all that's left is being sure that your custom action code detects errors and fails the install if appropriate, and that everything you've been told (where the files should install, their dependencies, COM info, prerequisites, etc) is correct.
I've never come across anyone that tests that what's in the MSI file was installed on the system because that's just testing that Microsoft software did its job. So if you are told to install a file in a certain place, then you could verify that the file was installed there, but if the app fails because it's the wrong place then you just wasted time! So IMO the only test that matters is a complete test of the installed product.