From Prerequisites on the setup property pages, I selected the option to Download prerequisites from the component vendor's web site
There is no Windows Installer 4.5 in the prerequisites to install option..
because every im installing my setup using the sqlexpr_x64_ENU.exe in custom actions to install, run the setup after the extracting sqlserver it pops up the window says
There is a problem with this Windows Installer package. Aprogram run as part of the setup did not finnish as expected. Contact your support personnel or package vendor
im using VS2017 on windows 10
Exclude MSI Engine: You should not include Windows Installer 4.5 with any package these days. This runtime is from back in the day of Windows Vista - we are long since on version 5. Windows Installer should be deployed only via Windows Update as of today - in my opinion. Leave the runtime out of your setup.
No Concurrency: You can not run an MSI setup from within another MSI setup via a custom action. This is due to technical restrictions. There is a mutex set when an MSI runs its actual installation sequence, and triggering another installation sequence from within it will fail. A similar answer on the topic.
Setup.exe Launcher: What you need is to install your pre-requisites via a setup.exe launcher instead. This runs installations in serial, not in parallel. Which version of SQL Server are you installing? I see only a couple of versions available in the Visual Studio Installer Project launcher.
Other tools have features to allow you to install a setup.exe with embedded packages of various kinds in sequence. I have explained a myriad of times how to do this in previous answers. Here are just a few that I found quickly: