I have few other applications which I have converted to .NET5 and want to convert .vdproj to .NET5 as well.
But I don't know how it can be converted.
The same application .vdproj works in .NET framework.
I have converted other WPF and WCF to .NET5 and want to convert .vdproj now.
One simple answer: you cannot convert Visual Studio vdproj to .NET 5 because this project template only supports creating installer for .NET Framework projects, not for .NET Core projects.
To create MSI installer for .NET Core projects, there are other alternatives. The most popular and free is using WiX. WiX is also used by .NET Core team to create installer for .NET Core SDK.
For more info, please visit WiX at: https://wixtoolset.org/
WiX supports Visual Studio 2015, 2017, 2019. Besides WiX, you could also use paid tools such as InstallShield and AdvancedInstaller for VS 2017 and VS 2019.
UPDATE 1:
Thanks to comment from LexLi, the latest release of Deployment project extension tool for VS 2019 (version 1.0.0) support .NET Core 3.0 or later project. My initial answer still stand: it reflects my experiences the last time I used Deployment Project in early 2020 around February 2020, at that time it didn't support .NET Core projects at all.
Still, the best option for now is to use WiX or the other paid alternatives such as InstallShield or AdvancedInstaller. Especially we have seen WiX being used by .NET Core team to create MSI installer of .NET Core SDK.