When running on builds in VSO's new host agent pool, I have a powershell script to restore the NuGet packages for each project.json using dnu restore.
This process can easily take more than 2 minutes.
Is it not possible to somehow cache these packages on the agent? Would this only be possible if we had our own custom agent?
Yes, you're right that you need to set up one on-premise TFS build controller and agent to avoid restoring NuGet packages each time when you build projects.
NuGet package restore will install all referenced packages before a project is built, it is possible for you to cache packages on on-premise build agent but not on hosted build agent.