I know PAX is doing a lot of stuff and that creating the container and copying all those jars is not cheap but are there any general tips to improve performance. I have tests that execute outside the container in a fraction of a second while inside they take much longer. I am using PAX primarily to verify that my manifests are accurate and the bundle would be deployable without any missing dependencies. I have tried Knopflerfish, Equinox, Felix and in general there is little difference they are relatively slow to a barebones containerless run.
As you realised, the underlying container does not make much of a difference.
If you want to have minimal bundles created on the fly, you can try out Pax Tinybundles: if this applies to your case, you can build a set of minimized bundles with only the content you actually need for testing. For example, you can just package your Manifest. I haven't benchmarked it myself for this particular purpose, but it is worth a shot.
As a sidenote, please consider that Pax Exam 2.3 introduced support (see here) for @Before and @After, thus coming to your rescue for more flexible load setup/teardown.