Let's say I've created a directory using module-starter, and written several additional modules and tests since.
make test would then run all tests in t/ on all modules in lib/, however make dist will only pack files mentioned in MANIFEST into tar.gz.
So I got burnt recently by running make test && make dist and still getting a broken package.
My question is: am I missing something, or this can be reported as a minor bug in MakeMaker? (Which Makefile.PL seems to rely upon).
You can use make disttest which will create a distribution directory from the MANIFEST (equivalent to make distdir) and run make test in that. This guarantees you're running against the same files as will be shipped.
I also rebuild my MANIFEST as part of making a release, which requires keeping your MANIFEST.SKIP up to date.
All in all, my basic release script is:
perl Makefile.PL
make manifest
make disttest
make dist