I want to experiment with changes to the java library in java.lang, and possibly eventually other packages. This means instead of using rt.jar, I want to be able to separate out at least java.lang and compile to them.
It is not possible to edit source code for classes in java.lang if using rt.jar because they are already found in rt.jar.
I found the repo in mercurial: http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/tip/src/share/classes/
but no instructions on how to build.
I just built jdk by installing a jdk9, then using mercurial:
hg clone http://hg.openjdk.java.net/jdk/jdk/
I can successfully build the java executable and all utilities, but rt.jar is not in this package.
The reason that your Java 9 build is not producing an "rt.jar" file is that Java 9 doesn't have an "rt.jar" or "tools.jar" anymore. As JEP 220 states:
"The class and resource files previously stored in lib/rt.jar, lib/tools.jar, lib/dt.jar, and various other internal jar files will now be stored in a more efficient format in implementation-specific files in the lib directory. The format of these files will not be specified and is subject to change without notice."
So ... basically ... your build of Java 9 is working.
If you want to do experiments with modifying parts of the Java class library, I suggest the following possible approaches: