I understand what these fields are and how to use them, but I'm wondering this:
How does the Java compiler actually handle special fields like this in its compilation step? Does it really look for variables just based on their name?
That seems very non-Java like... Other than serialVersionUID and serialPersistentFields, are there others?
Unfortunately, yes, these members are magically discovered by name, reflectively at runtime. (Actually, its worse; whether they are static or not, public or private, etc, fits into the lookup.) You are right, this isn't a good practice, and not very Java-like, but this is a legacy we are stuck with.
For the full list, see the serialization specification: https://docs.oracle.com/en/java/javase/11/docs/specs/serialization/index.html