java.lang.VerifyError: (class: GregorSamsa, method: ...) Illegal target of jump or branch

I'm facing a VerifyError while creating a lot of xml-transformers (javax.xml.transform.Transformer) with fairly complex XSLT document (2000 conditions). Please see example:

    public class XsltVerifyErrorTest {

    private static final int MAX_ITERATIONS_COUNT = 1000000;

    public static void main(String[] args) throws Exception {
        final byte[] xslBytes = Files.readAllBytes(new File(args[0]).toPath());

        for (int i = 0; i < MAX_ITERATIONS_COUNT; i++) {
            System.out.println(String.format("Iteration %d", i));

            final StreamSource xslSource = new StreamSource(new ByteArrayInputStream(xslBytes));
            final Transformer transformer = TransformerFactory.newInstance().newTransformer(xslSource);
        }
    }
}



m49216@ubuntu:~/xslt-verify-error-test$ javac XsltVerifyErrorTest.java
m49216@ubuntu:~/xslt-verify-error-test$ java -showversion XsltVerifyErrorTest XsltVerifyErrorTest.xsl
openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

Iteration 0
Iteration 1
Iteration 2
...
Iteration 79
Iteration 80
Iteration 81
Exception in thread "main" java.lang.VerifyError: (class: GregorSamsa, method: TestTemplate signature: (Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;ILjava/lang/Object;)V) Illegal target of jump or branch
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructor0(Class.java:3075)
        at java.lang.Class.newInstance(Class.java:412)
        at com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl.getTransletInstance(TemplatesImpl.java:455)
        at com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl.newTransformer(TemplatesImpl.java:486)
        at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:762)
        at XsltVerifyErrorTest.main(XsltVerifyErrorTest.java:25)

Full example with test data can be found here.

The issue can be reproduced on many versions of jdk: 8u101, 8u121, 8u152, 8u161, 8u162.

Has anyone run into this problem? Does it look like a jvm bug?

Edit 1: JDK 9 is also affected, but it's much harder to reproduce the issue - it takes ~20 minutes and 660 iterations on my machine.

java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)
Iteration 0
Iteration 1
Iteration 2
...
Iteration 658
Iteration 659
Iteration 660
Exception in thread "main" java.lang.VerifyError: (class: die/verwandlung/GregorSamsa, method: TestTemplate signature: (Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;ILjava/lang/Object;)V) Illegal target of jump or branch
    at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3110)
    at java.base/java.lang.Class.getConstructor0(Class.java:3315)
    at java.base/java.lang.Class.newInstance(Class.java:530)
    at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl.getTransletInstance(TemplatesImpl.java:552)
    at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl.newTransformer(TemplatesImpl.java:583)
    at java.xml/com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:817)
    at XsltVerifyErrorTest.main(XsltVerifyErrorTest.java:19)

Edit 2: Works fine on JDK 10.

Solution

I got a chance to look at the problem deeper and finally solved this puzzle.
This is indeed a bug in JDK's BCEL library.

TransformerFactory generates the bytecode basing on the source XSLT. One of the bytecode generation steps is to remove temporary NOPs and relocate their targeters (branch instructions pointing to them).

Targeters are maintained in a HashSet. And here is the problem. Instruction class does not define hashCode, and its equals method works totally wrong for branches: two branch instructions are considered equal if their targets are the same. Of course, this is not true.

But the problem does not happen as long as branch hashCodes are different. Since these are actually default identity hashCodes, their collision chance is quite small. But after many iterations, given that generated method is very large, it finally happens: two different branch instructions with the same target obtain the same identity hashCodes, and one overwrites another due to collision.

Run the program with -XX:hashCode=2 (this will force degenerated identity hashCodes), and it will crash immediately.

The bug was fixed in BCEL-195 and integrated into JDK 10 in JDK-8163121.

Earlier JDK versions still have this bug, but fortunately you can use the following workaround. Just call Instruction.setComparator sometime at the application startup:

    import com.sun.org.apache.bcel.internal.generic.*;
    ...

    Instruction.setComparator((i1, i2) -> {
        if (i1 instanceof BranchInstruction) {
            return i1 == i2;
        }
        return InstructionComparator.DEFAULT.equals(i1, i2);
    });