Scala script doesn't run on Ubuntu

I have a previously working Scala script that when I try to run it on a new PC, the compilation fails.

So I made simple script to test:

#!/bin/sh
exec scala -J-Xmx2g "$0" "$@"
!#

println("test")

And trying to run it I get:

test.scala 
error: Compile server encountered fatal condition: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
at scala.tools.nsc.io.SourceReader.read(SourceReader.scala:61)
at scala.tools.nsc.io.SourceReader.read(SourceReader.scala:40)
at scala.tools.nsc.io.SourceReader.read(SourceReader.scala:49)
at scala.tools.nsc.Global.getSourceFile(Global.scala:395)
at scala.tools.nsc.Global.getSourceFile(Global.scala:401)
at scala.tools.nsc.Global$Run$$anonfun$30.apply(Global.scala:1607)
at scala.tools.nsc.Global$Run$$anonfun$30.apply(Global.scala:1607)
at scala.collection.immutable.List.map(List.scala:284)
at scala.tools.nsc.Global$Run.compile(Global.scala:1607)
at scala.tools.nsc.StandardCompileServer.session(CompileServer.scala:151)
at scala.tools.util.SocketServer$$anonfun$doSession$1$$anonfun$apply$1.apply(SocketServer.scala:74)
at scala.tools.util.SocketServer$$anonfun$doSession$1$$anonfun$apply$1.apply(SocketServer.scala:74)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withOut(Console.scala:65)
at scala.tools.util.SocketServer$$anonfun$doSession$1.apply(SocketServer.scala:74)
at scala.tools.util.SocketServer$$anonfun$doSession$1.apply(SocketServer.scala:69)
at scala.tools.nsc.io.Socket.applyReaderAndWriter(Socket.scala:49)
at scala.tools.util.SocketServer.doSession(SocketServer.scala:69)
at scala.tools.util.SocketServer.loop$1(SocketServer.scala:85)
at scala.tools.util.SocketServer.run(SocketServer.scala:97)
at scala.tools.nsc.CompileServer$$anonfun$execute$2$$anonfun$apply$mcZ$sp$1.apply$mcZ$sp(CompileServer.scala:218)
at scala.tools.nsc.CompileServer$$anonfun$execute$2$$anonfun$apply$mcZ$sp$1.apply(CompileServer.scala:213)
at scala.tools.nsc.CompileServer$$anonfun$execute$2$$anonfun$apply$mcZ$sp$1.apply(CompileServer.scala:213)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withOut(Console.scala:53)
at scala.tools.nsc.CompileServer$$anonfun$execute$2.apply$mcZ$sp(CompileServer.scala:213)
at scala.tools.nsc.CompileServer$$anonfun$execute$2.apply(CompileServer.scala:213)
at scala.tools.nsc.CompileServer$$anonfun$execute$2.apply(CompileServer.scala:213)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withErr(Console.scala:80)
at scala.tools.nsc.CompileServer$.execute(CompileServer.scala:212)
at scala.tools.nsc.CompileServer$.main(CompileServer.scala:180)
at scala.tools.nsc.CompileServer.main(CompileServer.scala)

It seems like Scala is compiling something near my script, but I don't quite know how to debug it and fix it.

Solution

TL;DR

Ubuntu's Scala package used to be incompatible with Java 8 (this has been fixed in 2.11.12-4). The solution was to uninstall Ubuntu's Scala package and install one of the official Scala packages. You might still want to do that, this time around, not due to incompatibility with Java, but because Ubuntu's latest packaged Scala version is still 2.11, while Scala's latest version is currently 2.13.

sudo apt remove scala-library scala
wget https://downloads.lightbend.com/scala/2.13.4/scala-2.13.4.deb
sudo dpkg -i scala-2.13.4.deb

Since many people were asking for the reason behind this issue and I was also curious about what caused it, I did some digging...

The root of the problem

In Java 9, Buffer subclasses (including ByteBuffer) were changed to override methods that in the superclass return Buffer to return the respective subtype.

Bug: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-4774077
Commit: https://github.com/AdoptOpenJDK/openjdk-jdk9/commit/d9d7e875470bf478110b849315b4fff55b4c35cf

This change is not binary backward compatible. If some Java code which calls one these methods directly in one of Buffer's subclasses is compiled with JDK9+, the generated bytecode will not run in JRE8 (even if the returned value is not used at all). This happens because the signature of the method when called will be compiled as java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer which doesn't exist in JRE8. However, if compiled with JDK8, the signature compiled into bytecode would be java/nio/ByteBuffer.clear:()Ljava/nio/Buffer which exists in the Buffer calss in both JRE8 and JRE9+.

Where did Scala go wrong? (Or did it?)

Scala compiler does use some of the methods affected by the changes above. Particularly, in the SourceReader class where the error in OP's question happened.

Looking at Scala's compatibility matrix, it says that we need at least Scala 2.11.12 to use JDK11, but it doesn't say much explicitly about the opposite direction of compatibility. It does say though that "Scala 2.12+ definitely doesn't work at all on JDK 6 or 7", so we could expect that 2.12+ is still compatible with JDK8, and even more so Scala 2.11.

Why did they break the compatibility then? Couldn't they just compile Scala's source code with an older JDK version? They didn't and they could, so much, that they still do it.

If we download one of the official Scala packages and check the manifest file for scala-compiler.jar, this is what we find:

Scala 2.11.12:

Bundle-Name: Scala Compiler
Bundle-RequiredExecutionEnvironment: JavaSE-1.6, JavaSE-1.7
Bundle-SymbolicName: org.scala-lang.scala-compiler
Bundle-Version: 2.11.12.v20171031-225310-b8155a5502
Created-By: 1.6.0_45 (Sun Microsystems Inc.)

Scala 2.13.4:

Bundle-Name: Scala Compiler
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Bundle-SymbolicName: org.scala-lang.scala-compiler
Bundle-Version: 2.13.4.v20201117-181115-VFINAL-39148e4
Created-By: 1.8.0_275 (AdoptOpenJDK)

So it seems Scala 2.11 is still being compiled with JDK6 and Scala 2.13 is still being compiled with JDK8. Shouldn't that mean that they are both compatible with JRE8? Yes and indeed they are. Where's the error coming from then?

Where did Ubuntu go wrong?

Ubuntu, as most other Linux distributions do, likes to build its own packages that are made available through its package manager. This is done to ensure that everything works properly within the OS ecosystem, and that often means patching the source code of upstream projects.

Regarding the Scala package in particular, Ubuntu decided to ditch the upstream choices of JDK versions used to compile the Scala source code and has been using newer JDK versions to compile Ubuntu's Scala package for a while.

If we check the manifest file for scala-compiler.jar in Ubuntu's Scala 2.11.12-4, we can see that is was compiled with JDK11:

Created-By: 11.0.2+9-Ubuntu-3ubuntu1 (Oracle Corporation)
Bundle-Name: Scala Distribution
Bundle-SymbolicName: org.scala-ide.scala.compiler;singleton:=true
Bundle-Version: 2.11.12

Didn't you say the issue was resolved in 2.11.12-4? Yes, I did.

Ubuntu's solution for this problem was not to compile Scala with JDK8, but rather to patch Scala's source code to avoid calling the problematic methods directly in the subclasses. This was achieved by casting ByteBuffer (and CharBuffer) to its superclass Buffer before calling these methods. In practice, that meant changing Scala's source code from bytes.clear() to bytes.asInstanceOf[Buffer].clear().asInstanceOf[ByteBuffer] (not sure why they cast it back to ByteBuffer when the result from clear() doesn't seem to be used at all). Here is Ubuntu's patch.

Ubuntu's approach seems a bit dangerous, because other sources of incompatibility could have gone unnoticed and still be there waiting to happen in some very specific situation. Also having their own setup different from the official Scala releases means not having the whole Scala community testing these changes in real-case scenarios.