Compiling Dotty with TeaVM

[mbovel]

I gave a shot trying to compile Dotty, the Scala 3 compiler, with TeaVM. I know compiling such a big project with that many dependencies is probably not expected to work out of the box, but I wanted to record the steps I followed somewhere and thought here might be a good place.

Dotty uses SBT as its build system. It can generate plain JARs, and I decided to try to use them directly with TeamVM CLI. I found no documentation about direct CLI usage, but found https://github.com/konsoletyper/teavm/blob/master/tools/cli/src/main/java/org/teavm/cli/TeaVMRunner.java which what I used.

To run it, I used Coursier. Coursier automates downloading a package from Maven or another Java packages repository and running it. It enables me to run TeaVM CLI using:

➜  ~ coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner -- --help  
usage: java org.teavm.cli.TeaVMRunner [OPTIONS] [qualified.main.Class]
...

Then, on the Dotty side, we can generate all JARs with:

➜  ~ git clone https://github.com/lampepfl/dotty dotty-fresh
➜  ~ cd dotty-fresh
➜  ~/dotty-fresh git:(main) sbt "dist / pack"
[info] welcome to sbt 1.9.7 (Eclipse Adoptium Java 17.0.6)
...

This generate a bunch of JARs in dist/target/pack/lib:

➜  ~/dotty-fresh git:(main) ls dist/target/pack/lib 
annotations-15.0.jar
antlr4-runtime-4.7.2.jar
autolink-0.6.0.jar
compiler-interface-1.9.6.jar
...

Let's check these JARs are indeed runnable:

➜  ~/dotty-fresh git:(main) java -cp 'dist/target/pack/lib/*' dotty.tools.dotc.Main -help
Usage: scalac <options> <source files>
where possible standard options include:
...

Let's try to compile with TeaVM!

➜  ~/dotty-fresh git:(main) coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner -- --classpath 'dist/target/pack/lib/*' dotty.tools.dotc.Main
INFO: Running TeaVM
Analyzing classes... complete for 1.158 seconds
Compiling... complete for 0.224 seconds
INFO: Output file successfully built
INFO: Classes compiled: 37
INFO: Methods compiled: 455
Build complete for 1.57 seconds

Mmh, but actually classes.js does not contain any of my classes:

➜  ~/dotty-fresh git:(main) grep dotty classes.js
<empty>

After fiddling around for a while, I actually found I need -t js. Then TeaVM tells me it cannot find my main class:

➜  ~/dotty-fresh git:(main) coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner -- --classpath 'dist/target/pack/lib/*' -t js dotty.tools.dotc.Main
INFO: Running TeaVM
Analyzing classes... complete for 0.882 seconds
INFO: Output file built with errors
ERROR: There's no main class: 'dotty.tools.dotc.Main'
Build complete for 1.007 seconds

I tried without the wildcard, but got the same error.

Then I tried to manually unzip my JARs:

➜  ~/dotty-fresh git:(main) for jar in dist/target/pack/lib/*; do unzip -o $jar -d classes/; done

Confirmed these classes are still runnable:

➜  ~/dotty-fresh git:(main) java -cp classes dotty.tools.dotc.Main -help
Usage: scalac <options> <source files>
where possible standard options include:
...

And tried to compile them with TeaVM:

➜  ~/dotty-fresh git:(main) time coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner -- --classpath classes -t js dotty.tools.dotc.Main > teavm_output.txt
coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner --  437.74s user 11.72s system 229% cpu 3:15.71 total

Now TeaVM does actually compile my classes! 🥳

➜  ~/dotty-fresh git:(main) grep dotty classes.js
<a lot of lines>

However, it shows many errors for missing classes:

➜  ~/dotty-fresh git:(main) grep ERROR teavm_output.txt > teavm_errors.txt
➜  ~/dotty-fresh git:(main) head teavm_errors.txt
ERROR: Method scala.runtime.LazyVals$Waiting.countDown()V was not found
ERROR: Method scala.runtime.LazyVals$Waiting.await()V was not found
ERROR: Method scala.runtime.LazyVals$Waiting.countDown()V was not found
ERROR: Method scala.runtime.LazyVals$Waiting.countDown()V was not found
ERROR: Method scala.runtime.LazyVals$Waiting.await()V was not found
ERROR: Method scala.runtime.LazyVals$Waiting.countDown()V was not found
ERROR: Class java.nio.file.Path was not found
ERROR: Class java.nio.file.Path was not found
ERROR: Class java.nio.file.Path was not found
ERROR: Class java.nio.file.Path was not found

These errors basically fall into 2 categories:

• Java standard library classes missing
• runtime.LazyVals methods missing

Then I read from the documentation:

Due to number of reasons TeaVM does not rely on Java class library of host JVM. Instead, TeaVM comes with its own implementation of subset of Java class library. Of course, it's a subset of Java class library. The list of supported classes is available here.

So I wondered: where can I find a JAR with the TeaVM implementation of the Java standard library. I found the teavm-classlib package. I quickly tried to manually copy these classes, but it didn't work:

➜  ~/dotty-fresh git:(main) unzip -o /Users/mbovel/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/teavm/teavm-classlib/0.9.2/teavm-classlib-0.9.2.jar -d teavm_stdlib_classes
➜  ~/dotty-fresh git:(main) cp -r teavm_stdlib_classes/org/teavm/classlib/* classes
➜  ~/dotty-fresh git:(main) time coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner -- --classpath classes -t js dotty.tools.dotc.Main > teavm_output_2.txt
coursier launch org.teavm:teavm-cli:0.9.2 --main org.teavm.cli.TeaVMRunner --  622.46s user 53.47s system 225% cpu 4:59.13 total
➜  ~/dotty-fresh git:(main) grep ERROR teavm_output.txt | wc -l
    1643
➜  ~/dotty-fresh git:(main) grep ERROR teavm_output_2.txt | wc -l
    1643

What is the correct way to include the TeaVM version fo the Java standard library?

[konsoletyper]

I found no documentation about direct CLI usage, but found https://github.com/konsoletyper/teavm/blob/master/tools/cli/src/main/java/org/teavm/cli/TeaVMRunner.java which what I used.

TeaVM was never meant to be "JAR conversion tool", but rather "development tool", so CLI was never meant to be shipped to end users. I believe, there's 3rd-party plugin for SBT: https://github.com/sbt-teavm/sbt-teavm

However, it shows many errors for missing classes:

The output parts you presented are missing important stack trace information. These aren't usual runtime stack traces, they are merely approximations, but still have enough information to find out what went wrong.

So I wondered: where can I find a JAR with the TeaVM implementation of the Java standard library. I found the teavm-classlib package. I quickly tried to manually copy these classes, but it didn't work

Please, don't do it!

What is the correct way to include the TeaVM version fo the Java standard library

The correct question here is: how do I compile Dotty. And there are plenty of ways to do that.

1. You may fork Dotty and replace parts that require java.nio with something else (e.g. java.io)
2. You may examine Dotty sources and see if you can build your own entry point, using Dotty as a library. There's a small chance that core compiler does not depend on java.nio and uses some kind of own file system abstraction.
3. If it's still not an option for you, you can examine classlib sources and create a module that follows same naming pattern that implements java.nio.file, then compile it and use as a dependency to TeaVM build. Please note that this might be tough task, highly likely that if you just copy files from OpenJDK and rename them, they won't work with TeaVM.

[konsoletyper]

Another useful approach is to write TeaVM plugin. There's no documentation on this, see source code (find TeaVMPlugin interface and its implementations). One most useful thing is to implement class transformers. For Scala I implemented several runtime patches, see ScalaHacks class.

[mbovel]

Thanks a lot for the pointers!

I tried sbt-teavm: https://github.com/mbovel/sbt-teavm-scala3-test.

It works well with Scala 3 as long as I don't use lazy vals but, as hinted in sbt-teavm's Readme, fails with lazy val fields: https://github.com/mbovel/sbt-teavm-scala3-test/blob/lazy-vals/src/main/scala/main.scala.

I will not pursue trying to compile the Dotty codebase with TeaVM for now, but will keep your advices in mind if I do in the future!

[konsoletyper]

Looked into this issue and it looks like Scala guys took extremely bad decision in all ways to implement lazy vals. I don't know what exactly motivated them to do so, why didn't they use some more clean way, for example, AtomicFieldUpdater (like Kotlin compiler does). Not only they use sun.misc.Unsafe under the hood, their whole ABI (scala.runtime.LazyVals) is based on assumption that target JVM stores objects in a linear memory with constant field offsets, which is not true for case of TeaVM. So effectively no chance that TeaVM supports this ever, unless someone finds the way to convince compiler developers to remove this shit away from compiler and make it produce good bytecode with good ABI. For this I also don't see any reason for them, since community of Scala developers using TeaVM is almost inexistent.

[sjrd]

You might not have realized that you're directly talking to the "Scala guys". 😉

I don't know what exactly motivated them to do so, why didn't they use some more clean way, for example, AtomicFieldUpdater (like Kotlin compiler does).

Because we tried many alternatives, and this was the fastest one across a variety of access patterns, notably for many contended reads, an important access pattern for our lazy vals.

[konsoletyper]

Good. For the sake of extra 5% (of whatever numbers are) of performance in a synthetic test (and 0.005% impact in a typical application), you tied ABI with a thing that is being deprecated, that has problems with compatibility, for which even Graal VM had to write a workaround. I know there is high-performance Java, where people struggle for performance at any cost, using pooling, refusing from strings, even from objects in favour of large primitive arrays, etc. They also use some tricks with sun.misc.Unsafe. But, for 99% of normal Java development it is irrelevant, and as soon as it causes troubles with compatibility and tooling, I would not end up with such solution.

Ok, you decided to implement runtime with sun.misc.Unsafe, but did this really matter to repeat its shape with field offsets in ABI? I think JVM spec was designed deliberately to avoid term of address or pointer arithmetic. It's also helps performance: when you don't have direct access to memory, but only access to fields of objects, helps alias analysis, it allows to perform escape analysis, etc. What if someday a clever guy finds out another way to lay-out objects, where they don't have constant layout anymore? What if at some point AtomicFieldUpdater becomes faster than compareAndSwapLong? Except for some internal implementation detail (which is sun.misc.Unsafe) there's nothing about particular layout in Java spec. Why did't you create ABI that repeats, say, AtomicFieldUpdater API (and which is possible to implement either on top of AtomicFieldUpdater and compareAndSwapLong)? Why didn't you hide this behind invokedynamic`?

So I just hope that you, Scala guys, understand one day how extremely bad this decision was and find a way to fix it.