Dart vs Java (cont'd) — Richards and Tracer

This week I managed to port the rest of Dart’s benchmark_harness examples to Java.

The experience of porting Richards and Tracer was as smooth as that of porting the DeltaBlue benchmark. The only unfamiliar (and interesting) Dart feature I encountered that is worth noting was the ability to declare and pass method parameters by name.

Here are the numbers:

Richards Benchmark

Tracer benchmark

The results this time are limited to two recent nightly releases of the Dart SDK (22577 and 22720), and 32-bit and 64-bit Java. I ran each benchmark 3 times to warm up, and then 5 more times and took the best time of the 5 runs as the final number you see on the charts.

I did a lot of experimenting based on feedback I got on the Dart mailing list. I am well aware that Java needs to run a method some number of times before the JIT kicks in, and of the caveats of OSR. However, in my tests I found no substantial differences when running the benchmarks with a longer warm-up time.

Java 8 results were not different enough to warrant inclusion in my tests. I test both Java and Dart VMs with default settings, and do not intend to tweak custom VM flags in order to optimize each VM. It is obvious that a lot can be done by tweaking various VM parameters. My goal here is to get a gut feel for the relative out-of-the-box performance.

I was told that for a fair comparison Dart should be evaluated against the 32-bit client JVM, as the Dart VM is also optimized for use on client devices (with more focus on things such as faster startup vs long-term throughput). Hence, I include the 32-bit Client JVM in my tests. However, for all practical purposes, 64-bit JVMs are more relevant and in-use nowadays, so I feel obliged to still include the 64-bit server JVM in my tests. There is no client version of the 64-bit JVM by the way. To be fair, 64-bit compilation does have the advantage of access to a much larger set of registers, which can be used to gain performance.

Dart vs Java — the DeltaBlue Benchmark

As of the time of this writing the performance page on dartlang.org tracks Dart VM performance as measured by the DeltaBlue benchmark.

I ported the benchmark_harness Dart package (including the DeltaBlue benchmark) into Java and ran against the latest Java 7 and 8 JDKs.

The experience of translating Dart to Java was surprisingly smooth. Some of the most common small porting tasks included

  • Dart bool to Java boolean;
  • Dart C++-like super call syntax;
  • Dart constructor syntactic sugar;
  • Dart shorthand (=>) functions to Java full format;
  • Wrapping Dart top-level functions and variables inside a Java top-level class;
  • Changing the use of the Dart Function type to a Java Runnable;
  • The Dart truncating division operator ~/, which apparently is equivalent to plain division (/) when applied to integers;
  • Dart list access [] operator to Java List.get()

The trickiest part of the translation was the following piece of code that appeared absolutely befuddling at first sight:

<Strength>[WEAKEST, WEAK_DEFAULT, NORMAL, STRONG_DEFAULT,
           PREFERRED, STRONG_REFERRED][value]

As it turns out, this is simply an array literal

[WEAKEST, WEAK_DEFAULT, NORMAL, STRONG_DEFAULT,
 PREFERRED, STRONG_REFERRED]

prefixed by a generic type parameter specifying the type of the elements in the list

<Strength>

and followed by the list access ([]) operator, getting the element of the list at index value:

[value]

After working my way through this, the translation went smoothly, until I got to run the benchmark and hit a NullPointerException. In DeltaBlue, the BinaryConstraint constructor calls the addConstraint(), which is overridden in its subclasses. The ScaleConstraint sublcass implementation of addConstraint(), in particular, accesses ScaleConstraint fields that are initialized in the constructor. This pattern works in Dart, where apparently “this” constructor arguments are stored in their corresponding instance fields before the super constructor is invoked. Since this is not possible in Java (the super call must be the first statement in the constructor), I moved the addConstraint() call from BinaryConstraint to each of the subclass constructors. With that fix, the port was complete and I was able to run the Java version of the benchmark.

Here are the DeltaBlue numbers for Dart and Java on my ThinkPad W510:

Dart (22416)    2,810.39    355.82
Dart (22577)    2,283.11    438.00
Java (1.7.0_21-b11)    2,728.51    366.50
Java (1.8.0-ea)    2,693.14    371.31
Java (1.7 32-bit)    3,555.95    281.22

Update 5/11 More numbers: running for 45 seconds improves the performance of the 64-bit JVM (1.7,45s) but not the 32-bit one (1.7 32-bit,45s); the 32-bit Server JVM (32-server,45s) performs just as fast as the 64-bit JVM; the xxgreg version (xxgreg,45s) of DeltaBlue runs slower on the 64-bit JVM than my version ported from Dart; the xxgreg benchmark (xxgreg-run) uses a different harness and measurements include VM startup and warmup time.

Java (1.7 32-bit,45s)    3,533.99    282.97
Java (32-server,45s)    2,701.67    370.14
Java (1.7,45s)    2,559.38    390.72
Java (xxgreg,45s)    2,780.61    359.63
Dart (xxgreg-run)    2,356.70    424.32
Java (xxgreg-run)    2,800.10    357.13

The number in the first column is the runtime in us as reported by the benchmark harness at the end of a run. The second number is the score as defined on the dartlang.org performance page: “runs/second.” I ran the benchmark on each VM multiple times and as the variance between runs was small enough I picked the result from a random execution for each VM.

The first Dart VM (22416) is the current public release available on the Dart website, while 22577 is the current nightly build. I included the nightly build, as it is clearly visible on the dartlang.org performance page that Dart saw a major gain in performance as of build 22437. My test confirmed this observation.

The results are truly impressive. Dart, still a baby at 2 years of age and pre-1.0, already exhibits 15% better performance than Java, a veteran of 18 years. I think this truly deserves to be called a case of David vs Goliath.

Update: Both Dart VMs tested are 32-bit, while the two original Java VMs are 64-bit. Tested with the 32-bit Java 1.7.0_21 VM with even more disappointing results.