Dart vs Java — the DeltaBlue Benchmark

As of the time of this writing the performance page on dartlang.org tracks Dart VM performance as measured by the DeltaBlue benchmark.

I ported the benchmark_harness Dart package (including the DeltaBlue benchmark) into Java and ran against the latest Java 7 and 8 JDKs.

The experience of translating Dart to Java was surprisingly smooth. Some of the most common small porting tasks included

  • Dart bool to Java boolean;
  • Dart C++-like super call syntax;
  • Dart constructor syntactic sugar;
  • Dart shorthand (=>) functions to Java full format;
  • Wrapping Dart top-level functions and variables inside a Java top-level class;
  • Changing the use of the Dart Function type to a Java Runnable;
  • The Dart truncating division operator ~/, which apparently is equivalent to plain division (/) when applied to integers;
  • Dart list access [] operator to Java List.get()

The trickiest part of the translation was the following piece of code that appeared absolutely befuddling at first sight:

<Strength>[WEAKEST, WEAK_DEFAULT, NORMAL, STRONG_DEFAULT,
           PREFERRED, STRONG_REFERRED][value]

As it turns out, this is simply an array literal

[WEAKEST, WEAK_DEFAULT, NORMAL, STRONG_DEFAULT,
 PREFERRED, STRONG_REFERRED]

prefixed by a generic type parameter specifying the type of the elements in the list

<Strength>

and followed by the list access ([]) operator, getting the element of the list at index value:

[value]

After working my way through this, the translation went smoothly, until I got to run the benchmark and hit a NullPointerException. In DeltaBlue, the BinaryConstraint constructor calls the addConstraint(), which is overridden in its subclasses. The ScaleConstraint sublcass implementation of addConstraint(), in particular, accesses ScaleConstraint fields that are initialized in the constructor. This pattern works in Dart, where apparently “this” constructor arguments are stored in their corresponding instance fields before the super constructor is invoked. Since this is not possible in Java (the super call must be the first statement in the constructor), I moved the addConstraint() call from BinaryConstraint to each of the subclass constructors. With that fix, the port was complete and I was able to run the Java version of the benchmark.

Here are the DeltaBlue numbers for Dart and Java on my ThinkPad W510:

Dart (22416)    2,810.39    355.82
Dart (22577)    2,283.11    438.00
Java (1.7.0_21-b11)    2,728.51    366.50
Java (1.8.0-ea)    2,693.14    371.31
Java (1.7 32-bit)    3,555.95    281.22

Update 5/11 More numbers: running for 45 seconds improves the performance of the 64-bit JVM (1.7,45s) but not the 32-bit one (1.7 32-bit,45s); the 32-bit Server JVM (32-server,45s) performs just as fast as the 64-bit JVM; the xxgreg version (xxgreg,45s) of DeltaBlue runs slower on the 64-bit JVM than my version ported from Dart; the xxgreg benchmark (xxgreg-run) uses a different harness and measurements include VM startup and warmup time.

Java (1.7 32-bit,45s)    3,533.99    282.97
Java (32-server,45s)    2,701.67    370.14
Java (1.7,45s)    2,559.38    390.72
Java (xxgreg,45s)    2,780.61    359.63
Dart (xxgreg-run)    2,356.70    424.32
Java (xxgreg-run)    2,800.10    357.13

The number in the first column is the runtime in us as reported by the benchmark harness at the end of a run. The second number is the score as defined on the dartlang.org performance page: “runs/second.” I ran the benchmark on each VM multiple times and as the variance between runs was small enough I picked the result from a random execution for each VM.

The first Dart VM (22416) is the current public release available on the Dart website, while 22577 is the current nightly build. I included the nightly build, as it is clearly visible on the dartlang.org performance page that Dart saw a major gain in performance as of build 22437. My test confirmed this observation.

The results are truly impressive. Dart, still a baby at 2 years of age and pre-1.0, already exhibits 15% better performance than Java, a veteran of 18 years. I think this truly deserves to be called a case of David vs Goliath.

Update: Both Dart VMs tested are 32-bit, while the two original Java VMs are 64-bit. Tested with the 32-bit Java 1.7.0_21 VM with even more disappointing results.