Dart vs Java (cont'd) — Richards and Tracer

This week I managed to port the rest of Dart’s benchmark_harness examples to Java.

The experience of porting Richards and Tracer was as smooth as that of porting the DeltaBlue benchmark. The only unfamiliar (and interesting) Dart feature I encountered that is worth noting was the ability to declare and pass method parameters by name.

Here are the numbers:

Richards Benchmark

Tracer benchmark

The results this time are limited to two recent nightly releases of the Dart SDK (22577 and 22720), and 32-bit and 64-bit Java. I ran each benchmark 3 times to warm up, and then 5 more times and took the best time of the 5 runs as the final number you see on the charts.

I did a lot of experimenting based on feedback I got on the Dart mailing list. I am well aware that Java needs to run a method some number of times before the JIT kicks in, and of the caveats of OSR. However, in my tests I found no substantial differences when running the benchmarks with a longer warm-up time.

Java 8 results were not different enough to warrant inclusion in my tests. I test both Java and Dart VMs with default settings, and do not intend to tweak custom VM flags in order to optimize each VM. It is obvious that a lot can be done by tweaking various VM parameters. My goal here is to get a gut feel for the relative out-of-the-box performance.

I was told that for a fair comparison Dart should be evaluated against the 32-bit client JVM, as the Dart VM is also optimized for use on client devices (with more focus on things such as faster startup vs long-term throughput). Hence, I include the 32-bit Client JVM in my tests. However, for all practical purposes, 64-bit JVMs are more relevant and in-use nowadays, so I feel obliged to still include the 64-bit server JVM in my tests. There is no client version of the 64-bit JVM by the way. To be fair, 64-bit compilation does have the advantage of access to a much larger set of registers, which can be used to gain performance.