Benchmarks

We have graphed the performance of several popular implementations on the benchmarks that come with Gambit-C and on several other benchmarks that we have written or obtained from other sources. Please keep in mind that the results of benchmarking should not be taken too seriously.

To avoid comparing apples with oranges, we have separated our results into timings for

• unsafe compiled code, as in C and C++.
• safe compiled code, as in Java and Standard ML.
• safe compiled code with generic arithmetic.
• safe interpreted code with generic arithmetic.

The benchmarked systems:

• Larceny, using a development version that is intermediate between v1.0a1 and v1.0a2, with the default generational garbage collector.
• Chez Scheme 6.1.
• Gambit-C v3.0.
• gcc -O2. (For some allocation-intensive or garbage-collection-intensive benchmarks, we also wrote a custom storage allocator for the C version of the benchmark. These timings are identified by gcc (A).)
• Standard ML of New Jersey v110.0.3.
• Java v1.2 from Sun, with just-in-time compilation.

All of our timings represent the cpu time for a single run on a Sun Ultra 1 with no other users. (Eventually we will report the average of several runs, and for the garbage collection benchmarks we will eventually report real times, but the cpu times have been very repeatable.) When a few timings are missing for a system, this usually means that we have not been able to get the system to run those particular benchmarks; in some cases the problem is caused by non-portable code within a benchmark. When all of the timings are missing for a system, this usually means that the system cannot reasonably be configured to fit the category of systems being benchmarked. For example, gcc cannot be configured to generate safe code, to perform generic arithmetic, or to act as an interpreter.

The numbers shown are user cpu times obtained from a single run on an UltraSPARC with no other users. The bar graphs show relative performance. Longer is better.