The experimentBrice wrote an experiment to calculate the cost of going immutable and comparing it versus being mutable.
Although very risky, he emphasizes that his results cannot be used to draw bad/flame/godwin conclusions like "multable is soooo better that immutable, you functional suckers !". There are a lot of flawed benchmark on the internet that claim to be the only truth, so let's also rejoice when experiments reports try to be objective, complete and come with a disclaimer about their limits.
My specsI ran the code on my laptop, without any application running in the background (not even a X server) to make sute the system was quiet. I used OpenJDK 1.7.0_40. My CPU is a quad-core Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz.
Also, my memory bandwidth is about 2.5GB/s. This is an important information.
The attemptI ran the test with the following parameters :
- Parallel GC, PrintGCDetails, PrintTenuringDistribution, Xms1g, Xmx1g
- Parallel GC, PrintGCDetails, PrintTenuringDistribution, Xms6g, Xmx6g, Xmn5g
- CMS, PrintGCDetails, PrintTenuringDistribution, Xms1g, Xmx1g
- CMS, PrintGCDetails, PrintTenuringDistribution, Xms6g, Xmx6g, Xmn5g
- G1, PrintGCDetails, PrintTenuringDistribution, Xms1g, Xmx1g
- G1, PrintGCDetails, PrintTenuringDistribution, Xms6g, Xmx6g, Xmn5g
CMS and G1 runs were only for science ;-)
The resultsHere is the table of results that I built from this runs, sorted by full average :
|ImmutabilityTest||6mx 6ms 5mn||G1||10000||38612||1164||1803||1361|
|ImmutabilityTest||6mx 6ms 5mn||Parallel||10000||11813||1093||1605||1217|
|MutabilityTest||6mx 6ms 5mn||G1||10000||13058||1027||1310||1180|
|ImmutabilityTest||6mx 6ms 5mn||CMS||10000||12182||1081||1603||1171|
|MutabilityTest||6mx 6ms 5mn||Parallel||10000||31577||941||1336||997|
|MutabilityTest||6mx 6ms 5mn||CMS||10000||6518||770||973||819|
For each test, two files were generated :
- One file that has the console output (run-xxxxx)
- One file for the GC log
The file names follow this convention : <TYPE>-<GC>-<MX>-<MS>-<MN?>.log where
- TYPE is "run" or "gc", depending on the log
- GC is "parallel", "cms" or "g1", depending on the GC that was used
- MX is the max heap size in GB
- MS is the min heap size in GB
- MN is the young generation size in GB
So, what can we conclude from these results ? Well, Not much, I'm afraid.
There are lots of problems with this experiment on my machine. One of those is that we are measuring things that take about 1 millisecond. This resolution is very very very low, and we cannot see significant differences at this timescale.
Also, having a look at the GC logs show that the GC durations, for PSYoungGen and ParNew are spread between 1ms and 10ms. Say it differently : a GC cycle take between 1 and 10 times the duration of an iteration. Therefore a GC cycle would have a huuuge impact of our microbenchmark.
Now, the real stuff. If we look at the "Parallel GC, 1mx, 1ms" GC log, we can see two different phases in memory utilization. Those two phases are the immutable test first and then the mutable test.
Now, if we look at the allocation rate, things get really interresting.
See that almost flat cloud of dots at ~2.5GB/s ? This means that at that time, the application was creating that much garbage. At the beggining of this article, I mentionned that my memory bandwidth was about 2.5GB/s, now you can see it clearly.
The ImmutabilityTest is actually limited by the size of the wires between my CPU and my memory chips. To say this differently, the ImmutabilityTest does not measure how fast we can create Immutable dates holder, it measure how fast the data can go from my CPU to my RAM.
Therefore, all my ImmutabilityTest results in the previous table are invalid.
Final ThoughtsIt is quite often for microbenchmark not to measure different things than what we expect. This is another example of such misconceptions : the results were quite stable, they let us be confident of the experiment and yet were completely wrong.
I think Brice had noticed the limits of this benchmark, but did not push the analyze to its end. I hope this article added some clarifications and will give more materials for future experiments.
Btw, I did not mention CMS nor G1 because CMS never ran and because G1 needed to be tuned a bit. This could also be an interresting topic :-)