In a previous blog post I created a setup to compare JVM performance of several JVMs. I received some valuable feedback on the measures I conducted and requests to add additional JVMs. In this second post I’ll look at some more JVMs and I’ve added some measures like process memory usage and startup time. Also I’ve automated the test (with a bash script) and reduced the complexity of the setup by removing haproxy and testing a single JVM at a time. I’ve used the same application to test.

Most important results:

• OpenJ9 uses least memory, GraalVM most
• OpenJ9 and GraalVM take longer to start than Oracle JDK and OpenJDK distributions
• Not all OpenJDK distributions give the same performance measures
• Oracle JDK uses less memory than the OpenJDK distributions and gives best response times

These results are not 100% consistent (although mostly they are) with previous measures here for the same application/JVMs. Differences:

• Zulu gave better response times than Oracle JDK in a previous test
• GraalVM gave slowest response times during a previous test. During this test it appeared to be one of the faster JVMs

Setup

Test application

I’ve used the reactive Spring Boot application from here.

JVMs

The JVMs which were looked at;

• openjdk:8u181
• oracle/graalvm-ce:1.0.0-rc9
• adoptopenjdk/openjdk8:jdk8u172-b11
• adoptopenjdk/openjdk8-openj9:jdk8u181-b13_openj9-0.9.0
• azul/zulu-openjdk:8u192
• store/oracle/serverjre:8

The versions were the currently available latest versions of the respective JVMs. I also quickly looked at Azul Zing but couldn’t get a Docker image with my application running quickly enough so for now I skipped this.

Automated tests

I’ve used SoapUI loadrunner to automate my tests with. First I executed a 10s ‘primer’ loadtest to reach a steady state. Next I performed a 5 minute test with the following settings:

Dockerfile

I’ve used the following Dockerfile:

FROM openjdk:8u181
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-XX:+UnlockExperimentalVMOptions","-XX:+UseCGroupMemoryLimitForHeap","-jar","/app.jar"]

And of course varied the FROM entry. This way of automation also made it a bit more difficult to include Zing as there are no Docker Hub images available for it. Creating an image myself did not work out of the box with the supplied examples and since it is just one of the JVMs to look at I decided to post this without Zing.

Docker-compose

The people from Docker have reduced the options which are available in the v3 docker-compose.yml file. In order to set memory limits and configure your network stack, a v2 docker-compose.yml is required. I’ve used the following:

I’ve used a memory limit to make sure all the JVMs were running under a similar amount of available memory.

I stopped, removed and recreated the spring-boot-jdk container. Everytime with a different JVM.

process-exporter

Why hardcode the network settings in the docker-compose.yml file? I wanted to measure the complete JVMs memory. When using for example Micrometer, you only get the memory used inside the JVM and not the memory the OS process uses. In order to achieve this, I’ve used process-exporter with the following configuration process-exporter.yml in the proc-exp folder:

This monitors java processes which have app.jar in their command-line. If I didn’t also check the command-line, my Java test processes would also be included and I didn’t want that.

Next I started process-exporter on my host with:

docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`/proc-exp:/config ncabatoff/process-exporter --procfs /host/proc -config.path /config/process-exporter.yml

I wanted to monitor process exporter with Prometheus inside my Docker container. To make this possible, my host (=gateway from within the Docker network) should be available at the same IP so I could configure that in my Prometheus configuration.

Results

Response time

I did HTTP GET requests from SOAPUI. This is the average response time of the service measured after a steady state was reached.

The reported response times by Micrometer from within the applications, were as follows:

OpenJDK and Oracle JDK were fastest while AdoptOpenJDK was slowest.

When looking at what SOAP UI reported as response times, we see something different.

This differs from what I measured previously. In that previous measure GraalVM appeared to provide the slowest response times while during this test, that was clearly not the case and GraalVM was one of the faster JVMs when looking at measures from within the JVM but also from outside the JVM.

Between the different measures, there was also quite a lot of difference. The response times from OpenJDK are slowest here instead of fastest. This makes me wonder if the measures from within the JVM across JVMs are really comparable and if they are measuring the same thing. This might differ due to implementation differences? AdoptOpenJDK was slowest in response times both when looking at within JVM measures and outside.

Startup time

This is the period in seconds reported by Spring Boot about how long it took for the application to start and how long the JVM was running before the application was actually up.

Here again we see the results are not quite as reproducible as I would want. Adopt OpenJ9 was clearly slowest in both tests for application startup followed by GraalVM. There’s no clear winner though.

Process memory usage

This is a result from process exporter on how much memory the Java process took in total. This consists of virtual, reserved/resident and swap memory. Swap memory was for all JVMs zero during the test. Virtual memory also consists of shared libraries (which are also used by other programs). When looking at resident and virtual memory I saw the following (using https://grafana.com/dashboards/249):

Clear winner here with least memory usage is OpenJ9 followed at distance by Oracle JDK. OpenJDK and GraalVM use most memory (both virtual and resident).

JVM memory usage

This is the heap and non-heap inside the JVM measured with Micrometer and exposed to Prometheus. Non-heap consists of reserved memory, a cache and PermGen space. Heap consists of several memory areas in which the JVM moves stuff around.

Heap

I’ve used the following Grafana dashboard: https://grafana.com/dashboards/4701. When looking at heap memory, OpenJ9 seems clearly to be the winner followed again at distance by Oracle JDK. GraalVM uses most memory for the same application within the JVM.

When looking at the parts the heap consists of, the different JVMs show some remarkable differences. Especially OpenJ9 behaves really differently compared to the other JVMs.

Non heap

While the other JVMs do not reserve much memory for the non-heap, OpenJ9 does, although it uses less memory. GraalVM uses most non heap memory. When we look at a bit more detail of what happens in the non-heap area we see the following:

OpenJ9 (the 4th bar in the graphs) clearly behaves differently.

Threads

When looking at threads, GraalVM uses slightly more threads and OpenJ9 a lot when compared to the other JVMs.

It is interesting to notice that even though OpenJ9 uses more threads, it does not use more memory.

Conclusions

Difficult to reproduce / large error

Startup times

OpenJ9 and GraalVM are slowest to start. The results here are not that reproducible (could not draw clear conclusions from the Oracle JDK and OpenJDK distributions) so I should do more tests on this with larger applications.

Response times

Since the response times measured inside and outside of the JVM differed a lot and the results were not solidly reproducible, I won’t draw any conclusions here yet. The response time measures between this test and a previous one, were inconsistent. Here Oracle JDK appears fastest while during a previous test Zulu appears fastest. During a previous test GraalVM appeared slowest while now it appears to be one of the faster JVMs. The most important changes between the previous test and this one is that I removed haproxy from the setup and tested one JVM at a time now while previously doing them all together at the same time.

Reproducible results / small error

Memory usage

Upon request I also looked at OS process memory. I used process-exporter to do this. Also I’ve split up heap and non-heap memory. All memory measures provided similar results in that the JVM which used most memory was GraalVM and the JVM which used least memory (by far) was OpenJ9. If memory usage is a concern I would recommend you to consider OpenJ9 as an option.

Some notes

Not looked at yet

• larger applications containing more complex logic
• non-reactive Spring Boot
• only compared Java 8 JVMs because for GraalVM at the moment of writing there was no newer version available yet. Is Java 11 faster? (I’m going to skip 9 and 10, no Oracle LTS versions)
• Azul Zing should be added as it is claimed to be fast
• GraalVM can produce native executables. Interesting to also use them in a comparison.
• Garbage collection behavior also differs. I have measures but did not have the time yet to look at it in more detail.
• I should spend time to make the setup and measures available in a suitable way so others can reproduce them.
• Measures might differ when running on different systems (Windows, MacOS, Linux) or different processor architectures.
• The image inside the docker container, which influences versions of surrounding libraries used by the JVM, differed. Some used Oracle Linux, some Debian, some Ubuntu.

GraalVM

Of course GraalVM is much more than just a JVM in that is allows you to run other languages such as Javascript (not confuse this with Nashorn or Rhino) and R in a seamless matter and allows you to create native executables which are supposed to be much faster. Haven’t tested this yet though.