This module contains JMH microbenchmarks for Apache Pulsar, for questions about the performance of a single class or method, such as what a method, a data structure or a codec costs. JMH handles warm-up, dead-code elimination and measurement, which hand-written timing code gets wrong, so a benchmark is the strongest evidence for a small, self-contained optimization. The Pulsar Performance Testing Framework covers end-to-end scenarios that run a Pulsar cluster, and profiling and analysis.

Run benchmarks on Linux x86_64 when the numbers matter. That is Pulsar's most common deployment target, and results from elsewhere do not carry over. System.nanoTime() is far more expensive on macOS than on Linux, which skews the results in some cases — JMH's own measurement loop pays that cost on every invocation. async-profiler also supports only some of its sampling engines on macOS, so -prof async is less reliable there. Benchmarking on macOS or arm64 is fine while iterating — just confirm the result on Linux x86_64 before drawing a conclusion from it.

Results vary between runs when the CPU changes its frequency: turbo frequencies depend on the temperature and the power budget of the CPU, and power management changes CPU and device settings while the benchmarks run. On Linux, the performance testing environment setup configures the host for consistent results, with turbo disabled so that the CPU runs at a fixed frequency, and restores power saving afterwards.

The benchmarks are written using JMH. To compile & run the benchmarks, use the following command:

# Compile everything including the shaded microbenchmarks jar
./gradlew :microbench:shadowJar

# run the benchmarks using the standalone shaded jar in any environment
java -jar microbench/build/libs/microbench-*-benchmarks.jar

Display help:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -h

Listing all benchmarks:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -l

Running specific benchmarks:

java -jar microbench/build/libs/microbench-*-benchmarks.jar ".*BenchmarkName.*"

Running specific benchmarks with machine-readable output and saving the output to a file:

ts=$(date +%s)
java -jar microbench/build/libs/microbench-*-benchmarks.jar -rf json -rff jmh-result-$ts.json ".*BenchmarkName.*" | tee jmh-result-$ts.txt

The jmh-result-*.json file can be used to visualize the results using JMH Visualizer.

Checking what benchmarks match the pattern:

java -jar microbench/build/libs/microbench-*-benchmarks.jar ".*BenchmarkName.*" -lp

Profiling benchmarks with async-profiler:

Set LIBASYNCPROFILER_PATH to the path of the async-profiler library.

Corretto JDK ships with async-profiler (asprof binary and libasyncProfiler dynamic library)

LIBASYNCPROFILER_PATH=$(ls $JAVA_HOME/lib/libasyncProfiler.*)

Alternatively, download async-profiler from https://github.com/async-profiler/async-profiler/releases and install to ~/async-profiler directory.

Mac OS example:

LIBASYNCPROFILER_PATH=$HOME/async-profiler/lib/libasyncProfiler.dylib

Linux example:

LIBASYNCPROFILER_PATH=$HOME/async-profiler/lib/libasyncProfiler.so

Then run the benchmarks with the -prof argument:

java -jar microbench/build/libs/microbench-*-benchmarks.jar -prof async:libPath=$LIBASYNCPROFILER_PATH\;output=flamegraph\;dir=profile-results ".*BenchmarkName.*"

The default value for event is cpu, which is a request for the best available CPU sampling engine rather than a specific one, so what it resolves to depends on the platform. If the profiler fails to start, add \;event=itimer to the -prof argument: itimer is available everywhere.

It's possible to add options to the async-profiler that aren't supported by the JMH async-profiler plugin. This can be done by adding rawCommand option to the -prof argument. This example shows how to add all (new in Async Profiler 4.1), jfrsync (record JFR events such as garbage collection) and cstack=vmx options.

java -jar microbench/build/libs/microbench-*-benchmarks.jar -prof async:libPath=$LIBASYNCPROFILER_PATH\;output=jfr\;dir=profile-results\;rawCommand=all,jfrsync,cstack=vmx ".*BenchmarkName.*"

Outside Linux this particular command needs \;event=itimer as well. all turns on wall clock profiling, and where the cpu engine falls back to the wall clock engine the profiler refuses to start with Cannot start wall clock with the selected event, which shows up as a <failure> on the first warmup iteration and an empty result directory.

output=jfr writes one recording per benchmark, into a directory named after the benchmark under dir=. The jfrFlamegraphs Gradle task renders each recording into every view at once — point it at the whole output directory and it finds the recordings inside:

./gradlew jfrFlamegraphs -Pjfr=profile-results

Each recording gets a directory beside it named after the file without its extension plus a -flamegraphs suffix, holding cpu, wall, alloc and lock, each rendered merged (cpu.html), split per thread (cpu_threads.html) and grouped into async-profiler's categories (cpu_classify.html). A view whose event the recording does not contain is skipped.

The .jfr can also be opened in Eclipse Mission Control or IntelliJ IDEA, or analyzed by an AI agent. The Pulsar Performance Testing Framework leads to the analysis workflow and its tools.