Benchmarking without fooling yourself

Key points

• A hand-written System.nanoTime() loop measures almost nothing useful. The JIT can delete work whose result is unused, fold constants the compiler can see, and hoist invariants out of the loop.
• Dead code elimination is the most common failure: if you never use the result, the computation can be removed entirely and you time an empty loop.
• Constant folding is next: a benchmark over a literal input is computed once at compile time, so you measure a field read.
• Warm-up matters enormously. The first thousands of iterations run interpreted, then tiered compilation kicks in - so an un-warmed benchmark measures the interpreter.
• JMH addresses all of these: Blackhole consumes results, @State keeps inputs opaque, and it manages warm-up, forks and iterations for you.
• Fork more than once. A single JVM can land in one lucky or unlucky JIT profile, and running several forks is what exposes that variance rather than reporting it as a result.

Example

// WRONG. This can legitimately measure an empty loop.
long start = System.nanoTime();
for (int i = 0; i < 1_000_000; i++) {
    Math.sqrt(i);                  // result unused -> may be deleted entirely
}
long ns = System.nanoTime() - start;   // meaningless

// RIGHT - JMH, with the traps closed.
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@Warmup(iterations = 5, time = 1)      // let the JIT settle first
@Measurement(iterations = 10, time = 1)
@Fork(3)                               // separate JVMs expose JIT variance
public class HashBenchmark {

    // Not final and not a literal, so it cannot be constant-folded.
    private String input;

    @Setup
    public void setup() { input = "some-realistic-input-value"; }

    @Benchmark
    public int hashDirect() {
        return input.hashCode();       // returned -> JMH consumes it
    }

    @Benchmark
    public void hashIntoBlackhole(Blackhole bh) {
        bh.consume(input.hashCode());  // for results you cannot return
    }
}

/*
 * Read the error bar, not just the score:
 *
 *   Benchmark            Mode  Cnt   Score    Error  Units
 *   hashDirect           avgt   30   2.104 +- 0.031  ns/op
 *   hashIntoBlackhole    avgt   30   2.980 +- 0.052  ns/op
 *
 * If two scores overlap within their error, you have not measured a
 * difference - you have measured noise.
 */

Unused results get deleted and literals get folded, so a hand-rolled loop times nothing - use JMH and read the error bar.