Metrics and tracing with the Observation API

Key points

• Before Boot 3 you instrumented twice: a Micrometer Timer for metrics and a separate span for tracing, with two chances to disagree.
• The Observation API records the event once. Registered handlers then derive a timer, a span and log context from the same start and stop.
• Boot auto-instruments HTTP requests, client calls and scheduled tasks. Custom observations are for business operations you actually want on a dashboard.
• Cardinality is the thing to get right. Low-cardinality tags go on metrics, where each distinct value creates a separate time series. High-cardinality values go only on the span, which is sampled.
• Tagging a metric with a user id or an order number is how a metrics backend runs out of memory. Micrometer tags HTTP metrics with the templated URI - /users/{id} - precisely to bound this.
• Trace context propagates automatically over HTTP through the auto-configured clients. Across a message broker or onto another thread it must be carried deliberately, or the trace ends there.

Example

@Service
public class CheckoutService {

    private final ObservationRegistry registry;

    CheckoutService(ObservationRegistry registry) { this.registry = registry; }

    public Receipt checkout(Cart cart) {
        return Observation.createNotStarted("checkout", registry)
                // Becomes a metric tag: keep the value set small and bounded.
                .lowCardinalityKeyValue("payment.method", cart.paymentMethod().name())
                .lowCardinalityKeyValue("currency", cart.currency())
                // Span only - never a metric tag. Unbounded values belong here.
                .highCardinalityKeyValue("cart.id", cart.id())
                .highCardinalityKeyValue("customer.id", cart.customerId())
                .observe(() -> {
                    Receipt receipt = payments.charge(cart);
                    inventory.reserve(cart);
                    return receipt;
                });
    }
}

// Trace context does not follow work onto another thread by itself.
@Configuration
class AsyncTracingConfig {

    @Bean
    Executor taskExecutor(ObservationRegistry registry) {
        ThreadPoolTaskExecutor delegate = new ThreadPoolTaskExecutor();
        delegate.initialize();
        // Without this wrapper the async work starts a brand-new trace and
        // cannot be connected to the request that triggered it.
        return ContextExecutorService.wrap(delegate.getThreadPoolExecutor(),
                ContextSnapshotFactory.builder().build()::captureAll);
    }
}

Record the observation once; put bounded values on metrics and unbounded ones on spans, or the metrics backend pays for it.