The Deepest Layer

The trace showed you the slow span. But what was the CPU doing during those 800 milliseconds?
That answer lives one layer deeper.

trace waterfall trace_id: 7a3f9b2e

gateway

820ms

api-svc

810ms

db query

800ms

The trace tells you WHERE the time was spent: 800ms in the database query span.
But not WHY.

The trace shows the database query took 800ms. What can't it tell you?

The three golden signals observe the system from the outside — rates, durations, messages. Profiling looks inward. It captures the running code itself: which functions executed, where the CPU spent its cycles.
This is not another system signal. It is a window into a single moment of execution.

profile — db query span flamegraph

handleRequest 800ms

queryDatabase 780ms

getConnection 720ms

waitForPool 680ms

85% of the span's time: waiting for a connection from the pool

The flamegraph reveals the truth: 680ms of 800ms was spent in waitForPool.
The query itself was fast. The bottleneck was connection pool contention.

A span took 800ms. The trace shows it was a database call. What does profiling reveal?

What makes OTel's profiling different from traditional profilers?

Metrics tell you THAT. Traces tell you WHERE. Logs tell you WHAT. Profiling tells you WHY — by looking inside the code itself.

Profiling is OpenTelemetry’s newest signal, still experimental. Unlike traditional profilers, it connects each profile to a specific trace and span — slow span to flamegraph, in one click.

You began with silence. Now you can see — from the broadest metric to the deepest stack frame.

You have completed all 28 koans.

From a silent system to the deepest layer of observability.

You began with silence.

Now you can see.

← Return to the beginning