By Chason ArthurRevision 151.032Published Updated

OpenTelemetry / Observability / SLOs

Compare the choices.

Choose a pattern by the failure it addresses. Several safeguards can be necessary at once.

Which approach fits the problem?

Patterns and tradeoffs
ApproachUseful whenDecision or limitation
Health checkProcess readiness or dependency probesDoes not measure every user operation
MetricsRates, distributions and alertingAggregation loses individual request detail
TracesFollow work within one requestSampling and missing instrumentation can hide work
Structured logsDiscrete events with diagnostic contextNeed correlation, retention and data controls
Mean latencyA useful aggregate in some investigationsCan conceal a slow minority
Latency SLIA defined fraction of eligible fast responsesRequires a target, window and eligibility contract

Which environment should you use?

Execution environments
EnvironmentWhat you can doScope
This browser labClick, compare, reset and inspect exact sourceThe worker creates real OpenTelemetry spans around actual timer waits. The database is a timed fixture. Eight requests teach latency arithmetic, not a production SLO window. A local HTTP service and optional Prometheus/Grafana recipe accompany the download.
Independent downloadRun the same prebuilt lab with Python’s local serverNo package installation or automatic external calls needed for the prebuilt preview
Native exerciseRun language tests and the included local adapter or scriptRequires the documented local toolchain; does not provision a cloud service

A comparison you can reproduce

Both versions return eight successful responses. The safeguarded trace contains the request, service and database spans under one trace ID. A long tail remains visible even though availability is 100%.

What this adds to a conventional tutorial

Begin with an observable outcome, inspect the decisions that produced it, then test a repair. The companion tutorial explains the mechanism; the how-to guide takes the same source into an independent environment.

Read the primary documentation

Prefect’s platform opening explicitly includes service indicators, error budgets and OpenTelemetry-style monitoring.

Related role from the job review

Choose your next step

Build the safeguards

Eight explained steps, a source example and a checkpoint at each decision.

Read the tutorial

Compare the alternatives

Understand where each pattern helps and which tradeoffs remain.

Compare approaches

Run and extend it locally

Download the independent environment, run the native checks and try a specific extension.

Open the how-to guide

Lab discussion / Trace the Wait Lab

What would you try next?

Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.

All lab discussions

Reader comments

Open the discussion to read and add reader comments.

Read or write a comment →

Code walkthrough

Follow the implementation in code.

These short excerpts connect the lab’s behavior to its implementation. Each example links to the full tutorial step for its context, tradeoffs and checkpoint. Names and dependencies belong to that source; the excerpts are not standalone programs.

Example 01 · Tutorial excerpt

Start with the customer’s operation

The root span describes GET /report, not an internal implementation detail. Its duration represents the operation the visitor experiences in this fixture. A health endpoint checks a different question and cannot replace a user-facing indicator.

const root = tracer.startSpan("GET /report", {}, ROOT_CONTEXT);

Read the full tutorial step

Example 02 · Tutorial excerpt

Create spans with useful boundaries

The service and database-fixture spans surround work that can explain the request. Their names remain bounded and understandable. Attributes describe the operation without embedding arbitrary customer data. Too many tiny spans can create overhead without helping a diagnosis.

tracer.startSpan("db.query orders", {}, parentContext);

Read the full tutorial step

Example 03 · Tutorial excerpt

Carry parent context explicitly

The broken database span uses ROOT_CONTEXT and becomes an independent trace. The safeguarded path places the service span in the supplied context before creating the child. A real network boundary additionally needs compatible injection and extraction of trace headers.

const childContext = trace.setSpan(parentContext, serviceSpan);

Read the full tutorial step

Example 04 · Tutorial excerpt

Inspect the distribution, not just its mean

The example uses nearest-rank p95 on eight recorded durations. One long sample can dominate the tail while barely moving an average. A tiny fixture makes this arithmetic visible; production percentile estimates depend on aggregation methods, histogram buckets and traffic volume.

sorted[Math.ceil(0.95 * sorted.length) - 1]

Read the full tutorial step

Example 05 · Tutorial excerpt

Relate a target to an error budget

A target states the permitted fraction of bad eligible events over a specified window. The lab’s illustrative target is 90% under 100 ms. Real alerts need a window, minimum useful volume and burn-rate reasoning rather than an alarm on every slow request.

budgetFraction = 1 - target;
consumedFraction = badEvents / eligibleEvents;

Read the full tutorial step