Zyel
By Chason ArthurRevision 151.032Published Updated

OpenTelemetry / Observability / SLOs

Build the safeguards.

Follow Request Tracing Lab from its visible failure to a testable implementation. Each step names one decision and a checkpoint.

  1. Start with the customer’s operation
  2. Create spans with useful boundaries
  3. Carry parent context explicitly
  4. Distinguish availability from latency
  5. Inspect the distribution, not just its mean
  6. Relate a target to an error budget
  7. Bound metric dimensions
  8. Reproduce the signal outside the browser

Step 1 of 8

Start with the customer’s operation

The root span describes GET /report, not an internal implementation detail. Its duration represents the operation the visitor experiences in this fixture. A health endpoint checks a different question and cannot replace a user-facing indicator.

const root = tracer.startSpan("GET /report", {}, ROOT_CONTEXT);

Checkpoint: Explain why a successful health check does not prove the report is responsive.

Step 2 of 8

Create spans with useful boundaries

The service and database-fixture spans surround work that can explain the request. Their names remain bounded and understandable. Attributes describe the operation without embedding arbitrary customer data. Too many tiny spans can create overhead without helping a diagnosis.

tracer.startSpan("db.query orders", {}, parentContext);

Checkpoint: Identify what the database span actually measures in this exercise.

Step 3 of 8

Carry parent context explicitly

The broken database span uses ROOT_CONTEXT and becomes an independent trace. The safeguarded path places the service span in the supplied context before creating the child. A real network boundary additionally needs compatible injection and extraction of trace headers.

const childContext = trace.setSpan(parentContext, serviceSpan);

Checkpoint: Verify that all three safeguarded spans share one trace ID.

Step 4 of 8

Distinguish availability from latency

Eight responses succeed, but one intentionally waits much longer. An availability-only indicator remains 100%. The latency indicator counts requests under 100 ms, including queueing and work inside the measured boundary. Define eligibility before choosing a target.

goodLatencyEvents / eligibleRequests

Checkpoint: Read the under-100-ms fraction independently from response success.

Step 5 of 8

Inspect the distribution, not just its mean

The example uses nearest-rank p95 on eight recorded durations. One long sample can dominate the tail while barely moving an average. A tiny fixture makes this arithmetic visible; production percentile estimates depend on aggregation methods, histogram buckets and traffic volume.

sorted[Math.ceil(0.95 * sorted.length) - 1]

Checkpoint: Explain why averaging p95 values across servers is generally not a valid fleet p95.

Step 6 of 8

Relate a target to an error budget

A target states the permitted fraction of bad eligible events over a specified window. The lab’s illustrative target is 90% under 100 ms. Real alerts need a window, minimum useful volume and burn-rate reasoning rather than an alarm on every slow request.

budgetFraction = 1 - target;
consumedFraction = badEvents / eligibleEvents;

Checkpoint: Name the missing production choices before calling the fixture an SLO policy.

Step 7 of 8

Bound metric dimensions

The cardinality preset counts label combinations. A customer ID creates a new series for each customer; a route template keeps the dimension bounded. Trace and log storage have their own privacy and cost rules, but they need not force every identifier into metric labels.

lab_requests_total{route="/report"}

Checkpoint: Find the eight label sets in the broken example and the one bounded set in the safe example.

Step 8 of 8

Reproduce the signal outside the browser

The local Node service exports real request spans and a Prometheus text endpoint. An optional Compose setup connects it to Prometheus and a provisioned Grafana dashboard. The browser itself runs neither a collector nor Grafana. Inspect the service and scrape configuration before extending them.

GET /report
GET /traces
GET /metrics

Checkpoint: Make local requests, then verify that the request and good-latency counters change.

Practice the interview explanation

  1. Which user experience does the indicator measure?
  2. How is context carried across an asynchronous or network boundary?
  3. Why can a mean conceal a problem, and what can a trace still miss?

Use the experiment as an example. State the failure, explain the chosen boundary, show the test and name the tradeoff. Keep measured results separate from production assumptions.

Read the primary documentation

Prefect’s platform opening explicitly includes service indicators, error budgets and OpenTelemetry-style monitoring.

Related role from the job review

Choose your next step

Build the safeguards

Eight explained steps, a source example and a checkpoint at each decision.

Read the tutorial

Compare the alternatives

Understand where each pattern helps and which tradeoffs remain.

Compare approaches

Run and extend it locally

Download the independent environment, run the native checks and try a specific extension.

Open the how-to guide

Lab discussion / Trace the Wait Lab

What would you try next?

Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.

All lab discussions

Reader comments

Open the discussion to read and add reader comments.

Read or write a comment →