Build the safeguards
Eight explained steps, a source example and a checkpoint at each decision.
Read the tutorialGo / Distributed systems / Recovery
Build the safeguards.
Follow Go Service Resilience Lab from its visible failure to a testable implementation. Each step names one decision and a checkpoint.
- Count attempts, not just requests
- Assign one end-to-end budget
- Make the dependency honor cancellation
- Classify failures before retrying
- Spread retries within the remaining budget
- Limit active work and reject excess demand
- Let the circuit test recovery
- Join work before returning
Step 1 of 8
Count attempts, not just requests
A customer request can create several downstream attempts. The broken implementation allows up to three attempts per caller. Count dependency work and peak activity alongside completed requests so the cost of retrying is visible. Do not describe rejected work as a successful response.
requests != dependencyAttemptsCheckpoint: Find the three separate counters for success, failure and shedding.
Step 2 of 8
Assign one end-to-end budget
The safe request has an 85 ms budget that includes its attempts and retry delays. Individual attempts have a 35 ms limit. A fresh full timeout on every hop can exceed the time the caller is willing to wait; a shared deadline gives downstream work an upper bound.
budget, cancel := context.WithTimeout(parent, 85*time.Millisecond)
defer cancel()Checkpoint: Explain why three 35 ms attempts cannot fit inside this budget.
Step 3 of 8
Make the dependency honor cancellation
A deadline is useful only if work observes it. The fixture selects between its timer and ctx.Done. The native adapter attaches the same context to a real HTTP request. CPU loops, database drivers and external APIs may need different cancellation mechanisms, which should be tested rather than assumed.
select {
case <-ctx.Done(): return ctx.Err()
case <-timer.C: return nil
}Checkpoint: Reset a browser run and explain which execution boundary is terminated.
Step 4 of 8
Classify failures before retrying
A transient unavailable response may justify another attempt. A malformed request usually does not. The example only retries its unavailable sentinel or an attempt deadline, and caps the count. Real mutating operations also need a safe retry contract; a timeout can leave the original operation’s outcome unknown.
errors.Is(err, ErrUnavailable)
errors.Is(err, context.DeadlineExceeded)Checkpoint: Add a permanent error in the native test and verify that it is not retried.
Step 5 of 8
Spread retries within the remaining budget
The lab uses a deterministic bounded delay derived from request ID so its scheduling choices can be inspected. A production policy normally uses appropriately randomized jitter. Backoff is not a promise of recovery; it prevents callers from repeatedly retrying at the same instant.
delay := 5 + (requestID*7 + attempt*3)%13Checkpoint: Identify why deterministic teaching jitter is different from production randomization.
Step 6 of 8
Limit active work and reject excess demand
A buffered channel acts as a three-slot bulkhead. The example immediately sheds requests when no slot is available. That differs from an unbounded queue, which preserves requests by consuming memory and increasing wait time. A real service may choose a small bounded queue with its own deadline.
select {
case slots <- struct{}{}: defer func(){ <-slots }()
default: return capacityRejected
}Checkpoint: In the burst preset, verify that peak in-flight dependency work stays at or below three.
Step 7 of 8
Let the circuit test recovery
After repeated failures the safe circuit skips dependency calls for a cooldown, then admits one probe. A successful probe closes it. A breaker can make a transient failure last longer if configured poorly, so the open interval and failure classification must match the dependency.
closed -> open -> one half-open probe -> closedCheckpoint: Run recovery and locate a skipped call followed by a successful later request.
Step 8 of 8
Join work before returning
The harness joins every started goroutine, counts remaining active calls and returns a bounded timeline. Native tests check cleanup, capacity, invalid presets and recovery. Race instrumentation is a separate compiler-assisted check; the browser target cannot establish race-detector cleanliness.
wg.Wait()
// active must now be zeroCheckpoint: Run native tests and identify what a race-detector run would add.
Practice the interview explanation
- At which layer should retries happen, and how do nested retries multiply attempts?
- How does a deadline propagate to downstream work?
- What distinguishes load shedding, a concurrency limit and a circuit breaker?
Use the experiment as an example. State the failure, explain the chosen boundary, show the test and name the tradeoff. Keep measured results separate from production assumptions.
Read the primary documentation
Customer.io’s backend role emphasizes Go systems, recovery, backpressure and reliability tradeoffs.
Choose your next step
Compare the alternatives
Understand where each pattern helps and which tradeoffs remain.
Compare approachesRun and extend it locally
Download the independent environment, run the native checks and try a specific extension.
Open the how-to guideLab discussion / Go Service Resilience Lab
What would you try next?
Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.