Build the safeguards
Eight explained steps, a source example and a checkpoint at each decision.
Read the tutorialGo / Distributed systems / Recovery
One slow service takes everything down.
A dependency slows down. Callers pile up. Retries ask it to do even more work. Change the policy and see what happens to capacity, failures and recovery.
The idea in plain language
Resilience means deciding how much work a request may start, how long it may wait and what should happen when the dependency cannot keep up. A rejected request is visible; an uncontrolled pile of waiting work is harder to recover from.
Actual Go runs as WebAssembly with timed in-process dependency calls. The native download uses real loopback HTTP. These fixtures show control flow and cleanup; they are not a multicore benchmark, a network-fault simulator or a production capacity result.
Your independent testing environment
Make the failure visible.
Start with slow responses. Compare attempts, peak in-flight work and explicit rejections. The recovery scenario sends a short sequence so you can observe the circuit opening, skipping calls and permitting a later probe.
Six callers meet a slow dependency. Compare per-attempt timeouts and total request budgets.
Ready. Choose an example and run both implementations.
Explain the result
The safe implementation allows at most three dependency calls in flight and returns with zero active calls. Recovery eventually permits successful probes. Shed requests are counted separately from work that started and failed.
Open the exact source behind this edition. The viewer checks each file against its SHA-256 manifest and explains the decision it implements.
Choose your next step
Compare the alternatives
Understand where each pattern helps and which tradeoffs remain.
Compare approachesRun and extend it locally
Download the independent environment, run the native checks and try a specific extension.
Open the how-to guideBefore you try it
Do I need to know how to code?
Start with Run and read the comparison. The tutorial introduces the code after you have seen the behavior. The how-to guide separates a no-install route from the native developer exercise.
What happens to the experiment when I reset?
Reset discards this lab’s temporary state and cancels or terminates its pending work. These examples use invented records. No account, production data or cloud deployment is needed.
What does the browser experiment establish?
Actual Go runs as WebAssembly with timed in-process dependency calls. The native download uses real loopback HTTP. These fixtures show control flow and cleanup; they are not a multicore benchmark, a network-fault simulator or a production capacity result.
Lab discussion / Go Service Resilience Lab
What would you try next?
Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.
Code walkthrough
Follow the implementation in code.
These short excerpts connect the lab’s behavior to its implementation. Each example links to the full tutorial step for its context, tradeoffs and checkpoint. Names and dependencies belong to that source; the excerpts are not standalone programs.
Assign one end-to-end budget
The safe request has an 85 ms budget that includes its attempts and retry delays. Individual attempts have a 35 ms limit. A fresh full timeout on every hop can exceed the time the caller is willing to wait; a shared deadline gives downstream work an upper bound.
budget, cancel := context.WithTimeout(parent, 85*time.Millisecond)
defer cancel()Make the dependency honor cancellation
A deadline is useful only if work observes it. The fixture selects between its timer and ctx.Done. The native adapter attaches the same context to a real HTTP request. CPU loops, database drivers and external APIs may need different cancellation mechanisms, which should be tested rather than assumed.
select {
case <-ctx.Done(): return ctx.Err()
case <-timer.C: return nil
}Classify failures before retrying
A transient unavailable response may justify another attempt. A malformed request usually does not. The example only retries its unavailable sentinel or an attempt deadline, and caps the count. Real mutating operations also need a safe retry contract; a timeout can leave the original operation’s outcome unknown.
errors.Is(err, ErrUnavailable)
errors.Is(err, context.DeadlineExceeded)Limit active work and reject excess demand
A buffered channel acts as a three-slot bulkhead. The example immediately sheds requests when no slot is available. That differs from an unbounded queue, which preserves requests by consuming memory and increasing wait time. A real service may choose a small bounded queue with its own deadline.
select {
case slots <- struct{}{}: defer func(){ <-slots }()
default: return capacityRejected
}Join work before returning
The harness joins every started goroutine, counts remaining active calls and returns a bounded timeline. Native tests check cleanup, capacity, invalid presets and recovery. Race instrumentation is a separate compiler-assisted check; the browser target cannot establish race-detector cleanliness.
wg.Wait()
// active must now be zero