Build the safeguards
Eight explained steps, a source example and a checkpoint at each decision.
Read the tutorialGo / Distributed systems / Recovery
Compare the choices.
Choose a pattern by the failure it addresses. Several safeguards can be necessary at once.
Which approach fits the problem?
| Approach | Useful when | Decision or limitation |
|---|---|---|
| Attempt timeout | Bound one downstream wait | Does not bound all retries by itself |
| Request deadline | Bound an entire operation | Every layer must propagate and observe it |
| Retry budget | Cap extra attempts | Requires transient-error and operation-safety rules |
| Bulkhead | Bound concurrent dependency work | Needs a policy for rejected or queued requests |
| Circuit breaker | Skip calls while a dependency appears unhealthy | Requires recovery probes and suitable thresholds |
| Load shedding | Reject excess work before consuming scarce capacity | Makes an explicit availability tradeoff |
Which environment should you use?
| Environment | What you can do | Scope |
|---|---|---|
| This browser lab | Click, compare, reset and inspect exact source | Actual Go runs as WebAssembly with timed in-process dependency calls. The native download uses real loopback HTTP. These fixtures show control flow and cleanup; they are not a multicore benchmark, a network-fault simulator or a production capacity result. |
| Independent download | Run the same prebuilt lab with Python’s local server | No package installation or automatic external calls needed for the prebuilt preview |
| Native exercise | Run language tests and the included local adapter or script | Requires the documented local toolchain; does not provision a cloud service |
A comparison you can reproduce
The safe implementation allows at most three dependency calls in flight and returns with zero active calls. Recovery eventually permits successful probes. Shed requests are counted separately from work that started and failed.
What this adds to a conventional tutorial
Begin with an observable outcome, inspect the decisions that produced it, then test a repair. The companion tutorial explains the mechanism; the how-to guide takes the same source into an independent environment.
Read the primary documentation
Customer.io’s backend role emphasizes Go systems, recovery, backpressure and reliability tradeoffs.
Choose your next step
Compare the alternatives
Understand where each pattern helps and which tradeoffs remain.
Compare approachesRun and extend it locally
Download the independent environment, run the native checks and try a specific extension.
Open the how-to guideLab discussion / Go Service Resilience Lab
What would you try next?
Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.
Code walkthrough
Follow the implementation in code.
These short excerpts connect the lab’s behavior to its implementation. Each example links to the full tutorial step for its context, tradeoffs and checkpoint. Names and dependencies belong to that source; the excerpts are not standalone programs.
Assign one end-to-end budget
The safe request has an 85 ms budget that includes its attempts and retry delays. Individual attempts have a 35 ms limit. A fresh full timeout on every hop can exceed the time the caller is willing to wait; a shared deadline gives downstream work an upper bound.
budget, cancel := context.WithTimeout(parent, 85*time.Millisecond)
defer cancel()Make the dependency honor cancellation
A deadline is useful only if work observes it. The fixture selects between its timer and ctx.Done. The native adapter attaches the same context to a real HTTP request. CPU loops, database drivers and external APIs may need different cancellation mechanisms, which should be tested rather than assumed.
select {
case <-ctx.Done(): return ctx.Err()
case <-timer.C: return nil
}Classify failures before retrying
A transient unavailable response may justify another attempt. A malformed request usually does not. The example only retries its unavailable sentinel or an attempt deadline, and caps the count. Real mutating operations also need a safe retry contract; a timeout can leave the original operation’s outcome unknown.
errors.Is(err, ErrUnavailable)
errors.Is(err, context.DeadlineExceeded)Limit active work and reject excess demand
A buffered channel acts as a three-slot bulkhead. The example immediately sheds requests when no slot is available. That differs from an unbounded queue, which preserves requests by consuming memory and increasing wait time. A real service may choose a small bounded queue with its own deadline.
select {
case slots <- struct{}{}: defer func(){ <-slots }()
default: return capacityRejected
}Join work before returning
The harness joins every started goroutine, counts remaining active calls and returns a bounded timeline. Native tests check cleanup, capacity, invalid presets and recovery. Race instrumentation is a separate compiler-assisted check; the browser target cannot establish race-detector cleanliness.
wg.Wait()
// active must now be zero