Zyel
By Chason ArthurRevision 138.057Published Updated

Go / Distributed systems / Recovery

Run it. Change it. Verify it.

Start in your browser, then take the complete Go Service Resilience Lab into an independent local environment.

Try it without installing a toolchain

  1. Open the playground and keep its default scenario.
  2. Run the comparison and read the results before opening the code.
  3. Change the scenario, run again, and identify which decision changed.
  4. Open View the code, choose a source file and read the explanation above it.
  5. Reset the environment, including a run that is still pending.
Open the playground

Run your downloaded copy

Download and extract the ZIP into a new folder. With Python 3 installed, open a terminal in that folder and run the command below. It serves only this package on loopback; opening index.html directly is insufficient for workers and WebAssembly.

python preview.py

Open http://127.0.0.1:4210/ in your browser. Stop the server with Ctrl+C. The prebuilt preview needs no dependency installation and makes no automatic requests to another site. The source code and notices are included alongside it.

Download the independent lab

Run the native exercise

The commands below start in the extracted package root. Node-based examples require Node 24 or later; the Go adapter requires Go 1.27 or later; the Python adapter requires Python 3.11 or later. Install only the toolchain used by this lab. Exact dependency versions and the Node lockfile are in the download.

cd src/go
go test ./...
go run ./cmd/server

The safe implementation allows at most three dependency calls in flight and returns with zero active calls. Recovery eventually permits successful probes. Shed requests are counted separately from work that started and failed.

Read GETTING_STARTED.md for the native request examples and shutdown steps. Native servers bind to 127.0.0.1. The optional monitoring containers expose only loopback ports. They are a separate exercise from the browser preview.

Make one meaningful change

Add a retryable 429 response with a bounded Retry-After value and a nonretryable 400 response. Keep retries within the original request deadline. Then move the dependency into a separate process and compare cancellation behavior.

Add the expected result first, reproduce the failure, repair it, then rerun the native and browser checks. Use npm run rebuild for the Node/browser examples. The Go package includes rebuild_wasm.py; the Python package includes refresh_source.py.

Record what you learned

Capture the scenario, edition, input records, observed outcome and the test that checks it. Explain one alternative you rejected and why. Avoid converting a small local result into a throughput, security or accuracy claim about a production system.

Read the primary documentation

Customer.io’s backend role emphasizes Go systems, recovery, backpressure and reliability tradeoffs.

Related role from the job review

Choose your next step

Build the safeguards

Eight explained steps, a source example and a checkpoint at each decision.

Read the tutorial

Compare the alternatives

Understand where each pattern helps and which tradeoffs remain.

Compare approaches

Run and extend it locally

Download the independent environment, run the native checks and try a specific extension.

Open the how-to guide

Lab discussion / Go Service Resilience Lab

What would you try next?

Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.

All lab discussions

Reader comments

Open the discussion to read and add reader comments.

Read or write a comment →

Code walkthrough

Follow the implementation in code.

These short excerpts connect the lab’s behavior to its implementation. Each example links to the full tutorial step for its context, tradeoffs and checkpoint. Names and dependencies belong to that source; the excerpts are not standalone programs.

Example 01 · Tutorial excerpt

Assign one end-to-end budget

The safe request has an 85 ms budget that includes its attempts and retry delays. Individual attempts have a 35 ms limit. A fresh full timeout on every hop can exceed the time the caller is willing to wait; a shared deadline gives downstream work an upper bound.

budget, cancel := context.WithTimeout(parent, 85*time.Millisecond)
defer cancel()

Read the full tutorial step

Example 02 · Tutorial excerpt

Make the dependency honor cancellation

A deadline is useful only if work observes it. The fixture selects between its timer and ctx.Done. The native adapter attaches the same context to a real HTTP request. CPU loops, database drivers and external APIs may need different cancellation mechanisms, which should be tested rather than assumed.

select {
case <-ctx.Done(): return ctx.Err()
case <-timer.C: return nil
}

Read the full tutorial step

Example 03 · Tutorial excerpt

Classify failures before retrying

A transient unavailable response may justify another attempt. A malformed request usually does not. The example only retries its unavailable sentinel or an attempt deadline, and caps the count. Real mutating operations also need a safe retry contract; a timeout can leave the original operation’s outcome unknown.

errors.Is(err, ErrUnavailable)
errors.Is(err, context.DeadlineExceeded)

Read the full tutorial step