Build the safeguards
Eight explained steps, a source example and a checkpoint at each decision.
Read the tutorialPython / FastAPI / AI evaluation
The answer sounds good. Can it pass the test?
An answer cites a document. The document is old, unrelated or hostile. Check the actual claim, then make a slow dependency reveal whether the workflow respects its limits.
The idea in plain language
An AI-facing workflow needs a decision about whether an answer is supported, not just a string to display. A repeatable evaluation also makes abstention, invalid output and time limits visible.
Actual Python runs in the open-source Pyodide runtime. Candidate answers and documents are deterministic fixtures; no live model is called. The exact policy validator is intentionally narrow and does not establish general factual accuracy or comprehensive prompt-injection resistance.
Your independent testing environment
Make the failure visible.
Start with evidence checks. Compare the five fixture decisions. Then inspect the hostile-document case and slow-task cancellation. The download runs the same engine behind a local FastAPI endpoint.
Evaluate supported, unsupported, superseded, hostile and malformed candidate answers.
Ready. Choose an example and run both implementations.
Explain the result
The safeguarded evidence run matches all five expected fixture decisions. The slow run limits active tasks to two and cancels all five before returning. Active-at-finish is zero. These labels are regression cases, not an accuracy estimate.
Open the exact source behind this edition. The viewer checks each file against its SHA-256 manifest and explains the decision it implements.
Choose your next step
Compare the alternatives
Understand where each pattern helps and which tradeoffs remain.
Compare approachesRun and extend it locally
Download the independent environment, run the native checks and try a specific extension.
Open the how-to guideBefore you try it
Do I need to know how to code?
Start with Run and read the comparison. The tutorial introduces the code after you have seen the behavior. The how-to guide separates a no-install route from the native developer exercise.
What happens to the experiment when I reset?
Reset discards this lab’s temporary state and cancels or terminates its pending work. These examples use invented records. No account, production data or cloud deployment is needed.
What does the browser experiment establish?
Actual Python runs in the open-source Pyodide runtime. Candidate answers and documents are deterministic fixtures; no live model is called. The exact policy validator is intentionally narrow and does not establish general factual accuracy or comprehensive prompt-injection resistance.
Lab discussion / Python Evidence Evaluation Lab
What would you try next?
Ask about the code, compare an approach, or describe an edge case. Reader comments are reviewed before publication.
Code walkthrough
Follow the implementation in code.
These short excerpts connect the lab’s behavior to its implementation. Each example links to the full tutorial step for its context, tradeoffs and checkpoint. Names and dependencies belong to that source; the excerpts are not standalone programs.
Keep retrieval separate from generation
A simple token-overlap retriever ranks the invented documents. The candidate answers are recorded fixtures, so the same inputs can be tested repeatedly without a provider. Retrieval relevance, source authority and answer support are different questions; none is guaranteed by the others.
docs = retrieve(question, DOCUMENTS)Reject an invalid output shape
The validator requires an exact set of fields, a nonempty list of citation strings and a true integer period. Python booleans are a special case because bool subclasses int; the example uses an exact type check. A valid schema is a useful boundary, but it cannot establish a true claim.
type(answer["return_days"]) is intCheck authority, version and the actual claim
The active policy allows 14 days; the superseded policy allowed 30. A citation must identify an active relevant record, and the structured claim must match it. The text check is deliberately exact. General natural-language support would require a broader design and separately evaluated failure modes.
allowed = {d["id"]: d for d in docs if d.get("active") and "days" in d}Bound concurrency and queue wait
The safe workflow admits two active tasks through a semaphore. Each task’s timeout surrounds both acquisition and work, so queued tasks cannot wait indefinitely before their clock starts. TaskGroup waits for child completion or cancellation before the result is returned.
async with asyncio.timeout(.04):
async with sem:
await work()Move the same engine behind FastAPI
The local adapter uses Pydantic allowlists, a bounded run gate and a total deadline. It binds only to loopback and makes no model calls. A future provider adapter needs its own authentication, usage accounting, cancellation behavior and evaluation record before any live-performance claims.
class Experiment(BaseModel):
preset: Literal["evidence","injection","slow"]
variant: Literal["broken","safe"]