CONCEPT · 60 minutes
Learn why human oversight matters in high-stakes AI systems and how to design checkpoints that let an agent work autonomously while pausing for human judgment exactly when it counts.
Early in my work on a de-identified clinical decision-support system, we faced a familiar shortcut: remove the human-review step before an AI recommendation reached a clinician. The model was accurate in testing. The review step added latency. Product pressure was real.
We removed the gate for one category of recommendations. Within days, a recommendation reached a clinician that was technically correct given the model's training data but contextually wrong given the patient's chart notes — notes added after the model's input snapshot was taken. The model had no way to know what it did not see. A human reviewer would have caught the discrepancy in seconds.
We put the gate back, and this time we designed it rather than bolting it on.
That experience shaped every human-in-the-loop system I have built since, including the checkpoints in my Python orchestration framework. The lesson is not that AI is unreliable. The lesson is: an agent that cannot pause and ask for help will eventually act on incomplete information at exactly the wrong moment.
Think of a human-in-the-loop gate the way a surgical team thinks of the pre-incision pause — a brief, structured confirmation that everyone agrees on the patient, the site, and the plan before proceeding. The pause is not a sign of distrust in the team. It is a design feature that catches the errors the team itself cannot catch from the inside.
The gap you cannot see: The most dangerous agent failures are not wrong answers — they are right answers to the wrong question. A human gate is often the only mechanism that can detect when the agent's input snapshot and the real world have diverged since the snapshot was taken.
This module adds a human-checkpoint layer on top of the task-orchestration agent built across modules 01–03. By the end, the agent will:
A checkpoint is not an error handler. It is a deliberate pause — the agent is not failing; it is surfacing a decision that requires human judgment before it acts. The distinction matters because checkpoints should be designed in advance, not added reactively after something goes wrong.
Think of a checkpoint like the escalation button a call-centre agent reaches for before making a commitment the company cannot undo. The agent does not wait for the call to go wrong; the escalation path is built into the process from the start.
The simplest checkpoint has three fields: the question for the human, the context needed to answer it, and the set of allowed responses.
The confidence field is self-reported by the model. It is not a calibrated probability; treat
it as a relative signal. A model that reports 0.3 is telling you it has two plausible paths
and cannot choose — that information is useful even if the number itself is imprecise.
Why stakes matter: A low-confidence step with low stakes might be fine to auto-log and continue. The same confidence on a high-stakes step warrants a blocking pause. Encode that distinction in the dataclass rather than hard-coding a single threshold for everything.
Now write a function that builds a checkpoint from the output of a step:
Before continuing: what would happen if the model always reported a confidence of 1.0? Would any checkpoints ever be triggered? Pause and work through the escalation logic you are about to read.
A checkpoint is the mechanism. The escalation policy is the rule that decides when to trigger one. Without a policy you get one of two failure modes: checkpoints on every step (friction that defeats the purpose of automation) or checkpoints on no steps (no better than removing the gate entirely).
In the clinical decision-support system, the escalation policy had three inputs: the model's self-reported confidence, the clinical stakes of the recommendation category, and a short list of categories flagged as "always review." My SDLC harness uses a simpler two-input version: confidence and a hard-coded list of step types that require human sign-off regardless of confidence.
The ALWAYS_REVIEW set is the most important line. It makes certain actions unconditionally
human-reviewed, regardless of confidence. This is where the clinical lesson lives: the model
cannot assess what it does not know it does not know. For actions that are irreversible or
externally visible, bypass the confidence check entirely and always pause.
Keep the always-review list short: If it grows to 20 items, you have not designed human-in-the-loop checkpoints — you have rebuilt a fully manual workflow wrapped in a Python function. Three to five step types is the right target.
Now integrate the checkpoint into the execution loop from module 03. The change is minimal: after the model executes a step, parse its confidence, apply the policy, and if a checkpoint is needed, call the reviewer before continuing.
Two helper functions — _parse_confidence, _parse_reasoning, and _parse_result — extract
the labelled fields from the model's response. Each is a short string-split on the label
prefix. Keeping parsing separate from execution logic makes both easier to test independently.
Prompt discipline: The model's response format must be consistent across calls or the parsers break silently. In production, use structured JSON output mode. For this example, labelled fields keep the code readable without adding a response schema.
When all steps pass the escalation check, the agent runs to completion without any pauses.
When a step triggers a checkpoint, the agent pauses and calls human_review_fn. If the human
approves, execution continues. If the human rejects, a RuntimeError is raised — the caller
decides whether to abort, retry, or re-plan.
The human_review_fn parameter is the seam that lets you swap in a real UI, a Slack bot, a
CLI prompt, or a test stub without touching the agent logic. In my SDLC harness, it routes to
a terminal prompt for local runs and to a webhook when running in CI.
Two contexts have shaped how I think about checkpoints, and they point in different directions.
In the clinical decision-support system, the stakes were patient safety. The escalation policy was conservative: any recommendation touching a clinical judgment required human sign-off before reaching a clinician. Speed was a secondary concern. The "always review" set covered every output category that could influence a care decision. Reducing latency was out of scope until the model had a verifiable track record of correct escalations.
In the SDLC harness, the stakes are lower and the trade-off runs the other way. The harness executes many steps per session. A blocking checkpoint on every low-confidence step would make it unusable for day-to-day work. The escalation policy triggers only when the agent is genuinely uncertain about a step that cannot be easily reversed — a report write, an external notification, a file deletion.
Design principle: Calibrate your escalation policy to the cost of a wrong unreviewed action, not the cost of a delayed action. In clinical AI, an unreviewed error can affect patient care. In a coding harness, an unreviewed error means rewriting a file — recoverable, so the speed gain is worth the reduced oversight.
The HumanCheckpoint dataclass, the needs_human_review function, and the human_review_fn
seam are identical across both contexts. Only the contents of ALWAYS_REVIEW and the value of
CONFIDENCE_THRESHOLD differ.
Module 06 will show you why linear execution with shared context is the right default architecture for agents. Human-in-the-loop is not a competing idea — it fits naturally inside the linear loop. A checkpoint is just a step whose "tool" is a human reviewer.
A multi-agent system where agents run in parallel has no clean place to insert a blocking checkpoint. A linear agent does, because there is always a well-defined moment to pause before the next step begins. That is one more reason to prefer the linear architecture by default.
Application-level questions about human-in-the-loop design.
1. Under the escalation policy in this module, which of the following would NOT trigger a human checkpoint?
Correct Answer: C
'read_file' is not in ALWAYS_REVIEW, stakes are 'low' (not 'high'), and confidence 0.90 is above the 0.75 threshold. None of the three policy branches fire, so no checkpoint is triggered.
2. Why does the ALWAYS_REVIEW set override confidence scores?
Correct Answer: B
The clinical-AI lesson is that a model cannot know what it does not know. For categories where an unreviewed error is costly or irreversible, we do not trust the model's self-reported confidence regardless of its value. A stale input snapshot could produce a high-confidence but contextually wrong recommendation.
3. If the model always reported confidence 1.0, what would change about checkpoint behavior?
Correct Answer: C
The confidence branch (confidence < CONFIDENCE_THRESHOLD) would never fire if confidence is always 1.0. But needs_human_review has two other independent branches: step_type in ALWAYS_REVIEW and stakes == 'high'. Those are not gated on confidence, so they would still trigger checkpoints for the appropriate steps.
4. In the escalation path, a human returns 'reject'. What happens next?
Correct Answer: C
A 'reject' response causes execute_step_with_checkpoint to raise RuntimeError with the step id. The caller (run_agent or the orchestrator) decides what to do — abort, log, or re-plan. The agent does not retry or skip silently; the error surfaces the decision upward.
5. Why is human_review_fn passed as a parameter rather than called directly inside execute_step_with_checkpoint?
Correct Answer: B
Injecting the review function as a parameter is a dependency-inversion pattern. It decouples the agent's execution logic from the review mechanism. The SDLC harness uses this seam to route to a terminal prompt in local runs and a webhook in CI — zero changes to the agent code in either case.
6. What is the key difference between a checkpoint and an error handler?
Correct Answer: B
A checkpoint fires before the agent acts — it is a deliberate design decision about when to surface a choice to a human. An error handler fires after something has already gone wrong. The clinical-AI lesson was that waiting for an error is too late: the wrong recommendation had already reached the clinician before anyone could catch it.
7. Why is a linear agent better suited for blocking human checkpoints than a parallel multi-agent architecture?
Correct Answer: B
A linear agent executes one step at a time. There is always a clear, sequential point to pause and wait for human input before the next step begins. In a parallel architecture, steps run simultaneously — inserting a blocking checkpoint would halt all parallel branches, not just the one under review, making the interaction awkward and error-prone.