Human-Agent Collaboration

When I was deploying the de-identified clinical-AI system, the human review step was not optional — it was the product. An AI recommendation about patient care had to pass a clinician's eyes before it reached any record. No exceptions, no overrides, no "high confidence" bypass.

That constraint changed how I thought about system design. Most agent tutorials frame human-in-the-loop as a fallback — the thing the system does when the AI is uncertain. We did the opposite: the human gate was the first component we designed. Everything else existed to get work to that gate correctly and to get a decision back cleanly.

The result was not a slower system. It was a trustworthy one. The clinicians who used it said they felt in control, not bypassed. That distinction matters enormously — it is the difference between a tool that gets used and one that gets routed around. When a safety gate is invisible or unreliable, practitioners build informal workarounds that defeat its purpose.

Think of a human gate like an emergency stop on a factory line. You do not remove the emergency stop to make the line go faster. You design the line so the stop is reachable, reliable, and fast to act on — and then you optimize everything around it. The goal is not to eliminate the stop; it is to make every second the line is running, the stop is within reach and the operator trusts it.

That principle generalizes far beyond clinical AI. In the support-automation platform, the same pattern applied: tickets the agent could not classify confidently — ambiguous billing disputes, escalations with legal implications, anything touching account cancellation — went to a human before any response was sent. The support lead who reviewed escalations told us the gate made them more willing to let the agent handle the easy cases. They trusted the system precisely because they knew the hard cases still reached them.

This builds on another dynamic worth naming: systems that bypass reviewers — even with good intentions, even with high accuracy — erode trust rapidly when a mistake surfaces. In high-stakes domains, a mistake always surfaces eventually. A gate that is consistently in place is a gate that can be improved. A gate that was bypassed, even once, is a gate that cannot be trusted.

This is Factor 8 of the 12-factor agent methodology: Contact Humans as Tools. The escalation path is not a failure mode; it is a tool call — just like search or a database read. Just as you define a search_knowledge_base tool so the agent can retrieve relevant articles, you define an escalate_to_human tool so the agent can invoke human judgment. The model calls it when it recognizes it cannot resolve a situation confidently. This framing matters: treating escalation as a capability rather than a fallback shapes how you build, test, and measure the gate.

And it is Factor 9: Meet Users Where They Are — the review surface should be wherever your reviewers already work, not wherever a framework vendor thinks they should work. In the support-automation platform, that meant Slack. The support lead did not want a new portal; they wanted a message in the channel they already monitored. Response rates for Slack-based approval requests were roughly four times higher than for email-based ones in the support-automation platform. The channel matters.

The two factors work together. Factor 8 makes human judgment a first-class operation in the agent's tool set. Factor 9 makes the review surface frictionless enough that reviewers actually use it. A gate that triggers correctly but sends notifications to a portal nobody checks is not a working gate — it is a gate that has been socially bypassed.

By the end of this module, you will be able to:

• Add an escalate_to_human tool to the support triage agent using the Anthropic tool-use API
• Wire an approval gate that pauses execution and resumes after a human decision
• Handle both the auto-resolved and escalation paths cleanly
• Reason about when automation increases risk rather than reducing it

In the Anthropic SDK, human escalation is implemented as a tool call — the same mechanism used for search, database reads, and code execution. The agent sees escalate_to_human in its tool list and calls it when it lacks confidence. This keeps the interface consistent: the model decides when to escalate, and the code decides what to do with that decision.

The mental model for this: think of escalate_to_human the way you think of send_email or query_database. Each is a tool the agent uses to interact with the world. The difference is that send_email triggers an automated system and escalate_to_human triggers a human. From the model's perspective, the interface is the same — call the tool, provide the required inputs, and wait for the result. From the engineering perspective, the difference is that the human's response may take minutes or hours, so the agent must suspend between calling the tool and receiving the result. That suspension pattern is the core design challenge this module addresses.

Define the escalation tool as a JSON schema. This schema is also the agent's escalation policy: when a new engineer asks "what kinds of tickets go to human review?", the description field is the authoritative answer — not a Slack thread or a stale wiki page.

Notice the description field doubles as documentation. Because the policy is code, changes to it go through code review — adding a new escalation condition is a pull request, not a Slack message, and carries a reviewer, a diff, and a timestamp.

The schema also defines the information the model must provide when escalating. The reason field ensures the reviewer understands why the ticket was escalated, not just that it was. The priority enum — normal or urgent — routes the Slack notification: urgent escalations page the on-call lead immediately; normal ones go into the review queue. This routing logic lives in the notifier, not in the agent. The tool definition is the contract; the implementation is elsewhere.

Now use the tool in a single client.messages.create call. The model classifies the ticket and either drafts a response or calls escalate_to_human — in one round trip:

Copy this block, set ANTHROPIC_API_KEY, and run it. The function does one thing: ask Claude to handle a ticket, then check whether it wants to escalate. If any content block is a tool_use call to escalate_to_human, the model decided it cannot resolve the ticket confidently. Otherwise, it returns a draft response directly.

Using tool-use in the same call as the drafting step avoids two common mistakes. First, separate classification passes create two round trips and two chances for the model's reasoning to diverge: the classifier might say "escalate", but if you then ask the model to draft an escalation reason in a second call, the second call might disagree about the ticket's sensitivity. Keeping it in one call means the model's reasoning is self-consistent. Second, hard-coded rule lists — "if the ticket contains 'cancel' OR 'legal' then escalate" — become stale the moment ticket vocabulary evolves. The model's judgment adapts to new phrasings; your rule list does not.

There is also an audit benefit. The block.input dict the model returns when it calls escalate_to_human contains the model's stated reason for escalation. That reason is machine-readable and stored with the ticket. Over time, the distribution of escalation reasons is one of the most useful signals for improving the agent's prompt: if you keep seeing "ambiguous billing context" in the reasons, that is a flag to add billing-context handling to the triage prompt. The escalation call is both a handoff to a human and a signal back to the engineering team.

Now define the gate that receives an escalation and routes it to a human reviewer. Start with the data structure the gate needs:

The request_id is a UUID so that when multiple tickets are escalated simultaneously, the reviewer's approval response references the right one. The reviewer_note field carries the reviewer's edited draft or rejection reason back to the agent. A reviewer who approves the draft as-is leaves reviewer_note empty; one who wants to change the wording fills it in before clicking Approve. The agent prefers the reviewer's version when it exists.

Now the gate class itself. Its three responsibilities: serialize the escalation, notify the reviewer, and signal that execution should pause:

The state_store.save call precedes the notifier for a subtle but important reason: if a reviewer responds the instant the notification lands and the state has not been saved yet, the runner will fail to load the pending request. Always persist state before triggering external side effects.

The notifier and state_store are injected as dependencies rather than hard-coded. The gate does not need to know whether the notifier sends Slack, email, or a pager alert — injecting dependencies keeps the gate testable and replaceable without modifying the logic.

This testability was critical in the clinical-AI system. The gate had to be demonstrably correct before it could be trusted with patient data. A test injects a mock notifier that records what was sent and a mock state store backed by a Python dict. The gate's logic — save, notify, return — can be verified without touching a real database or sending a real notification. That same approach applies to the support triage agent: test the gate in isolation first, then test the full agent sequence.

What should a Slack notification contain? The minimal useful set from the support-automation platform: a summary of the ticket text, the model's escalation reason, a direct link to approve or reject, and the ticket ID. The whole decision should be makeable from the notification itself. Reviewers who had to click through to a separate UI to find context deferred decisions. Reviewers who could act from the notification responded quickly. Reduce friction at the decision point and response rates improve.

Mid-Module Checkpoint

Before moving on, consider: what would break if state_store.save and self.notifier.send happened in the opposite order? What specific failure mode does the current ordering prevent, and under what timing conditions would you actually see it?

Consider also: the HumanApprovalGate class takes a notifier and a state_store as constructor arguments. What would you need to change if you wanted to switch from Slack to email? What if you wanted to use a different database for the state store? How does this design compare to one that hard-codes a specific notifier implementation directly in the class?

In the support-automation platform, no escalated ticket could sit unanswered indefinitely. If the support lead did not respond within 30 minutes, the system escalated to the manager. If that also timed out, the ticket was blocked — never auto-approved and sent to the customer without a human sign-off.

Encoding this policy in code means it goes through code review, has a merge timestamp, and can be grepped by anyone on the team. Teams that skip this conversation discover the escalation policy at 2am during an incident. Writing the code forces the conversation before the system goes live.

Escalation policies also serve as a forcing function for defining accountability. The function get_escalation_tier forces the question: who is the secondary reviewer, and when are they available? In the support-automation platform, that conversation revealed that no one had defined who owned escalations after 5pm — a gap that would have surfaced during an incident if not caught in code review. The tier constants encode an organizational agreement, not just a technical parameter. When the agreement changes — the secondary reviewer changes roles, the SLA shifts, the support team expands — the code review process surfaces that change to everyone who needs to know.

The most important line is return 2. Tier 2 is blocked, not auto-approved. Many teams automate the fallback because they worry about throughput. In a support context where a wrong response to a legal or billing dispute can create obligations the company cannot reverse, that trade-off is wrong. A blocked ticket surfaces a process failure that can be investigated and fixed.

The constants at the top are placed there deliberately. When a compliance auditor asks "how long can a ticket sit before it escalates?", the answer is one grep away. Code that encodes operational policy should make that policy legible to non-engineers, not buried in configuration files.

One more pattern worth noting: the policy also creates a shared vocabulary for the on-call team. "This ticket is in Tier 1" means something specific — it has been waiting between 30 and 120 minutes, the support manager has been notified, and the system will not act on its own. Everyone on the team reads the same code and understands the same state. Shared vocabulary reduces coordination overhead during incidents, which is exactly when that overhead is most costly. A team that argues about what "escalated" means at 2am is a team that is not resolving the incident.

The gate fits into the triage agent as one step in a linear chain. Two paths exist: the happy path (agent resolves the ticket automatically) and the escalation path (agent calls escalate_to_human and the gate routes to a human). Each step takes a state and returns a state.

This is an important architectural choice. Many tutorials introduce human-in-the-loop as a special mode requiring separate infrastructure — a workflow engine, a long-running process, a message queue. Those tools are useful at scale, but they are not required for the pattern to work. A state store and a scheduled runner are enough to start. The support-automation platform ran for several months on a Postgres table and a scheduled background job before the team needed anything more complex. Start with the simplest implementation that is correct; add complexity only when the load demands it.

The TriageState dataclass makes every field the agent carries explicit. There is no hidden state. When the system is debugged after a production incident, the state at any point in the execution is inspectable. In the support-automation platform, incidents typically surfaced as "this ticket got a bad response — why?" The answer required tracing back through exactly what the agent knew at each step. With all state in a dataclass, that trace was a log query. Without it, the trace required instrumenting production code under pressure.

The boolean return from step_classify_and_draft is the branch point: True routes to the approval gate; False means the model drafted a response the runner can send directly. Most tickets in the support-automation platform auto-resolved — the gate was needed for a minority of edge cases. Designing for the common path (auto-resolve) keeps the happy path fast and low-friction, while the escalation path is robust for the cases that need it.

The second step requests approval and suspends. Notice it does not wait in a loop:

step_request_approval records the request_id and returns — it does not block. The caller, the agent runner, checks the gate's state store on a schedule and resumes the final step when a decision arrives. This keeps the agent stateless between steps: no thread is held open waiting for a human. The agent can be restarted, deployed to a different process, or crash entirely — the pending request lives in the state store, not in memory.

This is the same "stateless steps with explicit state" principle from the control-flow module, now applied to the specific challenge of asynchronous human decisions. The elegance of the pattern is that it requires no special infrastructure: no long-running process, no message queue, no distributed lock. A state store and a scheduled runner — the simplest possible implementation — are enough to handle the gate correctly. Add complexity only when throughput or reliability demands it.

When the human acts, the runner loads the ApprovalRequest, checks its status, and calls the appropriate path. There are exactly two paths: approve and reject. Making these two paths explicit functions keeps them independently testable and clearly named.

Before looking at the code, note the asymmetry: the approve path executes an action; the reject path only records data and stops. This asymmetry is intentional. Approval is a permission to act; rejection is a signal to stop and investigate. The reject path does not attempt to repair the draft or rerun the agent — it surfaces the rejection to the right people and closes the run. A human who rejected the draft knows something the agent does not. Capturing their rejection reason is how the system learns what it got wrong.

The approve path is intentionally simple: mark state as approved and send the response. Keeping it simple also makes it auditable — the log line shows exactly what was approved, by what process, and when. In the support-automation platform, the approve path log was the first thing the team consulted when a customer escalated a complaint about a previous response: was the response auto-resolved or human-approved? Which reviewer approved it? What was the original draft versus what was sent? All of that lived in the state log without any separate audit table.

The reject path does not retry by default. In the support-automation platform, a rejected response triggered a manual follow-up workflow — a human process, not an automated one. The agent had done its part by surfacing the draft; the rejection was a signal to the support team, not an error code to be handled programmatically. Treating rejections as feedback rather than errors is what enables the system to improve: the support lead who rejected a response knew something the model did not, and capturing that context is the first step toward addressing it.

The reviewer_note field in rejection_reason is also the feedback loop for prompt engineering. If a particular ticket type is rejected repeatedly with the same reason — "tone is too formal for a billing dispute" or "missing context about account age" — that pattern is visible in the state log. It can be translated directly into a prompt improvement. An agent that discards rejections silently cannot learn from them. An agent that records the reviewer's reasoning creates a dataset that makes future drafts better.

The runner ties all steps into a sequence and handles both paths. Notice the happy path returns early — most tickets should never reach the escalation gate:

Each step is a function; the runner is a sequence of function calls. Steps can be unit-tested independently, reordered without rewriting a class hierarchy, and understood at a glance. The runner receives state_store as a parameter — the same run_triage_agent function can be called in tests with an in-memory store and in production with a Postgres-backed store.

The thin runner pattern also means the runner can be swapped out for a proper workflow engine later. The support-automation platform ran for several months on a state store backed by a Postgres table and a scheduled background job. When review volume grew, the team introduced a workflow service — and because the steps were already decoupled from the runner, the migration was a change to the runner only, not to any step function. Design the steps to be portable; the runner is an operational concern.

One more pattern worth noting: the run_triage_agent function handles the happy path first — if needs_escalation is False, it returns immediately without touching the gate at all. This is intentional. Most tickets in the support-automation platform auto-resolved. The gate code was exercised only for the minority that needed it. A function that handles the common case without any overhead, and only calls the more complex path when needed, is easier to test and profile. The happy path should be short and fast; the escalation path should be correct and auditable.

Reflective Prompt

When would adding automated confidence-based approval — where the system sends the response if the model's reasoning looks complete — introduce more risk than keeping the human gate? What conditions would need to be true for automation to be the genuinely safer choice? What would you measure over time to know you had crossed that threshold?

Think about this in terms of the support-automation platform specifically: the ticket types that go through the human gate today are escalations with legal implications, account cancellations, and ambiguous billing disputes. Which of these, if any, could eventually be auto-resolved? What evidence would you need to gather — over how many tickets and what error rate — before you would feel comfortable removing the gate for that category? How would you define "error" in a way that is measurable before an incident rather than after?