CONCEPT · 35 minutes
Learn why replacing rule-based keyword matching with JSON extraction transforms a scripted bot into a real agent — grounded in the Internal Support Dashboard experience.
The first version of the Internal Support Dashboard I built used keyword matching to route customer tickets. A ticket containing "refund" went to the billing team. One containing "error" or "crash" went to tech support. Anything else landed in a general queue. It worked in testing, where I wrote the test cases.
It did not work for real traffic.
Real customers do not write test cases. They write things like:
"There's no error on my screen — I just think the invoice total looks off."
My keyword matcher read "error" and routed that to tech support. The billing team never saw it. The customer got a reply about clearing their browser cache.
Here is the full matcher, so the scale of the problem is clear:
Six lines. The entire logic of "understanding a customer" reduced to string search. Adding more keywords made it worse — every new rule created a new edge case, and the edge cases were the ones that mattered.
The fix was not more keywords. The fix was to stop matching keywords altogether and ask the model to read intent — and to return a structured dict my code could act on.
Factor 1 in a sentence: Replace keyword matching with a model call that returns JSON. The model reads intent; your code acts on structure.
Think of JSON extraction like asking a colleague to fill out a structured intake form instead of just listening to a customer and guessing. You could transcribe the customer's exact words and try to parse meaning yourself, or you could hand your colleague a form with fields for issue type, urgency, and order number — and get back something the system can act on immediately.
That is what client.messages.create() does when you instruct it to return JSON. The model
reads the customer's message, reasons about intent, and fills in the form.
Here is the first real call, replacing the keyword matcher entirely:
Run this and the ticket that broke the keyword matcher routes correctly. The model read "invoice" and inferred billing, even with the word "error" present.
Before reading on: if the model returns "Sure! Here is the JSON: {\"category\": ..." instead
of pure JSON, what breaks and why?
The answer is json.loads(). It expects the string to start with {. Any prefix text — even a
polite preamble — causes a JSONDecodeError. The instruction "Return JSON only — no other text"
reduces this failure mode, but it does not eliminate it. A fallback is always needed:
Always catch JSONDecodeError. Even a well-instructed model occasionally adds a prefix. The fallback ensures no ticket goes unrouted. Log the bad response so you can improve the prompt over time — it is a signal, not just a nuisance.
Frameworks call this "tool use." That name does a lot of work to hide something simple.
When a framework tells you your agent is "using a tool to send an email," what is actually
happening is: the model outputs a JSON dict like {"action": "send_email", "to": "..."}, and
your code calls the email function with those parameters. The model did not touch the email API.
Your code did. The model wrote instructions; your code followed them.
Factor 4 says: own this. Write the routing yourself. A routing function is shorter and cleaner than anything a framework generates, and it is trivially testable without running the model.
The routing function is entirely testable without the model. Pass any dict and assert the output. This separation — extraction (LLM) vs routing (code) — is the payoff of Factor 4.
Combining extraction and routing produces the first real triage agent. It handles any ticket type through the same two-step pipeline: extract intent, then route.
No keyword lists. No if-chains that grow every time a new ticket type appears. The model does the classification; the routing function translates that classification into action.
The same two functions handle billing tickets and technical tickets with no changes to the routing code — only the extracted category changes.
Spend a moment on this question: if the model returns invalid JSON, what are your three realistic options?
Three choices exist: raise the error and fail fast (good in development, where you want to see the bug immediately); retry with the same or a revised prompt (works for probabilistic failures); or return a safe default and log the bad response (right for production, where the ticket must be routed even imperfectly). The choice depends on context. Later modules add state tracking to the triage session, which creates a fourth option: checkpoint the state and resume after the failure without losing the conversation history.
Application-level questions on JSON extraction and tool routing.
1. A keyword matcher routes tickets containing 'error' to the tech team. A customer writes: 'There's no error on my screen — the invoice total just looks wrong.' Which team receives this ticket, and why is that wrong?
Correct Answer: A
The word 'error' appears in the message, so the rule-based matcher sends it to the tech team — even though the actual issue is a billing discrepancy. This is the brittleness that JSON extraction fixes: the model reads intent, not keywords.
2. What does response.content[0].text hold after a client.messages.create() call that instructs the model to return JSON?
Correct Answer: B
The API returns a string. You must call json.loads() on it to get a Python dict. This is why wrapping the parse in a try/except json.JSONDecodeError is always the right move.
3. Your extractor calls the model and gets back: 'Sure! Here is the JSON: {"category": "billing"}'. What breaks next?
Correct Answer: B
json.loads() expects the string to begin with '{'. Any prefix text — even a polite preamble like 'Sure! Here is the JSON:' — causes JSONDecodeError. The fix: instruct the model to 'return JSON only — no other text', and still catch JSONDecodeError as a fallback.
4. Factor 1 says 'Natural Language → JSON'. Which best explains why this is the foundational primitive for agent systems?
Correct Answer: B
The key insight: once the model converts natural language into a structured dict, every downstream step (routing, validation, action dispatch) is ordinary, deterministic software. The model is scoped to exactly one transformation; everything else is code you control.
5. In route_ticket(), what is the recommended behavior when the extracted category does not match any key in the routes dict?
Correct Answer: C
Returning a safe default ensures no ticket is dropped. Logging the unexpected category creates a feedback loop: if 'refund_dispute' starts appearing frequently, you know to add it as an explicit category in the extraction prompt.
6. A framework says your agent 'uses tools autonomously'. Under the hood, what is most likely happening?
Correct Answer: B
Factor 4 in practice: 'tool use' is always the model writing JSON, your code reading it, and calling the matching function. The framework automates the dispatch loop. Understanding this makes every agent system debuggable — you can log the exact JSON and trace exactly which function was called.
7. If the model returns invalid JSON in production, what are your three realistic options in order of increasing resilience?
Correct Answer: A
Crash fast works in development where you need to see the bug. Retry works for probabilistic failures (the model sometimes prefixes). Safe default plus logging works in production where every ticket must be routed. The right choice depends on context — the next module adds state checkpointing as a fourth option.