Prompt and Context Engineering

The first version of the support-automation platform I built was fast, cheap, and embarrassing. Each ticket came in as free text — "My login isn't working," "I was charged twice," "The export button is broken" — and the agent processed each one in isolation: no history, no system prompt, just a naive user message. It worked well in a test suite of twenty hand-picked tickets.

Two weeks into production, the regressions started. A user would open a ticket, add a follow-up comment, then a clarification. By the third turn, the agent had repeated the same troubleshooting step twice and then asked for information the user had already provided in turn one. The conversation history was there in the database — I was appending every prior turn verbatim. When a conversation hit eight or ten turns, the combined token count crossed the model's practical limit and older turns started getting trimmed silently.

The failure mode wasn't a crash. It was subtle and expensive: the agent re-classified tickets that had already been resolved, asked users to repeat themselves, and human agents — who could have been handling harder cases — spent their time re-triaging work the system should have owned.

Three problems I had ignored: what goes into the context, how much of it, and what happens when something goes wrong. This module addresses all three.

By the end of this module you will be able to:

• Write structured system prompts that give the model a clear operating contract (Factor 2)
• Build a sliding-window context manager that keeps multi-turn conversations on-budget (Factor 3)
• Handle malformed or truncated model output gracefully without crashing (Factor 10)

The running example is the customer support triage agent from the previous modules — the same system, growing one architectural layer at a time.

The single biggest difference between a prototype and a production agent is the system prompt. My first prompt was one sentence: "You are a support agent. Help the user." That is not a job description — it is a wish.

Think of the system prompt like a job description for a new contractor. A vague description ("help users") gets vague work: prose answers, inconsistent formats, and unpredictable escalation behavior. A precise one ("classify tickets into four categories, return structured JSON, route anything uncertain to human review") gets predictable, auditable results. The more specific the contract, the more consistent the output — because an LLM, like any function, is deterministic in its relationship to input.

Here is what the naive single-turn call looked like:

This returns freeform prose. Every run looks slightly different. There is no contract about output format, no category constraint, no escalation rule. Parsing the result downstream requires heuristics, not code.

The fix is a structured system prompt. It defines the agent's role, the exact output format, and what to do when the ticket doesn't fit cleanly into any category:

Now call it with both a system parameter and a user message:

The output is now a dict with a guaranteed schema. The contract is explicit. Every downstream step can rely on result["category"] existing and being one of the four allowed values.

Reflect before continuing: What would you add to TRIAGE_SYSTEM_PROMPT to handle a ticket that contains personally identifiable information — a phone number or account ID in the body? Consider the trade-off between specificity and prompt length.

Once the system prompt was solid, the next failure mode hit: multi-turn conversations. The triage agent handled follow-up messages by appending each turn to a growing message list and passing the full list to the model. This worked until it didn't. When a conversation hit around eight turns with long ticket descriptions, the combined history grew past 50,000 tokens and the model started losing track of earlier context.

Think of a context window like a whiteboard in a room. You can write everything on it, but eventually you run out of space and have to erase something. The question is not whether to erase — it is what to erase first. Relevance and recency are the two criteria. The most recent exchange is almost always relevant. A confirmation turn from six steps ago — "I understand your issue is X" — is a candidate for trimming because its content is recoverable from surrounding turns.

Mid-Module Checkpoint

Before seeing the sliding-window code, predict: if you drop the oldest turn from a 10-turn conversation to make room, what information have you permanently lost? Consider a conversation where turn 1 contained the original error message that all subsequent turns reference. What would the model do without it?

Here is a minimal sliding-window context manager. It keeps at most max_turns turns and drops the oldest when the count exceeds that limit:

Using it with the structured triage call:

The sliding window is a blunt instrument. It keeps the last N turns regardless of their content. For most support conversations that stay under eight or ten turns, this is enough. When conversations run longer — or when turn 1 carries information that all later turns depend on — you need a smarter approach: pin the first turn and trim from the middle.

Two kinds of errors broke the triage agent in production. The first: the model returning something that wasn't valid JSON — usually when the ticket text was unusually long and the model spent its token budget on analysis instead of the structured output. The second: a max_tokens truncation mid-response. The output cut off at an opening brace, the JSON parser threw, and the agent crashed with no fallback.

Neither failure was the model's fault. Both were mine. I had assumed the model always returns well-formed JSON and that max_tokens=512 was always enough. Factor 10 means assuming the opposite: treat every LLM call as if it could return garbage, and have a plan for each failure mode that is specific, auditable, and testable.

Here is the error-hardened triage function:

The fallback escalates to a human agent and does not update the context window. The next turn starts from the last known-good state rather than appending a broken exchange.

Putting it together — the full multi-turn triage session:

Reflect before continuing: This fallback always escalates to a human. In a high-volume queue where 15% of tickets have unusually long descriptions that trigger parse errors, what does "always escalate on failure" do to the human review queue? What would be a better default — and how would you decide when to try again vs. when to send to a human?