Introduce Router

Problem

A single prompt or agent handles several topics, carrying the rules and tools of all of them at once. These are the Routing System Prompt, Tool Sprawl, and Mega Agent smells.

Think of a mobile carrier’s virtual assistant. It started with sales. Then it got tech support, then billing. Each topic brought a block of rules to the system prompt and a few new tools. Today, when someone asks “why is my bill higher this month?”, the model also reads the number porting rules and the instructions for restarting a modem.

The symptoms build up slowly. A billing customer gets a plan offer in the middle of an explanation. A fix to the support rules makes sales answers worse. And nobody can tell, looking at the trace, which part of the prompt the model followed.

Solution

Classify the input first, with a short step and structured output, and send each type to a specialized prompt or agent that holds only that topic’s rules and tools. The routing decision lives in code.

Before and after

-SYSTEM = """You are the Carrier's assistant.
-IF sales: [800 tokens]
-IF tech support: [1,200 tokens]
-IF billing: [900 tokens]"""
-reply = agent(SYSTEM, tools=ALL_TOOLS, message=msg)
+class Route(BaseModel):
+    topic: Literal["sales", "tech", "billing", "other"]
+    confidence: float
+
+ROUTES = {
+    "sales":   Agent(SALES_PROMPT,   tools=[plans, offers]),
+    "tech":    Agent(TECH_PROMPT,    tools=[device_status, open_ticket]),
+    "billing": Agent(BILLING_PROMPT, tools=[get_invoice, explain_charge]),
+}
+
+route = llm_parse(ROUTER_PROMPT, msg, schema=Route, model="small")
+if route.topic == "other" or route.confidence < 0.7:
+    return handoff_to_human(msg)
+reply = ROUTES[route.topic].run(msg)

Before: one prompt and every tool for everything.

flowchart LR
  IN(["1. Message"]):::io --> A(["2. Single prompt with three topics"]):::llm
  A --> T["Every tool"]:::tool
  A --> OUT(["3. Reply"]):::io

After: classification, a decision in code, and specialists.

flowchart LR
  IN(["1. Message"]):::io --> R(["2. Classify topic"]):::llm
  R --> D{"3. Route"}
  D -->|sales| S(["4a. Sales"]):::llm
  D -->|support| T(["4b. Tech support"]):::llm
  D -->|billing| B(["4c. Billing"]):::llm
  D -->|other| H((("4d. Human agent"))):::human

1. The message arrives.
2. A small model classifies the topic, with a confidence.
3. Code picks the route.
4. Each route has its own prompt and tools; everything else goes to a person.

Worked example

Follow a real message through both designs: “My September bill came out 40 reais higher, what is this?”

Before, the model receives about 3,000 tokens of instructions (the three blocks plus the header) and six tools. It has to figure out on its own that only the billing block matters.

After, the classifier receives a short prompt with the list of topics and a few examples, and returns this:

{ "topic": "billing", "confidence": 0.93 }

The code reads topic, sees that confidence clears the threshold, and calls the billing agent. That agent receives only the 900 billing tokens and two tools: get_invoice and explain_charge. It cannot offer a plan, because it does not even know the offers tool exists.

In the trace, the conversation gains a new field: route=billing. When an answer goes wrong, you can tell whether the classification failed or the specialist did.

Check +20 XP on first tryAfter adding the router, a billing conversation ended with a wrong answer about the cancellation fee. The trace shows route=billing and confidence 0.95. Where do you look first?

At the classifier, because it is the single point of failure

At the confidence threshold, which should be higher

At the billing agent's prompt or tools

Why it works

A prompt with three topics forces the model to decide, on every message, which instructions apply. Sometimes it mixes them. Separating the decision (classify) from the execution (answer) turns an implicit, invisible choice into an explicit one that is recorded and testable.

Classifying is an easier task than answering. The output is one of four words, and a small model usually handles it. That is why the extra step is cheap and fast.

Each specialist stays small: only its topic’s rules and tools. That improves quality, lowers the cost per call, and lets different teams own different routes without stepping on each other. A change to the sales prompt can no longer break billing.

The target pattern is Routing. When only the tool set changes per route, and the prompt can stay the same, the target is the Tool Router. Anthropic’s “Building effective agents” post (December 2024) describes routing as one of the basic workflows, and its example is exactly customer service, with general questions, refund requests, and technical support going down different paths.

Safe mechanics

0. Measure. Run the eval per topic, with real conversations, and record each topic’s quality.
1. Define the routes from real messages grouped by hand.
2. Write the classifier with structured output (route and confidence) and an eval of its own.
3. Split the old prompt into per-route prompts, each with its section and its tools.
4. Implement the decision in code, with a confidence threshold and a default route (human or a generic reply).
5. Record the route in each conversation’s trace.
6. Measure again, per topic and overall.

The classifier eval

The step 2 eval is simple to build. Take a few hundred real messages, label each with the correct route, and compare against the classifier’s output. The most useful result is a confusion matrix: how many billing messages went to support, how many support messages went to sales, and so on.

The matrix shows where routes blur together. If billing and support get mixed up a lot, the boundary between them may be badly drawn, and the fix is in the list of routes, not in the prompt.

Common mistake: trusting the confidence number

The confidence the model writes in the JSON is its guess about itself, not a calibrated probability. A model may write 0.9 for almost everything. Do not pick the 0.7 threshold by gut feeling.

Use the eval to choose. See in which confidence band the errors cluster and put the cut there. If the number does not separate hits from misses, it is useless as a threshold, and an “other” route with good examples in the prompt does more.

Risks

• Messages with two topics. “I paid the bill and my phone won’t turn on.” Define the rule: the most urgent topic, or answer one after the other.
• Classifier as single point of failure. If it is wrong, the whole answer is wrong. The classifier eval is mandatory.
• Topic change mid-conversation. Reclassify on every message, or when the specialist signals the topic changed.
• Extra latency. One more call per message. Use a small model, and consider code rules for obvious cases (a message that only contains a bank slip barcode is billing).
• Too many routes. Twenty routes with thin boundaries confuse the classifier as much as the single prompt confused the model. Start with few and split only when the eval shows the need.

Check +20 XP on first tryThe team wants a new 'number porting' route, separate from 'sales'. In the classifier eval, 30% of porting messages go to sales. What is the right reading?

The boundary between the two routes is unclear, and porting may belong as a rule inside sales

The classifier needs a bigger model

Just raise the confidence threshold to 0.9

Challenge

An internal IT agent answers questions, resets passwords, and grants access to systems. How would you split the routes, and which one needs extra care?

Show solution

Three routes: questions (RAG over the IT knowledge base, no tools with side effects), password reset (a fixed flow in code, with identity verification), and access requests (an agent with a grant-access tool, always with manager approval). The access route needs extra care: it has side effects on permissions, so it goes through Human-in-the-Loop and tools with minimal scope. Low-confidence messages go to a human agent.

Notice that the password reset route does not even need a model after classification. A fixed flow is safer and more predictable than an agent for a task that always has the same steps.

Check your understanding

Check +20 XP on first tryWhat is the main difference between putting 'IF billing, do X' in the system prompt and using a router?

The router uses a smarter model to decide

With a router, the decision becomes an explicit value that code uses, records, and tests

The router removes the need for per-topic evals

Check +20 XP on first tryAn agent has 25 tools and one system prompt. The routes would share the same prompt and change only the available tools. Which target makes more sense?

Full routing, with one specialized agent per route

Keep all 25 tools and improve the descriptions

A Tool Router that picks the tool subset per message

Check +20 XP on first tryA customer writes: 'I paid the bill yesterday and they still cut my internet'. The classifier returns billing with 0.55 confidence. The threshold is 0.7. What happens in this page's design, and is it right?

It goes to a human agent, and that is right: the message mixes billing and support

It goes to billing, because that topic has the highest confidence

It goes to billing and support at the same time