Tool Router

Intent

Pick a small subset of tools before calling the main model. It is for agents that cover many topics and need many tools in total, but few in each task.

Problem

Fábio runs the back-office assistant at a distributor. The agent reconciles payments, issues invoices, checks stock, opens tickets with carriers, and answers commission questions. After merging duplicate tools, 34 remained, all legitimate and all different.

The agent still makes mistakes. On a reconciliation request, it calls check_stock instead of find_payment, because both take an order code and their descriptions start the same way. The trace shows the model hesitating between tools that should not even be on the table.

Then there is the bill. Every call carries all 34 definitions, so every task’s context starts with thousands of tokens of tools that have nothing to do with it. In a six-turn loop, those definitions go to the model six times.

Fábio cannot remove tools: someone uses every one of them. What he can change is how many the model sees at a time.

The question this pattern answers: how do you keep many tools in the system without making the model choose among all of them on every call?

Solution

Organize tools into groups by topic (reconciliation, invoices, stock, logistics). Before the agent loop, a short step decides which groups the task needs. The main agent runs with only those groups.

The router can be a small model with structured output, a similarity search between the task and the group descriptions, or plain rules when the channel already tells you the topic. There is always a way back: if the agent finds it needs another group, it can ask for it.

The key move is taking a choice away from the big model and putting it in a small, cheap, testable step. The main agent stops answering “which of these 34?” and starts answering “which of these 8?”.

Worked example

A task comes in: “Reconcile the payment for order 88231 with yesterday’s statement and, if it matches, settle the bill”. The router gets the task and four description lines, one per group:

billing:   Use to reconcile payments against statements and settle bills.
invoices:  Use to issue, cancel, or look up invoices (NF-e).
stock:     Use for stock levels, reservations, and movements.
logistics: Use for freight, pickups, and carrier tickets.

It returns a short structured output:

{ "groups": ["billing"], "confidence": 0.92 }

The main agent runs with the 7 billing tools plus request_group. The stock tool that caused the confusion never shows up. (NF-e is Brazil’s mandatory electronic invoice; the distributor issues one for every sale.)

Now the math, with illustrative numbers. Suppose each tool definition takes about 250 tokens. The 34 add up to roughly 8,500 tokens per call; over a six-turn loop, that is about 51,000 tokens of definitions alone. With 8 tools, the same task carries about 12,000. The router costs one short call on a small model, with the task text and four group lines.

The exact numbers depend on your catalog and your provider. The shape of the math does not: the router pays once, and the savings happen on every loop turn.

Structure

flowchart LR
  IN(["1. Task"]):::io --> R(["2. Tool router"]):::llm
  C[("3. Group catalog")]:::store --> R
  R --> S["4. Chosen subset"]:::tool
  S --> A(["5. Main agent"]):::llm
  A --> T["6. Group tools"]:::tool
  T --> A
  A -->|"needs another group"| R
  A --> OUT(["7. Answer"]):::io

1. The task arrives as user text or a system event.
2. A small model reads the task and the list of groups, one description line each.
3. The catalog stores the groups and each group’s tools, versioned.
4. The router returns the chosen groups and its confidence.
5. The main agent runs with those groups’ tools plus one tool to ask for another group.
6. Tools execute as usual.
7. The answer goes out and the trace records which groups were used.

Pseudocode

schema Selection { groups: list["billing" | "invoices" | "stock" | "logistics"], confidence: float }

GROUPS = catalog.load("tool-groups@4")

function handle(task):
    # 2. pick groups with a small model
    sel = llm(ROUTER_PROMPT + GROUPS.summaries(), task) -> Selection
    trace.span("tool_route", sel)

    if sel.confidence < 0.6:
        sel.groups = GROUPS.default_set()           # 4. safe default set

    tools = GROUPS.tools_for(sel.groups) + [request_group]
    # 5 and 6. main agent with the subset only
    while budget.check():
        reply = llm_with_tools(AGENT_PROMPT, tools, state)
        if reply.is_final:
            return reply.text                        # 7
        for call in reply.tool_calls:
            if call.name == "request_group":
                tools += GROUPS.tools_for([call.args.group])
                trace.span("tool_route_expand", call.args)
            state.add(execute(call))

Real-world scenarios

• Back-office automation with durable execution. Reconciliation, invoicing, and collections each have their own tools. Each process starts with its topic’s group and asks for another only when needed. What changes: the collections process never sees the tool that cancels invoices.
• Support bot with human escalation. The intent router already picks the route; each route gets only its topic’s tools, and side-effect tools stay out of question-only routes. What changes: a question about delivery dates has no way to trigger a refund.

Applicability

• The agent has more than 15 or 20 tools, all needed, across different topics.
• Each task uses tools from one or two topics.
• Tool definitions take a large share of every call’s context.
• You want to limit which side-effect tools are within reach for each task type.

When not to use

• The agent has few tools. Consolidate first: see Consolidate Tools.
• Tasks cross every topic all the time. The router would always pick everything.
• The provider offers on-demand tool search and your only problem is token cost. That mechanism solves it without an extra step; see also Progressive Disclosure.
• The input channel already defines the topic. Use a fixed rule in code, with no model.

How to implement

1. Group tools by topic, with at most 8 to 10 per group.
2. Write a one-line description per group, focused on when to use it.
3. Define a default set for low confidence: the most common group, or read-only tools.
4. Implement the router with structured output and a small model.
5. Give the agent a request_group tool to ask for a group that was left out.
6. Build an eval for the router alone: task and expected groups.
7. Run the full agent eval and compare it with the version without a router.

Common traps

• Grouping by system, not by task. “Everything that talks to the ERP” becomes a 25-tool group. Group by what the person wants to do: reconcile, issue, ship.
• Describing contents, not use. “Billing tools” does not help the router. “Use to reconcile payments against statements” does, because it matches how tasks arrive.
• A default set with side effects. Low confidence means the router is unsure. The safe default is read-only, not “everything, just in case”.
• Forgetting the escape hatch. Without request_group, a router mistake becomes an agent that answers without the right tool, or makes up the data.

Variations

• Embedding routing. Compare the task with examples or descriptions of each group. No model call, weaker on ambiguous tasks.
• Rule routing. Queue, channel, or event type defines the group. Zero cost, zero classification error, only works when the topic comes with the input.
• Provider tool search. The model sees an index and loads a full definition when needed. It solves the token cost and part of the confusion.
• Agent router. Instead of tool groups, each route leads to a specialized agent. Then the pattern becomes Routing or a handoff between agents.

Cost, latency, and failure modes

The most expensive failure is the silent one: the right group was left out and the agent did not ask. It answers “I could not find the payment” with total calm. That is why the coverage eval below matters more than the size one.

How to evaluate

• Coverage: in how many tasks the chosen groups include every tool the correct solution uses.
• Average size of the chosen subset.
• Expansion rate: how often the agent had to ask for a group.
• Full eval: completion rate and correct tool choice, before and after.

{ "task": "Reconcile the payment for order 88231 with yesterday's statement", "expected_groups": ["billing"] }

Build the dataset from the traces you already have. For each task solved successfully, the expected groups are the groups of the tools the agent actually called. Include cross-topic tasks on purpose (“issue the invoice and book the pickup”) and vague ones (“see what happened with order 88231”).

Coverage and size pull in opposite directions. A router that always picks everything has perfect coverage and is useless. Track both together.

In production

• A tool_route span with groups, confidence, and catalog version.
• A dashboard of request_group frequency per group. A group requested often should join the initial selection for that task type.
• An alert when the average number of groups per task rises, a sign of an unsure router.
• When you add a new tool, rerun the router eval. A changed group description can divert tasks that used to go to the right place.

Security

The router shrinks the surface: side-effect tools only appear on routes that need them. Do not treat this as access control. Malicious text inside the task can talk the router into a different group, and the agent can also ask for groups through request_group.

Real permission lives in the tool and the backend: who the user is, what they can do, what the amount limit is. See Narrow Tool Permissions.

Pros and cons

Relations

• Smell it cures: Tool Sprawl, when every tool is needed but not all at once.
• Refactorings that lead here: Introduce Router and, before it, Consolidate Tools.
• Neighbors: Routing picks the prompt or the agent; the Tool Router picks only the toolbox and keeps one agent. Progressive Disclosure lets the agent itself open more tools gradually, with no up-front step.

Code

The tabs pick groups with a small model and run the agent with only the chosen groups. Python and TypeScript build the list and call the model. LangGraph splits the two pieces: one node picks the groups, the next node creates the agent with only those groups’ tools.

from typing import Literal

import anthropic
from pydantic import BaseModel

client = anthropic.Anthropic()

GROUPS = {  # 3. catalog: group -> (description, tools)
    "billing": ("Use to reconcile payments against statements and settle bills.", BILLING_TOOLS),
    "invoices": ("Use to issue, cancel, or look up invoices.", INVOICE_TOOLS),
    "stock": ("Use for stock levels, reservations, and movements.", STOCK_TOOLS),
    "logistics": ("Use for freight, pickups, and carrier tickets.", LOGISTICS_TOOLS),
}

Group = Literal["billing", "invoices", "stock", "logistics"]


class Selection(BaseModel):
    groups: list[Group]
    confidence: float


def select_tools(task: str) -> list[dict]:
    summary = "\n".join(f"{name}: {desc}" for name, (desc, _) in GROUPS.items())
    sel = client.messages.parse(  # 2. router on a small model
        model="claude-haiku-4-5",
        max_tokens=200,
        system=f"Pick the tool groups this task needs.\n{summary}",
        messages=[{"role": "user", "content": task}],
        output_format=Selection,
    ).parsed_output
    groups = sel.groups if sel.confidence >= 0.6 else ["billing"]  # 4. safe default
    return [tool for g in groups for tool in GROUPS[g][1]]


def handle(task: str) -> str:
    tools = select_tools(task)
    return run_agent(task, tools)  # 5. the same loop as before, with fewer tools

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();

const GROUPS = {
  billing: { desc: "Use to reconcile payments against statements and settle bills.", tools: BILLING_TOOLS },
  invoices: { desc: "Use to issue, cancel, or look up invoices.", tools: INVOICE_TOOLS },
  stock: { desc: "Use for stock levels, reservations, and movements.", tools: STOCK_TOOLS },
  logistics: { desc: "Use for freight, pickups, and carrier tickets.", tools: LOGISTICS_TOOLS },
} as const;

type Group = keyof typeof GROUPS;

const Selection = z.object({
  groups: z.array(z.enum(["billing", "invoices", "stock", "logistics"])),
  confidence: z.number(),
});

export async function selectTools(task: string): Promise<Anthropic.Tool[]> {
  const summary = Object.entries(GROUPS).map(([k, g]) => `${k}: ${g.desc}`).join("\n");
  const { parsed_output: sel } = await client.messages.parse({
    model: "claude-haiku-4-5",
    max_tokens: 200,
    system: `Pick the tool groups this task needs.\n${summary}`,
    messages: [{ role: "user", content: task }],
    output_config: { format: zodOutputFormat(Selection) },
  });
  const groups: Group[] = sel && sel.confidence >= 0.6 ? sel.groups : ["billing"];
  return groups.flatMap((g) => [...GROUPS[g].tools]);
}

from typing import Literal

from langchain.agents import create_agent
from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel
from typing_extensions import TypedDict

Group = Literal["billing", "invoices", "stock", "logistics"]


class Selection(BaseModel):
    groups: list[Group]
    confidence: float


class State(TypedDict):
    task: str
    groups: list[str]
    answer: str


router = init_chat_model("anthropic:claude-haiku-4-5", temperature=0)


def select(state: State) -> dict:
    # 2. the same GROUPS as the Python tab: name -> (description, tools)
    summary = "\n".join(f"{name}: {desc}" for name, (desc, _) in GROUPS.items())
    sel = router.with_structured_output(Selection).invoke(
        [("system", f"Pick the tool groups this task needs.\n{summary}"),
         ("human", state["task"])]
    )
    groups = sel.groups if sel.confidence >= 0.6 else ["billing"]  # 4
    return {"groups": list(groups)}


def run(state: State) -> dict:
    tools = [tool for g in state["groups"] for tool in GROUPS[g][1]]  # 5
    agent = create_agent("anthropic:claude-opus-5-5", tools=tools,
                         system_prompt="You are the back-office assistant.")
    result = agent.invoke({"messages": [("human", state["task"])]}, {"recursion_limit": 16})
    return {"answer": result["messages"][-1].text}


graph = StateGraph(State)
graph.add_node("select", select)
graph.add_node("run", run)
graph.add_edge(START, "select")
graph.add_edge("select", "run")
graph.add_edge("run", END)
app = graph.compile()

Check your understanding