Model Cascade

Intent

Start with the cheapest model and move up to a larger one only when needed. It is for high-volume workloads where most inputs are easy and only a fraction needs the most capable model.

Problem

Bianca runs ticket triage at a software company. There are tens of thousands of tickets a day, and each needs a queue, a priority, and a product. The system uses the largest available model on all of them, because in tests it made fewer mistakes.

The monthly bill passed the whole department’s budget. When Bianca looked at the tickets, she saw that most were trivial: “I forgot my password”, “I need a copy of my invoice”, “the app won’t open”. Any small model gets those right. A small fraction was hard: long tickets, with several topics, or ambiguous.

A hard ticket looks like this (anonymized):

I paid the invoice via Pix yesterday, the receipt is attached, but the
system locked our access and now the whole team is stuck on close.
The invoice tax ID is the Campinas branch.

Pix is Brazil’s instant payment system. The text mixes billing, access, and a production outage. Sending it to the “access” queue at low priority is an expensive miss. Sending the password ticket to the same model that untangles this case is another miss, only on the bill.

Switching everything to the small model would drop quality exactly on the hard tickets, the ones that matter most. Keeping everything on the large model pays a premium for the easy ones.

The question this pattern answers: how do you pay the large model’s price only on the inputs that need it?

Solution

Arrange models in tiers, from cheapest to most expensive. Each input starts at the first tier. After each answer, a criterion decides whether it is good enough to go out or whether the input moves up a tier.

The criterion is the hard part and the most important one. Without it, you only have two models and a toll booth. Options, from most to least reliable:

• Verification in code: the output passes the schema, the business validation, the test. A tax ID with a bad check digit, an unknown queue, a p1 priority on a feature request: code rejects it, and the input moves up.
• Agreement: two cheap attempts agree; if they disagree, move up. It pairs with Voting.
• Stated confidence: the model returns a confidence with its answer. Useful, but poorly calibrated unless measured. If the small model writes 0.92 both when it is right and when it is wrong, the number filters nothing.
• Difficulty classifier: a prior step picks the starting tier. Then the pattern gets close to Routing.

The practical difference: routing picks the path before the first attempt; a cascade tries the cheap model and moves up if the attempt will not do. The two combine. A cheap rule (ticket over 300 words, mention of Pix and a lock at the same time) can skip the first tier. Everything else enters the cascade.

The FrugalGPT paper (Chen, Zaharia, and Zou, 2023) documents the same idea with models from different APIs: try the cheap one and move up only when the answer misses a quality threshold. The lesson that matters is not the paper’s name. The gain comes from the share that does not move up.

Structure

flowchart LR
  IN(["1. Input"]):::io --> S(["2. Small model"]):::llm
  S --> C1{"3. Passed the criterion?"}
  C1 -->|yes| OUT(["6. Answer"]):::io
  C1 -->|no| M(["4. Medium or large model"]):::llm
  M --> C2{"5. Passed?"}
  C2 -->|yes| OUT
  C2 -->|no| H((("Human review"))):::human

1. The input arrives with no assumption about difficulty.
2. The cheapest model tries first.
3. A criterion, ideally in code, decides whether the answer is acceptable.
4. If not, the same input goes to the next tier, sometimes with the previous attempt as context.
5. The last tier has its own criterion.
6. The answer goes out with the tier recorded on the trace. Whatever no tier resolves goes to a person.

Worked example

Two tickets, the same prompt, the same valid(). The JSON below is illustrative.

Ticket A. “I forgot my password for the store app.”

The small model returns:

{ "queue": "access", "priority": "p3", "product": "store-app", "confidence": 0.94 }

valid() passes: known product, known queue, no feature request. Calibrated confidence on this tier, for the band above 0.85, was accurate enough in the eval. Triage is done. One cheap call.

Ticket B. The Pix text, the lock, and the stuck team.

The small model returns:

{ "queue": "access", "priority": "p2", "product": "store-app", "confidence": 0.88 }

The schema passes. Confidence clears 0.85. If that is the whole criterion, the ticket goes to access at medium priority, and billing never sees it. Bianca’s rule in code is different: if the text talks about payment and about a lock, the queue cannot be access alone. valid() fails. The ticket moves up.

The large model returns:

{ "queue": "billing", "priority": "p1", "product": "store-app", "confidence": 0.81 }

Billing queue, production-outage priority. The large tier’s threshold is lower (0.60 here), because this model is already the last automatic step. If valid() passes, it goes out. If it fails again, it goes to the human queue with both attempts on the trace.

Compare the four ways to handle Ticket B:

The cascade is only better than the small model alone if the criterion catches Ticket B. Otherwise you shipped cheap mistakes with a nicer name.

Pseudocode

schema Triage { queue: string, priority: "p1" | "p2" | "p3", product: string, confidence: float }

TIERS = [
    { model: "small", min_confidence: 0.85 },
    { model: "large", min_confidence: 0.60 },
]

function triage(ticket):
    for tier in TIERS:
        result = llm(TRIAGE_PROMPT, ticket, model=tier.model) -> Triage     # 2 and 4
        trace.span("cascade." + tier.model, result)
        # 3 and 5. criterion: validation in code + calibrated confidence
        if valid(result) and calibrated(result.confidence, tier) >= tier.min_confidence:
            return result                                                   # 6
    return human.queue(ticket)

Real-world scenarios

• Automatic ticket triage. The small model handles short, clear tickets; long, ambiguous, or low-confidence ones move up.
• Call center voice assistant. Quick answers come from a small model to keep latency low; complex questions move up with a holding phrase (“let me check that”).
• Content moderation. A cheap classifier approves clearly safe content; doubtful content goes to a larger model and, at the end, to moderators.
• Support bot with human escalation. Frequent questions are answered on the cheap tier; multi-step cases move up.

Applicability

• High volume, where cost per call weighs on the bill.
• Uneven difficulty: most inputs are easy.
• There is a quality criterion you can check after the answer.
• The extra latency of moving up is acceptable for the fraction that does.

When not to use

• Low volume. The complexity does not pay off.
• There is no reliable criterion. Without knowing when the cheap answer is wrong, the cascade delivers cheap mistakes.
• Before building the cascade, measure the simpler alternative: the most capable model at lower effort. Often it matches on cost and quality with a single model to maintain, and the prompt cache is not split across models.
• Latency is critical for every input. Whatever moves up pays for two calls.

Common mistake: cascade as a toll booth

A team puts the small model in front, escalates on any doubt, and almost everything goes to the large one. On the trace, cascade.tier shows up on every span. On the bill, cost got worse: an extra call on almost everything, plus the large model.

That is not a cascade. It is the large model with a toll booth in front. The metric that gives it away: share that moves up. If it clears two thirds of your volume, go back to the eval and loosen the criterion, or drop the cascade.

How to implement

1. Run the eval on each candidate model and measure accuracy and cost per category. The overall average hides Ticket A and Ticket B.
2. Also measure the same capable model at lower reasoning effort. If it ties the cascade on quality and cost, keep one model.
3. Pick the escalation criterion and measure its calibration: when the small model says 90% confidence, is it right 90% of the time?
4. Set each tier’s thresholds from the calibration, not from intuition. Never pick 0.7 because it “feels reasonable”.
5. Simulate the cascade on the dataset: share that moves up, final accuracy, total cost, and the errors the cheap tier would approve.
6. Implement it with the tier recorded on every trace, and a final destination for whatever no tier resolves.
7. Recalibrate when you switch models or when the input distribution changes.

How to calibrate the threshold

Split the cheap-tier outputs into confidence bands, for example below 0.6, 0.6 to 0.8, above 0.8. In each band, measure real accuracy on the eval.

The useful threshold is the point where the band starts being accurate enough for the product. If “above 0.8” scores high and “0.6 to 0.8” scores poorly, a cut near 0.8 makes sense. If every band scores about the same, confidence carries no information. Then the criterion has to be something else: text length, presence of a tax ID, a business rule that failed.

Suppose 10,000 tickets a day and that the large model costs about 10 times the small one per call. If 80% stay on the small model, daily cost falls to about 28% of “large on everything”, and you still pay the large model on the hard 20%. These numbers are illustrative. Run the arithmetic with your provider’s price and your volume.

The refactoring that lands on this pattern is Introduce Model Cascade.

Variations

• Cascade with verification. The criterion is a check in code (schema, business rule). The most reliable option.
• Cascade by agreement. Two answers from the cheap model; if they diverge, move up. It pairs with Voting.
• Difficulty routing. A classifier (or a cheap rule) picks the tier before the first attempt. It avoids the wasted call, but sometimes picks the wrong tier. Use it when the eval shows a cheap, stable signal: text length, an attachment, mention of Pix and a lock.
• Effort cascade. Same model, increasing reasoning effort. Keeps a single model and the same cache. It pays off when the simple-alternative eval almost tied, and a little more effort on the cases that fail the criterion closes the gap.

Cost, latency, and failure modes

Whatever moves up pays for the prompt twice. Different models do not share prompt cache, so the second call does not reuse the prefix the first one already processed. If 25% move up, those 25% cost small + large, not large alone. Put that in the simulation.

How to evaluate

• Final accuracy of the cascade compared with the large model alone and with the capable model at low effort.
• Average cost per input and share per tier.
• Calibration of the criterion: accuracy curve by confidence band on the cheap tier.
• Hidden errors: cases the cheap tier approved with high confidence that were wrong.

{ "ticket": "I paid the invoice but the system locked our access and now the whole team is stuck", "expected": { "queue": "billing", "priority": "p1" }, "note": "should move up a tier" }

Build the dataset with both types. If it only has easy tickets, the cascade always “wins”. If it only has Ticket B, everything moves up and you only added latency. The dataset mix needs to look like production, or the thresholds lie.

In production

• A cascade.tier attribute on every trace, with a dashboard of share per tier.
• An alert when the share moving up changes a lot from one day to the next.
• A weekly sample of high-confidence cheap-tier answers for review, to catch confident errors.
• Threshold review on every model change.
• A visible final destination: a human queue with the previous attempts, not a silent drop.

The smell this pattern treats, in its high-volume form, is No Cost Budget. A cascade does not replace a per-run cap. It lowers average cost. The cap still exists for the run that goes off the rails.

Pros and cons

Code

Both tabs try the small model, check validation and confidence, and move up when needed.

from typing import Literal

import anthropic
from pydantic import BaseModel

client = anthropic.Anthropic()


class Triage(BaseModel):
    queue: Literal["billing", "access", "bug", "feature", "other"]
    priority: Literal["p1", "p2", "p3"]
    product: str
    confidence: float


TIERS = [  # thresholds from eval calibration, not intuition
    ("claude-haiku-4-5", 0.85),
    ("claude-opus-5-5", 0.60),
]

PROMPT = "Classify the ticket into queue, priority, and product. Give a confidence from 0 to 1."


def valid(t: Triage) -> bool:  # business rules in code
    return t.product in KNOWN_PRODUCTS and not (t.queue == "feature" and t.priority == "p1")


def triage(ticket: str) -> Triage | None:
    for model, min_conf in TIERS:
        t = client.messages.parse(
            model=model, max_tokens=300, system=PROMPT,
            messages=[{"role": "user", "content": ticket}], output_format=Triage,
        ).parsed_output
        trace_tier(model, t)  # record the tier on the trace
        if valid(t) and t.confidence >= min_conf:
            return t
    return None  # human review queue

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();

const Triage = z.object({
  queue: z.enum(["billing", "access", "bug", "feature", "other"]),
  priority: z.enum(["p1", "p2", "p3"]),
  product: z.string(),
  confidence: z.number(),
});
type Triage = z.infer<typeof Triage>;

const TIERS: [string, number][] = [
  ["claude-haiku-4-5", 0.85],
  ["claude-opus-5-5", 0.6],
];

const valid = (t: Triage) => KNOWN_PRODUCTS.has(t.product) && !(t.queue === "feature" && t.priority === "p1");

export async function triage(ticket: string): Promise<Triage | null> {
  for (const [model, minConf] of TIERS) {
    const { parsed_output: t } = await client.messages.parse({
      model,
      max_tokens: 300,
      system: "Classify the ticket into queue, priority, and product. Give a confidence from 0 to 1.",
      messages: [{ role: "user", content: ticket }],
      output_config: { format: zodOutputFormat(Triage) },
    });
    if (t && valid(t) && t.confidence >= minConf) return t; // 3 and 5
  }
  return null; // human review
}

Check your understanding