Voting

Intent

Run the same task several times, or with several models or prompts, and decide by majority or threshold. It is for classification decisions where an error is expensive and a single model answer is not reliable enough.

Problem

Natália runs comment moderation for an online course network. A classifier decides whether each comment violates the harassment policy. On clear cases, it gets it right. On borderline cases (irony, slang, someone quoting an insult to criticize it), the same message is sometimes approved and sometimes removed, depending on the run.

Removing a legitimate comment brings complaints and appeals. Leaving harassment up brings something much worse: an exposed student and a press story. The team wants two things. A stable decision, and the genuinely doubtful cases reaching a person instead of being settled by a coin flip.

Natália rewrote the prompt three times. Each version improved the clear cases. On the ambiguous ones, a single call keeps flipping, because the ambiguity lives in the comment, not in the prompt.

The question this pattern answers: how do you make a decision more reliable when a single model answer varies?

Solution

Ask for the same decision several times, independently, and combine the votes in code. The diversity of the votes can come from three sources:

• Same setup, several samples. The model’s natural variation becomes information. It only works with temperature above zero.
• Different prompts. Each one looks at the case from an angle: the literal text, the author’s intent, the conversation’s context.
• Different models. One model’s errors tend not to coincide with another’s, because they were trained differently.

The combination is an explicit rule: simple majority, unanimity for serious actions, or a threshold (“remove if 2 of 3 vote to remove”). Disagreement among votes is the pattern’s most useful signal. It tells you the case is hard, and hard cases go to human review.

Anthropic’s “Building effective agents” post (December 2024) describes voting as one of two forms of parallelization, next to Sectioning. The post’s examples match this pattern: several prompts reviewing code for vulnerabilities, and several prompts judging whether content is inappropriate, with different thresholds to balance false positives and false negatives.

Why the majority is wrong less often

An illustrative calculation shows the mechanism. Suppose each vote is right 90% of the time and the errors are independent. A majority of three is only wrong when two or three votes fail together. That happens in about 2.8% of cases, so the majority is right about 97% of the time.

The word holding the calculation up is “independent”. If all three votes use the same model, the same prompt, and temperature zero, they are the same opinion repeated three times. They fail together, and the majority stays at 90%. You paid three times for one answer.

In practice errors are never fully independent. Every vote reads the same comment and trips on the same irony. So the real gain is smaller than the calculation, and only the eval tells you how much it is worth in your case.

Worked example

A student writes on the forum:

Someone called me "stupid" yesterday in the class group chat. That is no way to treat a classmate.

The three votes, each with structured output:

[
  { "voter": "literal", "decision": "remove", "policy": "harassment", "rationale": "contains the insult 'stupid'" },
  { "voter": "intent",  "decision": "allow",  "policy": "harassment", "rationale": "the author reports an insult, attacks no one" },
  { "voter": "context", "decision": "allow",  "policy": "harassment", "rationale": "the thread is about how classmates treat each other" }
]

With a simple majority, the comment stays up. With Natália’s rule (remove only with unanimity), it also stays, and the 1 to 2 split sends the case to the review queue with all three rationales. The moderator decides in seconds, because she already has each side’s argument.

Notice that the rule changed the outcome more than the prompt did. Choosing the rule is choosing which error you would rather make.

Structure

flowchart LR
  IN(["1. Comment"]):::io --> V1(["2. Vote A"]):::llm
  IN --> V2(["2. Vote B"]):::llm
  IN --> V3(["2. Vote C"]):::llm
  V1 --> AG{"3. Combine votes"}
  V2 --> AG
  V3 --> AG
  AG -->|"consensus"| OUT(["4. Automatic decision"]):::io
  AG -->|"split"| H((("5. Human review"))):::human

1. The input arrives, with the context the votes need.
2. Three independent votes run in parallel, each with structured output.
3. Code combines the votes with an explicit rule. The model does not pick the rule.
4. With consensus (or above the threshold), the decision is automatic.
5. With a split, the case goes to a person, with the votes and the rationales.

Pseudocode

schema Vote { decision: "allow" | "remove", policy: string, rationale: string }

VOTERS = [
    { prompt: LITERAL_PROMPT, model: "medium" },
    { prompt: INTENT_PROMPT,  model: "medium" },
    { prompt: CONTEXT_PROMPT, model: "other_provider" },
]

function moderate(comment, thread):
    parallel {                                                   # 2
        votes = [llm(v.prompt, comment, thread, model=v.model) -> Vote for v in VOTERS]
    }
    removes = count(v.decision == "remove" for v in votes)       # 3
    trace.span("votes", votes)

    if removes == len(votes): return remove(comment, votes)        # 4. unanimity
    if removes == 0:           return allow(comment)
    return human.review(comment, votes)                            # 5. split

Real-world scenarios

• Content moderation. Automatic removals only with unanimity. Split cases go to moderators, with all three rationales side by side.
• Resume screening with bias control. Three independent evaluations with the same rubric. A large gap between scores leads to human review, because that is where one evaluator’s bias tends to show.
• Automatic ticket triage. A cheap classifier decides the clear tickets alone. Voting runs only on the ones it marked as uncertain, so you do not pay three times on everything. The routing itself follows the Routing pattern.

Applicability

• Classifications or decisions with a discrete answer: approve, remove, a category from a closed list.
• Expensive errors, where stability matters more than the extra cost.
• Frequent ambiguous cases, where a single call flips between runs.
• There is a human path for split cases, with capacity for the expected volume.

When not to use

• Open tasks, such as writing text. You cannot count votes across three different drafts. To pick the best one you need a judge with criteria, as in Evaluator-Optimizer.
• Answers with calculation or long reasoning. Generating several reasoning paths and keeping the most frequent final answer is Self-Consistency (Wang et al., 2022). The mechanics are similar, but the vote is on the result, not on the path.
• High volume with almost always clear cases. Use a Model Cascade and vote only on the uncertain ones.
• The problem is a bad prompt. Three bad votes do not make a good one. If every vote misses the same case, fix the prompt or the examples first.
• Nobody reviews the split cases. Without that path, a split becomes a lottery at three times the cost.

Common mistake: voting in free text

Beginners often ask “answer whether the comment should be removed” and count votes by looking for the word “yes” in the reply. One vote says “Yes, with caveats”, another says “I would remove it”, another “I see no serious problem”. Counting turns into interpretation.

Each vote needs Structured Output with closed values. The rationale goes in a separate field, for the person reviewing, and never enters the count.

How to implement

1. Define the decision as structured output with closed values and a short rationale.
2. Choose the source of diversity: samples with temperature above zero, prompts with different angles, or different models.
3. Write the combination rule from each error’s cost. For example: unanimity to remove, majority to approve.
4. Run the votes in parallel, so latency is that of the slowest vote and not the sum.
5. Send split cases to human review, with the votes and rationales.
6. Measure in the eval the combination’s accuracy, the split rate, and each vote’s accuracy alone.
7. If volume is high, vote only on uncertain cases or use adaptive voting.

Variations

• Simple majority. An odd number of votes, decided by majority. Good for categories with no serious action.
• Unanimity for serious actions. Remove, reject, or block only when everyone agrees. Trades false positives for more human review.
• Weighted vote. Votes from more reliable models weigh more. The weights come from the eval, never from a hunch.
• Adaptive voting. Start with two votes and call the third only if they diverge. Saves money on clear cases.

An illustrative calculation for adaptive voting: suppose 100,000 comments a day, and the first two votes agree on 85% of them. You pay 200,000 calls for the first two votes and 15,000 more for the third. That is 215,000 calls, against 300,000 with three votes every time.

Cost, latency, and failure modes

How to evaluate

• Accuracy of the combined decision versus each vote alone. If the combination does not beat the best single vote, voting is not paying for itself.
• Split rate and accuracy within splits. Splits with low accuracy are genuinely hard cases, and it is good that they reach a person.
• Agreement among votes across the dataset. Too high suggests little diversity; too low suggests prompts measuring different things.

An eval case for sarcasm:

{ "comment": "This teacher is a genius... at being terrible", "thread": "fixtures/thread-88.json", "expected": "allow", "note": "sarcasm, no harassment" }

Build the dataset from the cases that flip today. Run the old classifier five times on each comment in a sample and keep the ones whose answer changed: those are exactly the cases voting needs to improve.

In production

• One span per vote with the decision and rationale, and a combination span with the rule applied.
• A dashboard of unanimity and split rates per policy.
• An alert when the split rate exceeds the review queue’s capacity.
• A review queue for split cases, with response time.
• Human decisions on split cases become new eval cases.

Pros and cons

Code

Both tabs run three votes and decide by unanimity, sending splits to review.

import asyncio
from typing import Literal

import anthropic
from pydantic import BaseModel

client = anthropic.AsyncAnthropic()


class Vote(BaseModel):
    decision: Literal["allow", "remove"]
    policy: str
    rationale: str


VOTERS = [  # diversity of angle and model
    ("claude-sonnet-5-5", "Judge only the literal text against the harassment policy."),
    ("claude-sonnet-5-5", "Judge the author's intent: an attack on a person or criticism of ideas?"),
    ("claude-haiku-4-5", "Judge the comment in the conversation's context against the harassment policy."),
]


async def vote(model: str, angle: str, comment: str, thread: str) -> Vote:
    reply = await client.messages.parse(
        model=model, max_tokens=500,
        system=f"You moderate comments on a course platform. {angle}",
        messages=[{"role": "user", "content": f"<thread>{thread}</thread>\n<comment>{comment}</comment>"}],
        output_format=Vote,
    )
    return reply.parsed_output


async def moderate(comment: str, thread: str) -> dict:
    votes = await asyncio.gather(*(vote(m, a, comment, thread) for m, a in VOTERS))  # 2
    removes = sum(v.decision == "remove" for v in votes)  # 3
    if removes == len(votes):
        return {"action": "remove", "votes": votes}  # 4. unanimity
    if removes == 0:
        return {"action": "allow", "votes": votes}
    return {"action": "human_review", "votes": votes}  # 5

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();
const Vote = z.object({ decision: z.enum(["allow", "remove"]), policy: z.string(), rationale: z.string() });
type Vote = z.infer<typeof Vote>;

const VOTERS: [string, string][] = [
  ["claude-sonnet-5-5", "Judge only the literal text against the harassment policy."],
  ["claude-sonnet-5-5", "Judge the author's intent: an attack on a person or criticism of ideas?"],
  ["claude-haiku-4-5", "Judge the comment in the conversation's context against the harassment policy."],
];

async function vote([model, angle]: [string, string], comment: string, thread: string): Promise<Vote> {
  const { parsed_output } = await client.messages.parse({
    model, max_tokens: 500,
    system: `You moderate comments on a course platform. ${angle}`,
    messages: [{ role: "user", content: `<thread>${thread}</thread>\n<comment>${comment}</comment>` }],
    output_config: { format: zodOutputFormat(Vote) },
  });
  if (!parsed_output) throw new Error("invalid vote");
  return parsed_output;
}

export async function moderate(comment: string, thread: string) {
  // adaptive voting: two votes first
  const first = await Promise.all(VOTERS.slice(0, 2).map((v) => vote(v, comment, thread)));
  if (first[0].decision === "allow" && first[1].decision === "allow") return { action: "allow", votes: first };
  const votes = [...first, await vote(VOTERS[2], comment, thread)]; // third only if needed
  const removes = votes.filter((v) => v.decision === "remove").length;
  if (removes === votes.length) return { action: "remove", votes };
  return { action: "human_review", votes };
}

Check your understanding