Deep research with citations

The business

A strategy consultancy runs dozens of market studies a month: market size, competitors, regulation, trends. Each takes a junior analyst two to three days, most of it reading sources and organizing notes.

The product the client pays for is a report in which every claim has a source you can open. A number without a source does not go in. A passage that is not on the cited page does not go in. The junior analyst does that work today with ten browser tabs, a notes doc, and a lot of copy and paste.

The goal is a system that takes an open question (“what does the residential solar market look like in northeast Brazil?”) and delivers, in minutes, a report in which every claim has a verifiable source. The analyst then reviews and goes deeper instead of starting from scratch. Nordeste, here, is Brazil’s Northeast: strong sun, state programs, and federal distributed-generation rules that change the return on a rooftop system.

This is not a writing nicety. In Mata v. Avianca (2023), lawyers in the United States filed a brief with case law invented by ChatGPT. The court found out, and the episode became a lesson in fake citation. A market report whose source does not say what the text claims is the same defect, sold by the hour.

What this scenario measures: claims with a valid source, time per report, and cost per report.

Version 1

The innovation team built the first version with an agent framework and five roles. The config looked like a human team:

agents:
  - role: Researcher
    tools: [web_search, fetch_page]
  - role: Analyst
  - role: Writer
  - role: Reviewer
  - role: Editor
flow: sequential

flowchart LR
  Q(["1. Question"]):::io --> P(["2. Researcher"]):::llm
  P --> W["Web search"]:::untrusted
  W --> P
  P --> A(["3. Analyst"]):::llm
  A --> R(["4. Writer"]):::llm
  R --> V(["5. Reviewer"]):::llm
  V --> E(["6. Editor"]):::llm
  E --> OUT(["7. Report"]):::io

1. The question arrives.
2. The researcher runs searches one after another and writes a summary of everything it read.
3. to 6. Analyst, writer, reviewer, and editor rewrite the text in a queue, each reading the previous one.
4. The report goes out.

In the demo, the question was narrow (“what is the ICMS rate in Bahia for distributed generation?”). ICMS is a Brazilian state tax; Bahia is a Northeast state; distributed generation is the rooftop and small-scale regime. The researcher found a page, the other four rewrote, the report came out in a few minutes and sounded professional. Nobody measured a broad question, nobody checked whether the URLs opened, nobody put a cost cap.

Anthropic’s “Building effective agents” (December 2024) recommends starting with the simplest flow that does the job and adding agents only when there is a technical reason. Five job titles in a queue is the opposite of that.

What broke

Week 1. A report took 25 minutes. The researcher ran a search, read, ran another, read. Nothing ran in parallel. “Market size” and “ANEEL regulation” do not depend on each other. ANEEL is Brazil’s electricity regulator. The serial queue added the latencies.

Week 2. Cost per report was several times the expected amount. The trace showed that analyst, writer, and reviewer reread the whole text and returned nearly identical versions. Removing three of the five agents did not change quality in the eval. The editor replied “approved”. You were paying for four rewrites of a single search job.

Week 3. The researcher piled dozens of pages into the same context. By the time the writer got the summary, facts from different sources were mixed together, and citations pointed to the wrong source. A sentence about Lei 14.300/2022 (Brazil’s legal framework for distributed generation) came out linked to a 2020 blog post. The junior analyst would have opened both links. The system opened neither.

Week 4. A broad question (“what about green hydrogen worldwide?”) made the researcher search for over an hour. There was no step or cost limit. The research cost more than the price charged for the report. The client got an incomplete draft, and the provider bill arrived in full.

The four incidents do not ask for a better model. They ask for fewer agents, more structure, and a cap.

Check +20 XP on first tryThe trace shows five agents in a queue. Only the first has a search tool. The other four return the text in different words. Removing three of them does not change the eval. What is the diagnosis and the first cut?

Add a sixth quality agent to catch what the rewriters drop

The researcher is weak: swap in a larger model and keep the five roles

Decorative multi-agent: collapse the rewriters and keep who researches and who writes from checked claims

Smells found

Decorative Multi-Agent

Five job titles, three of them only rewriting. Version 1’s YAML is the passage that shows the smell: company roles, always-sequential flow, tools only on the first agent. It caused week 2.

Serial Latency Stack

Independent searches run in sequence. The researcher only started “competitors” after finishing “market”. It caused week 1. Parallelism here is not premature optimization: the sub-questions were already independent in the client’s brief.

Shared Scratchpad

Every source in the same text, with no structure. The researcher wrote a prose summary. The writer cited from memory of that summary. It caused the wrong citations in week 3. Mix pages into one paragraph and you lose the owner of each sentence.

No Cost Budget

No limit per study. It caused week 4. A report has a price. A run with no cap is not “thorough”: it is a loop the card stops.

Refactorings applied

1. Add Loop Budget

The cap first: steps, time, and cost per report, shared by every agent. At the limit, the report goes out marked as partial, with the gaps listed. Stopping with a silent exception leaves the analyst with no text and finance with no cap.

The budget is one number for the tree. A cap of 20 steps per subagent, with three subagents that spawn three more, is not a cap. It is the neighboring smell of cascading subagents.

2. Collapse Agents

Analyst, reviewer, and editor went away. An orchestrator that plans and writes remained, and source review became a check in code. Anyone with no tool who only rewrites prose does not justify a call.

3. Extract Subagent

Each sub-question is investigated by a subagent with its own context, which returns structured claims with a source for each. Pages read stay with that subagent. The orchestrator does not receive HTML. It receives rows on the board: claim, URL, passage, access date.

That is context isolation, not “one more job title”. The technical reason is the window: dozens of pages in the same history mix sources and push the right passage into the middle, again the effect from Liu et al. (2023).

4. Parallelize Independent Steps

Subagents run at the same time. Market, competitors, and regulation do not wait for each other. Report time stops being the sum of the reads and becomes the slowest one, plus planning and writing.

Check +20 XP on first tryThe researcher pastes 40 pages into a single summary and the writer cites Lei 14.300 while pointing at a 2020 blog post. Which change attacks the mixing of sources?

Each subagent writes structured claims (text, URL, passage) to a board, and the report uses only checked rows

Tell the writer’s prompt to ‘not mix sources’ and list URLs at the end

Send the summary to a second judge model, which scores each citation from 0 to 1

Final architecture

flowchart TD
  Q(["1. Question"]):::io --> O(["2. Orchestrator plans"]):::llm
  O --> S1(["3. Subagent: market"]):::llm
  O --> S2(["3. Subagent: competitors"]):::llm
  O --> S3(["3. Subagent: regulation"]):::llm
  S1 --> B[("4. Board of sourced claims")]:::store
  S2 --> B
  S3 --> B
  B --> C["5. Check citations"]:::tool
  C --> O2{"6. Gaps and budget?"}
  O2 -->|"gap, budget left"| O
  O2 -->|"done or limit"| W(["7. Write report"]):::llm
  W --> OUT(["8. Cited report"]):::io

1. The question arrives with a budget of steps, time, and cost.
2. The orchestrator splits the question into sub-questions and defines what counts as a sufficient answer. That is Plan-then-Execute plus Orchestrator-Workers: the plan depends on the question, not on five fixed job titles.
3. Subagents research in parallel, each in its own context, with search and page reading. Web content comes in as untrusted: the page may ask in HTML that the agent ignore the brief.
4. Each subagent writes structured claims (short text, source, supporting passage) to a shared board. The board has a schema and an owner per row. It is not a prose notepad.
5. Code checks that the cited passage exists on the page and that the claim is in it, with an automated check.
6. The orchestrator reviews the board and decides whether points are missing and whether budget remains. Stop conditions are three: sufficient, limit reached, declared gap.
7. The report is written from the checked claims, not from the pages. The writer does not visit the web.
8. The report goes out with every claim linked to its source, marked as partial if it stopped at the limit.

When the report template is stable (always market, competitors, regulation, risks), Sectioning fixes the sections and runs them in parallel without the orchestrator choosing the split. When the question changes shape (“compare three ANEEL tenders”), the orchestrator builds the plan. The two patterns coexist: fixed sections for the recurring product, a free plan for the odd brief.

Map-Reduce appears when a single source is huge: a 200-page EPE report, read by chapters, each chapter returning claims, the reduce merging without duplicates. EPE is Brazil’s energy research company. That is scanning one document. It does not replace subagents on different themes.

Worked example: residential solar in the Northeast

The question arrives: “what does the residential solar market look like in northeast Brazil?”. The job budget is a step cap, a minute cap, and a money cap, set from the price the consultancy charges for that kind of report. The exact values live in the reference code, not on this page.

The orchestrator returns a plan, not a draft:

{
  "enough_when": [
    "federal legal framework identified with a primary source",
    "at least one ANEEL rule cited with a passage",
    "two competitors or state programs with a source",
    "one recent trend with a date"
  ],
  "subquestions": [
    "What is the legal framework for distributed generation in Brazil?",
    "What does ANEEL define today for micro and mini generation?",
    "Which programs or constraints show up in Northeast states?",
    "Who are the visible players in residential?"
  ]
}

Three subagents start in parallel. The regulation one searches “Lei 14.300 distributed generation” and opens the text on Planalto, Brazil’s official statute site, not a blog summary. It writes:

{
  "id": "c-014",
  "claim": "Lei 14.300/2022 established the legal framework for micro and mini distributed generation in Brazil.",
  "url": "https://www.planalto.gov.br/ccivil_03/_ato2019-2022/2022/lei/l14300.htm",
  "passage": "Institui o marco legal da microgeração e minigeração distribuída",
  "accessed_at": "2026-10-02",
  "subagent": "regulation"
}

Lei 14.300/2022 exists, and the Portuguese passage above is in the official ementa (the statute’s summary line). That is the kind of claim the system should prefer: primary source, short span, access date. What the system does not do is invent gigawatts installed in Ceará or a market share. If the source with the number failed the check, the number does not enter the report.

The verifier fetches the URL (or uses the HTML the subagent already stored), looks for passage as a substring, and rejects the row if it cannot find it. A paraphrase that is too loose fails on purpose: the subagent has to copy a span that can be checked.

The orchestrator looks at the board. “Recent trend with a date” is missing. Budget remains: it launches a fourth subagent for that only. If the cap hits first, the report goes out partial, with the gap explicit. The junior analyst sees the hole. They do not get a fluent paragraph that hides the absence.

Final writing only sees rows with status: checked. Web HTML does not enter that context. Indirect injection on a competitor’s page may try to tell the subagent to “ignore the brief”. The subagent has no tool to send email or to edit the final report. It only writes rows to the board, and the board only accepts the schema.

How the citation is checked

flowchart LR
  L[("1. Board row")]:::store --> D["2. Fetch URL"]:::untrusted
  D --> V{"3. Passage on the page?"}
  V -->|no| X["4. Reject row"]:::tool
  V -->|yes| K["4. Mark checked"]:::tool
  K --> W(["5. Writer reads checked rows only"]):::llm

1. The row arrives with a URL and a literal passage.
2. The fetch is untrusted data: the page may have changed or be hostile.
3. Substring, in code. It is not a model saying “this looks true”.
4. Failed: the row does not feed the report. Passed: status checked.
5. The writer never sees a page. It sees claims already grounded.

This check is the antidote to Mata v. Avianca. The model is good at producing text that looks like case law, a paper, or an ANEEL note. Opening the URL is cheap next to the loss of a client who decides on a ghost number.

Common traps

• An org chart of agents. Researcher, analyst, writer, reviewer, editor. If only one has a tool, you bought expensive rewriting. Anthropic’s December 2024 post is blunt: complexity has to earn its keep with a measured gain.
• One summary with every page. That is a shared scratchpad. The citation slides. A structured claim is the contract.
• Research with no cap. “Until it looks good” is not a stop condition. Good needs a criterion in the plan, and the card needs a limit.
• An LLM judge instead of a fetch. A second model scoring the citation reproduces the fluency of invention. The proof is the passage on the page.
• A subagent that returns prose. If the output does not fit a schema, the board becomes a notepad again, and week 3 returns.
• Parallelizing work that has a dependency. Do not ask “compare with the number from section 2” before section 2 exists. Parallelism is for independent sub-questions.

The eval that holds the refactorings

The question set has anonymized real briefs and, for each one, a list of claims an analyst marked as required, with the URL they opened themselves. The eval does not ask for a “nice” report. It asks: did each required claim appear with a source that passes the verifier? Did the report land inside the time and cost caps? Are refusals (gaps) declared, rather than hidden in safe-sounding prose?

An eval case looks like this:

{
  "id": "solar-ne-014",
  "question": "What does the residential solar market look like in northeast Brazil?",
  "required": [
    {
      "theme": "GD legal framework",
      "source_type": "statute",
      "acceptable_example": "planalto.gov.br ... lei/l14300"
    }
  ],
  "forbidden": ["numeric claim without a checked URL"]
}

Version 1 was measured on this set before the collapses. Without that, “remove three agents” would have been an opinion. With it, it was week 2’s eval.

Check +20 XP on first tryA broad question has been searching for 40 minutes, still with no cap, and the client wants the report today. Which piece was missing in version 1?

One more reviewer, to slim the text while the researcher keeps going

A shared budget of steps, time, and cost, with a partial report and listed gaps when the cap hits

Replace web search with a model that has more cutoff knowledge

Numbers

This scenario’s metrics are claims with a valid source, time per report, and cost per report, comparing version 1 with the final architecture on a set of research questions with human review of the sources.

The three pull on each other. Minimum time with one sequential agent replays week 1. Minimum cost with no search replays Mata v. Avianca: fluent text, no source. Valid source on 100% of rows, with no cap, replays week 4.

The right reading is: of the claims that made the report, how many passed the passage verifier; how long until the PDF; how much the run cost relative to the price charged. A partial report with a declared gap counts as a correct stop, not as a quality failure.

The numbers are not published yet. They will come from the runnable reference code, running both versions against the same questions, and this section will be updated with the result and the script that produces it. Until then, no numbers appear here.

Patterns used

Pattern	Role in the system
Plan-then-Execute	The orchestrator defines sub-questions before researching
Orchestrator-Workers	The split depends on the question
Subagent Delegation	Each sub-question in its own subagent
Context Isolation	Pages read stay with the subagent
Sectioning	The report template’s fixed sub-questions run in parallel
Shared blackboard	Structured claims with owner and source
Citation grounding	Every claim checked against the cited passage
Stop Conditions	Sufficient, limit, or declared gap
Cost budget	A per-report cap shared across agents
Map-Reduce	A long document read in parts, then merged
Agentic RAG	The subagent picks the next search from what it just read

The full list of patterns tied to this scenario is under Relations, below.

Reference code

The complete system, with both versions, the eval questions, and the script that produces this page’s numbers, lives in the ai-guru reference code repository, under scenarios/deep-research/. The repository is being prepared, and the link goes here once it is published.

Lessons

1. Split by context, not by job title. The subagents that remained have a technical reason: each isolates the reading for one sub-question. Analyst and reviewer in a queue isolated nothing.
2. A structured claim beats a summary. Recording “claim, source, passage” ended the mixing of sources. The report became a reduce of checked rows.
3. Checking citations in code changes the outcome. The report stopped carrying sources that do not say what the text claims. Mata v. Avianca is what happens when the check is “this looks true”.
4. A cost cap is a product requirement. A study has to fit within the report’s price. With no cap, the most expensive run beats the sale.
5. Parallelism is where time drops. The sub-questions were independent from the start; they just needed to run at the same time. A queue of rewriters is not parallelism.

Check your understanding

Check +20 XP on first tryThe team wants to speed the report up by running writer, reviewer, and editor in parallel on the same draft. What happens?

Nothing useful: the three read the same ownerless prose, and you bring back a shared scratchpad at triple the cost

Time drops to a third, because the three rewrites leave the serial queue

It is orchestrator-workers applied the right way

Check +20 XP on first tryA subagent returns a numeric claim about installed capacity in Ceará, with the URL of a JavaScript infographic. The verifier cannot find the passage in the HTML. What does the system do?

Accept it, because the infographic probably has the number and the model ‘saw’ the page

Ask the subagent for a confidence score and publish if it is high

Reject the row. If the number is required in the plan and budget remains, try a primary source in HTML or text PDF

Check +20 XP on first tryThe consultancy’s report template is always the same: market, competitors, regulation, risks. When do you use fixed sectioning instead of letting the orchestrator invent the split on every brief?

Never: a flexible orchestrator is always better, because every client is unique

When the sections are stable and independent, so you can run them in parallel without paying a planner on every report

When cost does not matter, because sectioning uses more models at once