Parallelize Independent Steps

Problem

Calls that do not depend on each other run in a queue, and the user waits for the sum of all of them. This is the Serial Latency Stack smell.

Picture a copilot for bank support agents. When a customer sends a WhatsApp message (WhatsApp is the default support channel for Brazilian banks), the system prepares four things before suggesting a reply: the intent of the message, a summary of the customer’s CRM history, the policy that applies, and the customer’s sentiment. Each one is a model call. The code calls one, waits, calls the next, waits.

None of the four needs another’s result. They all read only the conversation or the history. Even so, the support agent stares at a spinner while they finish one after another.

Solution

Draw the pipeline’s dependency graph and run every step that waits on no other result at the same time. Latency becomes that of the longest path, not the sum.

Before and after

-intent    = await classify_intent(transcript)
-customer  = await summarize_customer(crm_history)
-policy    = await find_policy(transcript)
-sentiment = await detect_sentiment(transcript)
+intent, customer, policy, sentiment = await asyncio.gather(
+    classify_intent(transcript),
+    summarize_customer(crm_history),
+    find_policy(transcript),
+    detect_sentiment(transcript),
+)
 reply = await generate_reply(intent, customer, policy, sentiment)

Before: four steps in a queue.

flowchart LR
  IN(["1. Input"]):::io --> A(["Intent"]):::llm
  A --> B(["Customer"]):::llm
  B --> C(["Policy"]):::llm
  C --> D(["Sentiment"]):::llm
  D --> R(["2. Reply"]):::llm

After: all four at once, and the reply when the last one finishes.

flowchart LR
  IN(["1. Input"]):::io --> A(["Intent"]):::llm
  IN --> B(["Customer"]):::llm
  IN --> C(["Policy"]):::llm
  IN --> D(["Sentiment"]):::llm
  A --> R(["2. Reply"]):::llm
  B --> R
  C --> R
  D --> R

1. The input fans out to every step that depends only on it.
2. The step that needs them all waits for the slowest, not the sum.

Worked example

Suppose these durations, taken from the copilot’s trace (illustrative numbers):

In a queue, the support agent waits 0.8 + 2.4 + 1.5 + 0.6 + 1.8 = 7.1 seconds. In parallel, the first phase lasts as long as the slowest step (2.4 s), and then the reply runs (1.8 s). Total: 4.2 seconds.

The number of calls is the same, the token count is the same, and the final text is the same. Only the clock changed.

Why it works

A model call spends almost all its time waiting for the provider to generate tokens. The calling process sits idle. While waiting on one, it can fire the others. For independent steps there is no reason for an order: the result is the same, and the clock runs once.

This kind of wait is the classic I/O-bound case (input and output). You need no extra CPU and no extra threads. An asyncio.gather in Python or a Promise.all in TypeScript is enough, because the program is only waiting on the network.

The gain has a clear limit: the critical path. If the slowest step takes 2.4 seconds, the parallel phase takes close to that, not less. So after parallelizing, the next target is the slowest step: a smaller model, a smaller input, or a cache.

The idea has a name in Anthropic’s “Building effective agents” post (December 2024): the parallelization workflow. It comes in two flavors. In Sectioning, each call handles a different part of the task, as in the copilot. In Voting, several calls do the same task and you compare the answers. When the same operation repeats over many items, the pattern is Map-Reduce.

Safe mechanics

0. Measure. Run the pipeline with tracing and note each step’s duration and the total latency at p50 and p95.
1. Draw the dependencies. For each step, list what it reads. Those that only read the original input are candidates.
2. Confirm there is no hidden dependency: shared state, write order in a database, side effects.
3. Replace the sequence with concurrent execution (asyncio.gather, Promise.all, or your framework’s feature).
4. Handle partial failures: decide whether one step’s failure brings everything down or the pipeline continues with a default value.
5. Cap concurrency if the number of simultaneous calls could exceed the rate limit.
6. Make sure the trace shows overlapping spans under the same parent.
7. Measure again. Total latency should be close to that of the slowest step, and quality should not change.

Common mistake: the dependency that lives in the prompt

Beginners tend to look only at the code. Sometimes the dependency lives in the text. Imagine the find_policy prompt says “consider the intent identified above”, and the code builds that prompt from the intent variable. In the queued version it worked. In the parallel version, intent does not exist yet when the policy step starts.

To catch this, build the dependency list from the variables that go into each prompt, not from the order in which the steps appear in the file.

Risks

• Hidden dependency. A step that used another’s result through an indirect path (a global variable, a cache) now reads a stale value. Running the eval before and after catches it.
• Rate limits. More simultaneous calls can produce 429 errors. Use a concurrency cap (a semaphore) and retry with backoff.
• One failure takes everything down. With default asyncio.gather, the first error propagates to the caller and you lose the other results, even though the other calls keep running and keep being billed. Promise.all behaves the same way. If the pipeline can continue without one step, use return_exceptions=True or Promise.allSettled and handle each result.
• Same cost, wrong expectation. Parallelizing saves no money. If the goal is cost, look for another refactoring, such as Introduce Cache.

The rate limit math is simple. If the copilot serves 50 conversations at once and each fires four calls together, the peak becomes 200 simultaneous calls, against 50 before. Total consumption per minute does not change, but the peaks are four times higher. The peak is what hits the provider’s limit.

Challenge

Which of these contract reviewer steps can run in parallel?

1. extract_text(pdf)
2. identify_parties(text)
3. list_clauses(text)
4. classify_risk(each clause)
5. check_parties_against_sanctions_list(parties)
6. write_opinion(risks, check)

text = extract_text(pdf)
parallel { parties = identify_parties(text); clauses = list_clauses(text) }
parallel {
    risks = parallel_map(classify_risk, clauses)
    check = check_parties_against_sanctions_list(parties)
}
opinion = write_opinion(risks, check)

Check your understanding