Classify Errors Before Retry

Problem

The assistant schedules a Pix. The bank API returns a timeout. Code retries five times with the same call. The bank had already recorded the first one. Five Pix transfers go out. Every error triggers another attempt, including errors that will never succeed, and side effects end up duplicated. This is the Blind Retry smell.

Solution

Before retrying, classify the error. Transient errors (network, rate limit, momentary unavailability) get retries with growing backoff and a limit. Permanent errors (validation, permission, balance, business rule) go back to the caller with an explanation. Every side-effect call carries an idempotency key.

Before and after

-for attempt in range(5):
-    try:
-        return bank.schedule_pix(key, amount, date)
-    except Exception:
-        time.sleep(2 ** attempt)
+TRANSIENT = {"TIMEOUT", "CONNECTION_RESET", "RATE_LIMITED", "SERVICE_UNAVAILABLE"}
+
+def schedule_pix(key, amount, date, request_id):
+    for attempt in range(3):
+        try:
+            return bank.schedule_pix(key, amount, date, idempotency_key=request_id)
+        except BankError as e:
+            if e.code not in TRANSIENT:
+                return {"ok": False, "error": e.code, "retryable": False,
+                        "message": HUMAN_MESSAGES.get(e.code, e.message)}
+            time.sleep(min(2 ** attempt + random.random(), 10))   # backoff with jitter
+    return {"ok": False, "error": "UNAVAILABLE", "retryable": True,
+            "message": "The bank is unstable. Try again in a few minutes."}

Before: any error, same response.

flowchart LR
  C["1. Call"]:::tool --> E{"2. Error?"}
  E -->|"any"| C

After: the error type decides.

flowchart LR
  C["1. Call with idempotency key"]:::tool --> E{"2. Error type"}
  E -->|"transient"| W["3. Wait and retry, with a limit"]:::tool
  W --> C
  E -->|"permanent"| R(["4. Explain to the caller"]):::io
  E -->|"ok"| OK(["5. Result"]):::io

1. Every side-effect call carries a key the service uses to avoid repeating the effect.
2. The error is classified by code or status, never by its message text.
3. Only transient errors are retried, with growing backoff and a cap.
4. Permanent errors become a clear answer the model or the user can act on.
5. Success happens once, even with retries.

Why it works

Retrying only helps when the error’s cause can go away on its own. Unstable networks and rate limits pass; insufficient balance and an invalid Pix key do not. Classifying avoids spending attempts on what will not work, and avoids an attempt “succeeding” at the wrong moment.

The idempotency key solves the most dangerous case: a call that succeeded on the server, but whose response was lost on the network. Without the key, the retry repeats the effect. With it, the server recognizes the repeat and returns the first result. Serious payment APIs (Stripe’s, Brazilian banks’ Pix APIs) accept that key for this reason.

In an agent, a well-explained permanent error also helps the model: instead of repeating schedule_pix with the same arguments, it informs the user or changes course. Classify by code (INSUFFICIENT_FUNDS, 400, 429), never by if "timeout" in str(e). Messages change; codes do not.

Worked example: a timeout that was success

Input: schedule R$250 for tomorrow, request_id=chk_9f3a. The bank processes it, the network drops before the response. With no key, the retry creates a new chk and schedules again. With a key, the second POST returns the same schedule. Output on the trace: attempts=2, effect=1.

Treating the timeout as transient is right. What stops the duplicate is the key born with the user’s intent, not with the attempt.

Check +20 XP on first tryThe API returns 400 with INSUFFICIENT_FUNDS. Your loop retries with growing backoff, because ‘sometimes the balance changes’. What does that cause?

A fair second chance: the user may have deposited in that window

A retry on a permanent error: it burns attempts, delays, and does not fix the balance. Return the message to the model or the user

Nothing, if there is an idempotency key the bank ignores the extra attempts

Safe mechanics

0. Measure. Pull from the logs each dependency’s most common errors and how many retries each produced. Look for duplicated effects (two Pix transfers for the same request).
1. Classify each service’s error codes as transient or permanent. When in doubt, treat as permanent.
2. Add an idempotency key to every side-effect call, generated once per user intent.
3. Implement retries only for transient errors, with growing backoff, jitter (a little randomness in the wait, so a thousand clients do not hit the same second), and a cap. Honor Retry-After when the service sends it.
4. Write useful messages for permanent errors, for the model and the user.
5. Detect repetition in the agent: the same tool with the same arguments after a permanent error ends that path.
6. Measure again: retries per error, duplicated effects (should be zero), and time to answer.

Trap: three layers of retry

The provider SDK retries twice. Your function retries three times. The agent loop calls the tool again. Nine attempts, and nobody agreed on the cap. Turn off the SDK retry (max_retries=0) on calls you classify. One place decides.

Another trap: generating uuid4() inside the for attempt. Each turn is a new key, and idempotency vanishes. The key is born with the click, the ticket, the conversation’s request_id.

If the service does not accept a key, check state before repeating (“is there already a Pix with this request_id?”). With no check, do not retry a side effect.

Check +20 XP on first tryYou generate idempotency_key=str(uuid4()) at the start of each for attempt turn. The bank accepts the key. Why do Pix transfers still duplicate on a timeout?

The bank ignores the key on timeout; idempotency only applies to 200

uuid4 is not deterministic enough; hash the amount and the Pix key instead

Each attempt sends a different key, so the bank sees new requests. The key must be born with the intent, outside the loop

Risks

• Wrong classification. A permanent error treated as transient brings back the original problem. Review the list when the service changes.
• An idempotency key generated per attempt. Then it protects nothing.
• A service without idempotency support. In that case, check the state before repeating.
• Stacked retries. The SDK retries, your function retries, and the agent retries. Pick a single place responsible for retries. Combine with Add Fallback Path when the transient error persists: after the cap, a breaker, not one more turn.

Challenge

Classify these errors from an email-sending API and say what to do with each one.

429 Too Many Requests
400 Invalid recipient address
503 Service Unavailable
401 Invalid API key
409 Duplicate idempotency key (message already sent)

Show solution

• 429: transient. Wait (respecting Retry-After) and try again, with a limit.
• 400 invalid address: permanent. Tell the model or the user the address needs fixing.
• 503: transient. Retry with backoff; if it persists, open the breaker and use the fallback.
• 401: permanent and a configuration problem. Do not retry; alert the team, because no call will work.
• 409 duplicate key: a disguised success. The email was already sent; treat it as a positive result, without resending.

Related

• Treats the Blind Retry smell.
• When the transient error persists: Add Fallback Path.
• An effect that should not even be retried without a person: Add Approval Before Side Effect.

Check your understanding

Check +20 XP on first tryThe agent calls schedule_pix, gets INVALID_PIX_KEY, and calls again with the same arguments on the next step. Which piece of this refactoring is missing in the agent?

More SDK retries, because an invalid key is sometimes a DNS delay

A repetition detector: same tool, same arguments, after a permanent error, end the path and explain

A new idempotency key on each agent call, so the bank accepts the correction

Check +20 XP on first tryThe provider returns 429 with a Retry-After: 20 header. Your backoff was 2s, 4s, 8s. What do you do on this attempt?

Wait 20s (what the service asked) and only then retry, counting against the cap

Ignore the header and follow 2s, 4s, 8s, which is the internal policy

Open the breaker immediately, because 429 means the provider is down

Check +20 XP on first tryA 409 Duplicate idempotency key arrives after a timeout. The email was already sent. How does the code treat it?

As a permanent error: tell the user it failed, so they can resend if they want

As transient: retry with the same key until it becomes 200

As a disguised success: return ok to the model, with no new send, and record the reconciliation on the trace