Prompt Template Registry

Intent

Store prompts as versioned artifacts, with a name, declared variables, history, and a publishing process. It is for systems with several prompts in production, edited by more than one person or more than one team.

Problem

Henrique leads the AI platform at a retail company. There are 23 prompts in production, spread across four services: SDR, support, product descriptions, and review summaries. Some live in constants in code, others in a config table edited through a panel.

Three problems come up every week. Nobody knows which version produced the answer behind a complaint. The marketing team wants to adjust the tone without waiting for a deploy, but there is no way to test before publishing. And two services use different copies of the same persona, which drifted apart over time.

On Tuesday a customer files a complaint about an SDR message that promised a Pix discount (Pix is Brazil’s instant payment system) in the first conversation. Henrique opens the trace: it has the generated text, the model, the timestamp. It does not have the prompt. The config panel was edited Wednesday morning. Tuesday’s text is gone.

What is missing is not a better prompt. It is a home for prompts, with the same guarantees code has: versions, review, testing, and rollback.

The question this pattern answers: how do you manage dozens of prompts in production with traceability and safe changes?

Solution

Build a template registry. Each template has:

• A stable name and an immutable version: publishing a change creates a new version; old ones never change.
• Declared variables, with types, so code cannot assemble half a prompt.
• Metadata: target model, author, reason for the change, eval result.
• Environment labels: production, staging, canary, pointing to specific versions.

Code asks for the template by label (or a pinned version), renders it with the variables, and records name and version on the trace. Publishing to production means moving the label after the eval passes. Rolling back means moving the label back.

This treats Unversioned Prompt and Scattered Prompt Strings. The refactoring that lands here is Extract Versioned Template.

Structure

flowchart LR
  E((("1. Author edits"))):::human --> V["2. New immutable version"]:::tool
  V --> EV["3. Eval runs on the new version"]:::tool
  EV --> R[("4. Registry: versions and labels")]:::store
  R -->|"production label"| S["5. Service renders with variables"]:::tool
  S --> M(["6. Model"]):::llm
  M --> T[("7. Trace with name and version")]:::store

1. Someone proposes a change to the text or the variables.
2. The change becomes a new version; the previous one keeps existing.
3. The eval runs automatically on the new version.
4. The registry stores every version and which labels point to which.
5. The service loads the labeled version and fills in the declared variables.
6. The model receives the rendered prompt.
7. Every run records the template and version used.

Worked example

The complaint points to conversation conv-4412, Tuesday at 4 p.m. The trace now has prompt.name=sdr_first_message and prompt.version=14. Henrique opens the registry:

# prompts/sdr_first_message/labels.yaml
production: 14
staging: 15

Version 14, immutable, is still on disk. The body says: “Do not offer a discount in the first message. If the lead asks about price, point them to the current price list.” Version 15, in staging, relaxed that rule for a conversion test. The eval for 15 dropped 0.3 on the tone rubric and has not earned the production label yet.

Henrique answers legal with the exact text the model received. Marketing keeps the test in staging. Nobody rewrote the past.

Compare the two ways of changing:

Pseudocode

schema Template { name: string, version: int, model: string, variables: map[string, type],
                  body: string, author: string, reason: string, eval_result: EvalSummary? }

function publish(name, body, variables, author, reason):
    v = registry.create_version(name, body, variables, author, reason)   # 2. immutable
    v.eval_result = eval_harness.run(name, v)                            # 3
    if v.eval_result.regressed(baseline=registry.get(name, label="production")):
        return reject(v)
    registry.set_label(name, "staging", v.version)
    return v

function promote(name, version):
    registry.set_label(name, "production", version)                      # rollback = point back

function render(name, label, values):
    t = registry.get(name, label=label)                                  # 5
    assert set(values) == set(t.variables)                               # nothing missing or extra
    trace.span("prompt", name=t.name, version=t.version)                 # 7
    return t.render(values)

Real-world scenarios

• Sales SDR. Prospecting messages have one template per segment, versioned. The sales team proposes tone changes, the eval runs, and the label only moves with approval.
• Content pipeline. Each stage (research, draft, brand review) has its template; the published post stores the versions of all of them.
• Support bot with human escalation. Each route’s prompts live in the registry, and every conversation’s trace points to the version that answered.

Applicability

• Several prompts in production, across more than one service.
• People outside engineering need to propose text changes.
• Complaints and incidents must be traced back to the prompt version.
• You want to switch models route by route, with comparison and rollback.

When not to use

• One or two prompts, in one service, maintained by the same team. The repository with git and the commit hash on the trace already give you versions and history.
• Prompts generated dynamically by another model. Version the generator, not each output.
• Examples and knowledge. They change on a different cycle; see Few-Shot Selector and RAG.

A registry does not replace the eval. Without an Eval Harness at publish time, the production label becomes a “ship and hope” button.

How to implement

1. Inventory the prompts in production and where each one lives.
2. Define the template format: name, version, typed variables, model, body, metadata.
3. Choose where to store it: a versioned directory in the repository, a table with immutable versions, or an off-the-shelf prompt management tool.
4. Migrate prompts one by one, checking that the rendered output matches what was assembled before.
5. Tie publishing to the eval: a new version only gets the production label if there is no regression.
6. Record name and version on every model call span.
7. Document the rollback (moving the label) and rehearse it once.

Start with the repository. YAML files with labels.yaml and one vN.yaml per version cover most cases. Only move to a service with a panel when marketing or legal need to edit without waiting for a merge, and even then the eval stays the gate.

Common mistakes

• Logic in the template. {{ "formal" if segment == "industry" else "casual" }} looks harmless until it has eight branches. Pull the decision into code; leave the template with already-resolved variables.
• An optional variable that is truly optional. If coupon sometimes appears and sometimes does not, declare two versions or a field that is always present (empty string). Half-rendering is the bug the schema exists to catch.
• A label pointing at a version with no eval. Treat that as an incident, not a Friday shortcut.
• Copying the template across services. Two copies drift on the second edit. One name, several labels.

Variations

• Registry in the repository. YAML or Markdown files versioned by git, with the label as a config file. Simple and auditable.
• Registry as a service. An API with versions and labels, editable through a panel. Allows changes without a deploy, with the eval as the gate.
• Template per model. One template version per supported model, to migrate and compare without breaking things.
• Canary. The canary label receives a share of traffic before becoming production. It pairs with the idea of a shadow deploy: the new version answers in parallel, without reaching the customer, until the production eval matches staging.

Cost, latency, and failure modes

The eval on publish is not cheap, and it does not need to be. Run the small set (cases that catch tone, discount refusal, required variables) on every version. The large set runs overnight on the staging label.

How to evaluate

• Eval per version, stored alongside the version, to compare history.
• Traceability: pick random answers from the trace and check that the version points to an existing template with an eval.
• Time to fix: how long it takes from a complaint to a new, tested version in production.

{ "template": "sdr_first_message", "candidate": 15, "baseline": 14, "eval": "evals/sdr.jsonl", "block_if": "tone_rubric < baseline - 0.2" }

A concrete eval case: the message must not mention price or Pix in the first turn. If version 15 breaks that, the production label does not move.

In production

• prompt.name and prompt.version attributes on every model span.
• A dashboard with each template’s active version and the date of its last change.
• An alert when a version without an eval gets the production label.
• A local cache of active versions, so the registry is not a single point of failure.

Rehearse the rollback. Moving production: 15 to production: 14 has to be a documented step, tested once in staging, not an invention in the middle of an incident.

Security

Templates are text, and text attracts shortcuts. An API key pasted into the body, “just so the model can call the CRM”, becomes the Secret in the Prompt smell. The registry should not accept that: credential variables do not exist; tools receive secrets in the backend. See Move Secret Out of Prompt.

Who can publish to production is a separate role from who can propose a version. The marketing panel writes to staging. Engineering (or an eval bot) moves the label.

Template output still goes to the model along with lead data. Treat the rendered body as what it is: your instruction plus outside data. Structured Output on the response side reduces the damage of a template that asks for too much.

Pros and cons

Code

Both tabs implement a simple registry on versioned files, with a production label and the version recorded on the trace.

from dataclasses import dataclass
from pathlib import Path

import yaml
from jinja2 import Environment, StrictUndefined

ROOT = Path("prompts")
jinja = Environment(undefined=StrictUndefined)  # a missing variable becomes an error


@dataclass(frozen=True)
class Template:
    name: str
    version: int
    model: str
    variables: tuple[str, ...]
    body: str

    def render(self, **values) -> str:
        missing = set(self.variables) - set(values)
        extra = set(values) - set(self.variables)
        if missing or extra:
            raise ValueError(f"{self.name}@{self.version}: missing {missing}, extra {extra}")
        return jinja.from_string(self.body).render(**values)


def load(name: str, label: str = "production") -> Template:
    labels = yaml.safe_load((ROOT / name / "labels.yaml").read_text())  # {"production": 14, "staging": 15}
    version = labels[label]
    data = yaml.safe_load((ROOT / name / f"v{version}.yaml").read_text())  # files never edited after creation
    return Template(name, version, data["model"], tuple(data["variables"]), data["template"])


# usage
tpl = load("sdr_first_message")
prompt = tpl.render(lead_name="Ana", company="Sunrise Bakery", segment="retail",
                    pricing_policy="no price in the first message", campaign="launch")
span.set_attributes({"prompt.name": tpl.name, "prompt.version": tpl.version})  # 7

type Template = { name: string; version: number; model: string; variables: string[]; body: string };

const cache = new Map<string, { tpl: Template; fetchedAt: number }>();

export async function load(name: string, label = "production"): Promise<Template> {
  const key = `${name}:${label}`;
  const hit = cache.get(key);
  if (hit && Date.now() - hit.fetchedAt < 60_000) return hit.tpl;
  try {
    const res = await fetch(`${process.env.PROMPT_REGISTRY_URL}/templates/${name}?label=${label}`);
    if (!res.ok) throw new Error(`registry answered ${res.status}`);
    const tpl = (await res.json()) as Template;
    cache.set(key, { tpl, fetchedAt: Date.now() });
    return tpl;
  } catch (err) {
    if (hit) return hit.tpl; // registry down: use the last known version
    throw err;
  }
}

export function render(tpl: Template, values: Record<string, string>): string {
  const missing = tpl.variables.filter((v) => !(v in values));
  const extra = Object.keys(values).filter((v) => !tpl.variables.includes(v));
  if (missing.length || extra.length) {
    throw new Error(`${tpl.name}@${tpl.version}: missing ${missing}, extra ${extra}`);
  }
  return tpl.body.replace(/\{\{\s*(\w+)\s*\}\}/g, (_, k: string) => values[k]);
}

Check your understanding