Trail 02 · 13 lessons · 13 on video
Evals and reliability
How to know an LLM feature works before your users tell you: evals, golden datasets, structured contracts, and deterministic work moved back into code.
- 01Fundamental▶ 9 minWhat an eval isTests for systems that never answer the same way twice, from unit case to benchmark.
- 02Smell▶ 7 minUntested PromptThe prompt changes in production with no test set to prove it did not get worse.
- 03Refactoring▶ 8 minAdd Eval Before Changing the PromptBuild a test set and measure the current version before any tweak.
- 04Pattern▶ 9 minEval HarnessRun test cases automatically on every prompt, model, or code change.
- 05Pattern▶ 9 minGolden DatasetKeep a curated set of inputs and expected answers that grows with real bugs.
- 06Smell▶ 7 minFree-Text HandoffOne step hands loose text to the next, which has to guess the format.
- 07Refactoring▶ 9 minReplace Free Text with Structured OutputDefine a schema for the step's output and validate it before moving on.
- 08Smell▶ 8 minLLM Doing Deterministic WorkThe model calculates taxes, formats dates, or validates IDs, things a function does without error.
- 09Refactoring▶ 8 minReplace LLM Call with CodeMove calculation, formatting, and validation into a function the model calls.
- 10Smell▶ 7 minBlind RetryEvery error triggers another attempt, including errors that will never succeed.
- 11Refactoring▶ 7 minClassify Errors Before RetrySeparate transient errors from permanent ones and retry only what can succeed.
- 12Real-world scenario▶ 8 minInvoice and contract extractionRead invoices and contracts in PDF and fill the ERP with no manual typing.
- 13Real-world scenario▶ 10 minNatural language to SQLLet business teams query the database without knowing SQL and without risk of damage.
- Final projectLangGraph LabInvoices into the ERPRead supplier invoices and post them to the ERP without anyone typing.