Overview

Prerequisites

• Prior topics: creating-ai-powered-apps (prompting, structured output, and RAG -- the alternatives this course requires you to exhaust first) -- [Unverified] this course is not yet present in the AyoKoding course library on disk, so no link is given here; evaluating-ai-systems-in-depth (hard prerequisite: a fine-tune with no eval is unfalsifiable, and every decision in this course is an eval comparison) -- [Unverified] not yet present in the AyoKoding course library on disk; Statistics for Evaluation (the base-versus-adapted comparison is a paired significance test, not two printed numbers); Inference Serving & Model Deployment (where an adapter is actually served); data-engineering (the dataset pipeline) -- [Unverified] not yet present in the AyoKoding course library on disk; just enough python (4 · Just Enough Python) is assumed for reading and writing fully type-annotated Python.
• Tools & environment: a macOS/Linux terminal; Python 3.13, the standard library only (json, math, dataclasses, pathlib, typing) -- no fine-tuning library, no network call, no real Hugging Face download, and no paid API key is required anywhere in this topic, deliberately. Every base model, tokenizer, and trainer this topic's examples reference is mocked exactly the way a real object's shape and failure modes behave.
• Assumed knowledge: the model-application material and its eval discipline; what tokens and a context window are; loading and transforming a dataset in Python; the idea of gradient-based training at a conceptual level. No deep-learning course is assumed -- this course explains what it needs and does not teach model architecture or backpropagation theory.

Why this exists -- the big idea

The problem before the solution: fine-tuning is the intuitive response to "the model does not do what I want," and it is usually the wrong one. The one idea worth keeping if you forget everything else: fine-tuning teaches behaviour, not facts -- exhaust prompting, retrieval, and scoping first, and when you do adapt, the dataset is the whole job.

Cross-cutting big ideas: correctness-vs-pragmatism -- adaptation is a costly bet on a measured gap, not a default; abstraction-and-its-cost -- an adapter is a compact approximation of a full fine-tune, with the trade-offs approximations carry; taming-state -- adapted weights are a versioned artefact with a lifecycle, not a configuration change.

Confirm your toolchain

Every example in this topic is standard-library-only -- no fine-tuning library, no network call, no real GPU, and no paid API key required anywhere:

$ python3 --version
Python 3.13.12
$ python3 -c "import json, math, dataclasses, pathlib, typing; print('stdlib primitives OK')"
stdlib primitives OK
$ pyright --version
pyright 1.1.411
$ ruff --version
ruff 0.15.9

Captured 2026-07-26 -- these are point-in-time patch versions, not a durable claim; see this topic's Accuracy notes if you are reading this after a later patch has shipped.

Every worked example is a complete, self-contained, fully type-annotated (strict pyright) runnable Python file colocated under learning/code/, actually executed against Python 3.13 to capture its documented output -- every printed value on this topic's pages is a genuine, captured transcript, never a fabricated one. Every code-bearing worked example is independently runnable with no cross-example imports; the capstone's five files are the sole exception, each committing a JSON artefact the next step's file reads, so the five steps run as one small, ordered project.

How this topic's examples are organized

• Band A -- The Decision, Not the Technique (Examples 1-16, 51-58) -- measuring a real gap, triaging behaviour-shaped versus knowledge-shaped complaints, exhausting prompting/retrieval/scoping in that order, the written decision gate, legitimate cases the gate should pass, and the costs nobody budgets.
• Band B -- The Dataset and the Training Run (Examples 17-34, 59-67) -- assembling and auditing a supervised fine-tuning dataset, sourcing strategies and their biases, leak-free splits, full fine-tuning versus parameter-efficient adapters, rank as a capacity knob, and the hyperparameters that actually matter.
• Band C -- Evaluation, Distillation, and Operation (Examples 35-50, 68-75) -- the paired comparison against the base, the forgetting-regression suite, overfitting invisible in training loss, distillation's ceiling, serving and hot-swapping adapters, version pinning, and retirement.
• Capstone -- the complete adaptation arc for one real behaviour gap: a measured decision gate, a curated and audited dataset, a rank-justified adapter, a paired evaluation with a forgetting-regression suite, a served swappable artefact, and a written maintenance and retirement plan.

Read more

• AI Engineering: Building Applications with Foundation Models -- Chip Huyen (O'Reilly, 2025). The corrected source for this course's "exhaust prompting/RAG/scoping before fine-tuning" framing -- see this topic's Accuracy notes for the correction detail.
• LoRA: Low-Rank Adaptation of Large Language Models -- Hu, Shen, Wallis, Allen-Zhu, Li, S. Wang, L. Wang & Chen, arXiv:2106.09685 (2021). The originating paper for the low-rank-adaptation principle this course teaches as durable. https://arxiv.org/abs/2106.09685
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks -- Patrick Lewis et al. (2020). The alternative this course requires you to rule out before adapting weights. https://arxiv.org/abs/2005.11401
• Language Models are Few-Shot Learners -- Tom B. Brown et al. (2020). The result that made instruction-and-example steering a substitute for task-specific training. https://arxiv.org/abs/2005.14165

Examples by Level

Beginner (Examples 1–16, 51–58)

• Worked Example 1: Measure the Gap First
• Worked Example 2: Behaviour-vs-Knowledge Triage
• Worked Example 3: The Knowledge-Injection Mistake
• Worked Example 4: Retrieval Beats It
• Worked Example 5: Prompting Closes the Gap
• Worked Example 6: Structured Output Closes the Gap
• Worked Example 7: Scoping Closes the Gap
• Worked Example 8: The Decision Procedure
• Worked Example 9: A Correct No-Go
• Worked Example 10: Legitimate Case -- Format
• Worked Example 11: Legitimate Case -- Register
• Worked Example 12: Legitimate Case -- Smaller Model
• Worked Example 13: Total Cost of a Fine-Tune
• Worked Example 14: The Maintenance Obligation
• Worked Example 15: Licence and Data-Rights Check
• Worked Example 16: The Decision Diagram
• Worked Example 51: Legitimate Case -- Tool Use
• Worked Example 52: A Knowledge-Heavy Request, Rejected
• Worked Example 53: A Gap Too Small to Matter
• Worked Example 54: The Decision Procedure as Code, Table-Tested
• Worked Example 55: Comparing Alternatives Side by Side
• Worked Example 56: A Request That Should Be Scoped, Not Trained
• Worked Example 57: The Decision Record Template
• Worked Example 58: Revisiting a No-Go Decision

Intermediate (Examples 17–34, 59–67)

• Worked Example 17: First SFT Dataset
• Worked Example 18: Quality Beats Quantity
• Worked Example 19: Inconsistent Examples
• Worked Example 20: Consistency Audit
• Worked Example 21: Source from Production Traffic
• Worked Example 22: Expert-Authored Examples
• Worked Example 23: Synthetic Generation
• Worked Example 24: Teacher Errors Propagate
• Worked Example 25: Train/Validation/Test Split
• Worked Example 26: Leakage Inflates the Result
• Worked Example 27: First Full Fine-Tune
• Worked Example 28: Full Fine-Tune Cost
• Worked Example 29: First LoRA Adapter
• Worked Example 30: Adapter vs. Full Cost
• Worked Example 31: Rank Sweep
• Worked Example 32: Rank Too High Overfits
• Worked Example 33: Learning Rate and Epochs
• Worked Example 34: Hyperparameters Cannot Fix Data
• Worked Example 59: Deduplicating the Dataset
• Worked Example 60: Balancing Task Coverage
• Worked Example 61: A Mixed Sourcing Strategy
• Worked Example 62: Detecting Synthetic Drift
• Worked Example 63: Stratified Splitting
• Worked Example 64: Near-Duplicate Leakage
• Worked Example 65: Targeted Modules and Adapter Placement
• Worked Example 66: Alpha and Rank Together
• Worked Example 67: Early Warning Signs of Overfitting

Advanced (Examples 35–50, 68–75)

• Worked Example 35: Evaluate Against the Base
• Worked Example 36: Regression Suite
• Worked Example 37: Catastrophic Forgetting, Measured
• Worked Example 38: Forgetting Is Worse for Full Fine-Tune
• Worked Example 39: Overfitting Invisible in Training Loss
• Worked Example 40: Early Stopping on Validation
• Worked Example 41: The Fine-Tune That Did Not Help
• Worked Example 42: Distil a Smaller Student
• Worked Example 43: Student Cannot Exceed Teacher
• Worked Example 44: Distillation Decision Record
• Worked Example 45: Serve an Adapter
• Worked Example 46: Hot-Swap Adapters
• Worked Example 47: Adapter Memory and Routing
• Worked Example 48: Version-Pinning to a Base
• Worked Example 49: Retire an Adapter
• Worked Example 50: Capstone-Justified Adaptation
• Worked Example 68: Statistical Significance of the Improvement
• Worked Example 69: Regression Suite Severity Weighting
• Worked Example 70: A Fine-Tune That Helped and Hurt
• Worked Example 71: Distillation with a Held-Out Teacher Eval
• Worked Example 72: Adapter Registry and Discovery
• Worked Example 73: Load-Testing Adapter Swaps
• Worked Example 74: A Licence-Blocked Retirement
• Worked Example 75: Writing the Maintenance and Retirement Plan

← Previous: Overview · Next: Band A: The Decision, Not the Technique →