Chapter 14 · Quality

Skill Training

Where Chapter 13 measures skill quality, Chapter 14 improves it. The skill-training subsystem is a port of microsoft/SkillOpt, and it rests on one unusual idea: the thing being trained is not the model. The model never changes. What changes is a single SKILL.md file — a markdown document of instructions — which the subsystem edits, tests, and keeps or discards, over and over, until it gets better.

"Training" a markdown file?

Yes — and the borrowed vocabulary is deliberate. Normal machine learning nudges millions of numeric weights toward a better score. Here the document plays the role of the weights, and the same disciplines apply: changes are small and bounded, each one has to prove it improved a measured score before it is kept, and the size of the changes shrinks over time. The big practical difference is that it costs nothing at runtime. You end up with a better text file, not a bigger model, so there is no extra inference cost afterwards.

The 30-second tour

# From the repo root — runnable demo with deterministic stubs (no LLM calls).
dotnet run --project src/Content/Presentation/Presentation.ConsoleUI -- --example skill-training

You get the per-step audit trail (rollout → reflect → aggregate → select → apply → gate) for a small 2-epoch × 3-step run, plus the final BestSkill, BestScore, and HasAcceptedAny flag. The demo uses stubs so the loop runs without an agent endpoint — the same orchestrator powers real training once an IPatchProposer and IRolloutRunner are plugged in.

The DL ↔ skill-training analogy

Every concept in the subsystem has a deep-learning counterpart. If you've trained a network before, you already know how to tune this loop.

Deep learning	Skill training	Where in code
Model weights	Skill document (markdown)	`currentSkill` string
Forward pass	Rollout	`IRolloutRunner.RunAsync`
Loss function	Eval metric (hard / soft)	`RolloutBatchScorer`
Backpropagation	Reflect	`IPatchProposer.ProposeAsync`
Gradients	Edit patches	`Patch{Edits[], Reasoning}`
Gradient aggregation	Patch aggregation	`IPatchAggregator`
Gradient clipping	Top-K edit selection	`IEditSelector`
Learning rate	Edits per step	`ILrScheduler.GetLearningRate`
LR scheduler	Cosine / Linear / Constant	`Schedulers/`
SGD step	Skill update	`PatchApplier.Apply`
Validation set	Selection split (val rollouts)	`RolloutBatch{Split="val"}`
Early stopping	Patience-based reject counter	`TrainSkillConfig.Patience`
Momentum	Slow update (longitudinal pair)	`SlowUpdateCommand`
Meta-learning	Cross-epoch strategy memory	`MetaSkillUpdateCommand` → `IKnowledgeMemory`
Checkpointing	Per-step snapshot	`ISkillTrainingCheckpointStore`

The six stages, per step

┌──────────────────────────────────────────────────────────────┐
│  for epoch in 1..Epochs:                                     │
│    for step in 1..StepsPerEpoch:                             │
│      1. Rollout    — IRolloutRunner on train batch          │
│      2. Reflect    — IPatchProposer turns failures into Patch│
│      3. Aggregate  — dedup edits, sum SupportCount           │
│      4. Select     — clip to LR budget                       │
│      5. Apply      — PatchApplier walks edits over the doc   │
│      6. Gate       — score candidate on val, decide          │
│    Epoch boundary:                                           │
│      • SlowUpdate longitudinal pair (anti-forgetting)        │
│      • MetaSkillUpdate cross-epoch memory                    │
└──────────────────────────────────────────────────────────────┘

All six stages run on the same call stack inside TrainSkillCommandHandler — no MediatR re-entrance inside the inner loop, which keeps per-step allocation low and the call trace easy to follow. The two epoch-boundary mechanisms are dispatched via IMediator so they participate in the standard pipeline (validation, audit, telemetry) on equal footing with every other CQRS command.

How it's wired

Piece	Where	Role
`TrainSkillCommand`	Application.AI.Common/CQRS/SkillTraining/TrainSkill	Orchestrator entry. Chains the six stages with patience-based early stop and per-step checkpointing.
`GateCandidateSkillCommand`	…/GateCandidateSkill	Pure decision over pre-computed (hard, soft) scores; standalone if you score skills outside the orchestrator.
`ReflectOnFailuresCommand`	…/ReflectOnFailures	Delegates to `IPatchProposer`; surfaces proposer failures as stable scrubbed Result.Fail codes (no exception text leaks).
`SlowUpdateCommand`	…/SlowUpdate	Paired longitudinal classification + guidance synthesis at epoch boundary.
`MetaSkillUpdateCommand`	…/MetaSkillUpdate	Persists cross-epoch memory via `IKnowledgeMemory` under `skill-training/meta/{skillId}/{runId}`.
`PatchApplier`	Application.AI.Common/Services/SkillTraining	Pure four-op editor (`Append \| InsertAfter \| Replace \| Delete`). First-occurrence match, per-edit success/failure report, no whitespace coercion.
`GateEvaluator`	…/Services/SkillTraining	Strict-greater accept semantics, finite-checked, NaN/Infinity guard. Hard/Soft/Mixed projection.
`PatchAggregator`	…/Services/SkillTraining	Dedup by (Op, Target, Content) with trailing-whitespace tolerance; sums SupportCount.
`TopKEditSelector`	…/Services/SkillTraining	Stable rank by support → merge depth → insertion order.
`ILrScheduler` impls	…/Services/SkillTraining/Schedulers	CosineScheduler (default), LinearScheduler, ConstantScheduler. Shared `SchedulerArgs.Validate`.
`InMemorySkillTrainingCheckpointStore`	…/Services/SkillTraining	Default: best + last 64 per run. Swap to EF Core for durability.
`NotConfigured*` defaults	…/Services/SkillTraining	Fail-fast stubs for `IPatchProposer` and `IRolloutRunner`. Replace with agent-backed Infrastructure impls before invoking `TrainSkillCommand`.

The four edit ops

Edits are bounded by design — the optimizer cannot rewrite the document arbitrarily, only emit a sequence of these four operations. Every edit is auditable, reversible, and aggregable.

Op	Target required?	Behavior
`Append`	No	Append `Content` to end. Empty doc → bare content; non-empty → `\n\n` separator.
`InsertAfter`	Yes	Insert `Content` verbatim after first occurrence of `Target` — no injected separator. The optimizer carries any required leading newline in `Content`.
`Replace`	Yes	Replace first occurrence of `Target` with `Content`. Empty `Content` is rejected — use `Delete` instead.
`Delete`	Yes	Remove first occurrence of `Target`. `Content` ignored.

Edits that target text not present in the document are recorded in PatchApplyReport.FailedEdits with a reason string rather than thrown — the caller decides whether the residual partial patch is still worth gating.

Gate metrics

The gate compares the candidate against the current and the running best. Strict greater-than semantics — ties Reject — to avoid thrashing across statistically equivalent skills.

Metric	Projection	When to use
`Hard`	exact-match accuracy in [0, 1]	Default. Large selection set, binary outcomes are sensitive enough to detect improvement.
`Soft`	graded / partial-credit score	Small selection set where binary signal is too noisy.
`Mixed`	`(1 − w) · hard + w · soft`	When neither pure metric is decisive on its own. `MixedWeight` ∈ [0, 1].

Configuration knobs

Held as a strongly-typed TrainSkillConfig record (Domain.AI/SkillTraining), so the same configuration round-trips through checkpoint resume and across run-restarts.

Knob	Default	Meaning
`Epochs`	3	Outer iterations. SkillOpt experience: 2–4 usually enough.
`StepsPerEpoch`	5	Rollout/reflect/apply cycles per epoch.
`LrStart` / `LrMin`	8 / 1	Maximum and floor edits applied per step.
`LrScheduler`	`"cosine"`	`"cosine"` · `"linear"` · `"constant"`
`TrainBatchSize` / `ValBatchSize`	8 / 16	Rollout batch sizes for the training and gate steps.
`GateMetric`	`Hard`	`Hard` · `Soft` · `Mixed`
`MixedWeight`	`0.5`	Soft weight when `GateMetric == Mixed`.
`Patience`	6	Early stop after this many consecutive Rejects.
`UseSlowUpdate`	`true`	Run the paired longitudinal analysis at each epoch boundary.
`UseMetaSkill`	`true`	Maintain cross-epoch strategy memory via `IKnowledgeMemory`.
`Seed`	`0`	Deterministic batch sampling when non-zero (required for SlowUpdate item-id overlap).

Operational invariants worth knowing

Result.Fail codes are stable, scrubbed strings. skill_training.reflect.proposer_call_failed, skill_training.meta.persist_failed, etc. Raw exception text only flows to structured logs — never into orchestrator surfaces. Defends against HTTP-backed proposers leaking SAS tokens via exception messages.
HasChanges is content equality, not applied-edit count. A Replace where Target equals Content is an applied no-op — it should not trigger spurious gating or checkpoint writes.
Skill strings capped at 256 KB (MaxSkillLength = 262_144). Catches runaway-rewrite bugs before they triplicate into the GateResult and downstream audit log.
Gate scores must be finite. NaN or Infinity at the gate boundary throws — a corrupted upstream aggregation should fail loudly, not silently push every candidate into Reject.
Cross-field validator: BestStep ≤ GlobalStep. Catches checkpoint-resume bugs that reset GlobalStep without resetting BestStep.
FP determinism note. The orchestrator re-projects current and best scores via IGateEvaluator.SelectGateScore on every gate call rather than persisting projected values — a checkpoint round-trip (REAL → text → REAL) can flip Accept/Reject by 1 ULP otherwise.
HasAcceptedAny on the run result. When false, the run made no progress; BestSkill equals the initial skill and BestScore is 0. Surfaced explicitly so a Reject-only run doesn't masquerade as having a meaningful "best".

Plugging in real proposer + runner

The subsystem ships complete except for two deliberately empty slots. Both need an actual LLM behind them, so the template cannot fill them in for you:

IPatchProposer — the "optimizer". It reads what happened during the trial runs and proposes an edit to the skill document, as a JSON Patch. This is the part that decides what to change.
IRolloutRunner — the "test bench". It takes a candidate version of the skill and runs it against a batch of eval items, so the edit can be scored. This is the part that decides whether the change helped.

Until you supply both, the registered defaults (NotConfiguredPatchProposer / NotConfiguredRolloutRunner) throw on first use — loudly and immediately, rather than silently doing nothing. Skeleton outline:

// Infrastructure.AI/SkillTraining/AgentPatchProposer.cs
public sealed class AgentPatchProposer : IPatchProposer
{
    private readonly IMediator _mediator;
    // ...
    public async Task<Patch> ProposeAsync(ReflectionInput input, CancellationToken ct)
    {
        var result = await _mediator.Send(new RunOrchestratedTaskCommand
        {
            OrchestratorName = "skill-optimizer",
            TaskDescription  = BuildOptimizerPrompt(input),
            // ...
        }, ct);
        return ParsePatchJson(result.FinalSynthesis);  // tolerate prose + balanced-braces
    }
}

Authoring a skill-optimizer SKILL.md with a JSON-schema-constrained output is the natural follow-up — it lets you swap optimizer model deployments without touching code. For IRolloutRunner, wrap the existing IAgentInvoker with a skill-content override hook so the candidate skill content lands in the system prompt for each invocation.

Where to go next

Read src/Content/Application/Application.AI.Common/README.md — Skill Training CQRS section lists every command, validator rule, and pure component.
Run the demo: dotnet run --project src/Content/Presentation/Presentation.ConsoleUI -- --example skill-training — deterministic stubs, no LLM cost.
Cross-reference Chapter 13 — Evaluation Framework: the eval framework is where rollouts come from in production wiring.
Or jump to Chapter 11 — Extending the Harness to author a skill-optimizer SKILL.md and wire a real AgentPatchProposer.
Upstream methodology paper and code: microsoft/SkillOpt.