Northwind Support LoRA

A support assistant that has to follow a policy handbook usually carries that handbook in its system prompt. Every single request then pays for it, roughly a thousand tokens of policy text and output contract, on every call, forever. The question this project set out to answer is whether that cost can be paid once at training time instead, by moving the policy and the output format into the weights.

Northwind Outdoors is a fictional retailer with a 20-entry policy knowledge base. For each customer message the assistant must reply in 2–4 sentences grounded in the actual policy numbers, refuse to invent policy, hand off to a human on injury, fraud, chargeback, legal threat, or an order over $1000, and close with exactly one fenced JSON ticket. That ticket is what makes the whole thing measurable: intent, priority, escalation, cited policies, and next action are all scored by rule against gold labels.

The dataset is generated scenario-first: sample an intent, a variant, and complication flags, then derive the gold labels by rule, so ground truth is exact by construction rather than annotated after the fact. About 38% of conversations carry a follow-up, and some of those flip the labels mid-conversation, so escalation has to be tracked dynamically. One customer phrasing per scenario is reserved for test and never appears in training; the generator asserts zero verbatim overlap.

Qwen3-4B was then LoRA fine-tuned (r=16, alpha=32, all attention and MLP projections) against the short prompt, with one example per assistant turn and loss masked to that turn. Four arms were scored on the same 200 held-out conversations.

• Cut the system prompt from 1,079 to 99 tokens (91%) while beating the prompted baseline on every accuracy metric
• Intent accuracy 64.0% → 98.5%; policy-citation F1 58.6 → 90.5; escalation F1 70.0 → 90.3
• Escalation recall 58.3% → 87.5%, the metric that matters when a case involves injury or fraud
• Halved inference time: 2.93s → 1.47s per example on a single RTX 5070 Ti
• Reports its own weakness: 48% of fine-tuned prose is memorised from templates, so accuracy claims are scoped to format and policy fidelity

Three things came out of this, and the third is the uncomfortable one.

The adapter replaces the prompt outright, at 91% fewer prompt tokens while beating the prompted baseline on every accuracy metric. Strip the prompt without fine-tuning and the base model collapses to 1% schema-valid JSON and 3% intent accuracy, which is a clean measure of exactly how much work the prompt was doing and how much the 132 MB adapter absorbed.

Escalation recall is where the baseline actually fails. Both prompted arms sit near 90% precision but only ~57% recall: when they escalate they are right, but they miss four in ten cases that need a human. For injury and fraud, recall is the number that matters.

And nearly half the fine-tune's prose, 48% of it, reproduces a training answer verbatim. On template-generated data that is expected, but it means these numbers are evidence about format and policy fidelity, not about writing quality. Reporting it matters more than hiding it.

• Qwen3-4B
• PyTorch
• PEFT / LoRA
• Transformers
• Gradio
• Hugging Face