The quote was sitting right there in the CTO’s inbox: $45,000 to fine-tune their customer support model so it would “follow company tone more consistently.” The vendor had attached a slide deck with architecture diagrams. There was even a timeline. Six weeks, delivered.
The CTO forwarded it to me and asked: does this make sense?
I asked to see the system prompt first.
It was 3,400 tokens. Forty-seven numbered rules. Rules about tone nested inside rules about format nested inside rules about topics the model was not allowed to discuss. Rule 31 told the model to be concise. Rule 38 told it to “always provide thorough context.” Nobody had noticed the contradiction. The model was not misbehaving. It was doing its best with a prompt that was actively fighting itself.
There was no fine-tuning needed. There was a prompt that needed to be rebuilt from scratch.
The Instinct Is Understandable
When your AI feature starts producing inconsistent output, the first diagnosis that surfaces tends to be: the model needs more training on our data. It sounds logical. The model doesn’t know us well enough. Give it more examples, teach it our voice, our rules, our edge cases. That’s fine-tuning.
The instinct makes sense. It’s also wrong about 70 percent of the time.
The research on this is fairly consistent: roughly seven out of ten LLM behavior problems can be resolved through better prompt design, clearer instructions, or restructured context. Fine-tuning is a tool for a specific class of problems. Most teams reach for it before they’ve exhausted the cheaper options.
This matters because fine-tuning is not a quick fix. It’s a commitment. And the teams who don’t understand that before they start are the ones who end up six months later with a model that’s expensive to maintain, locked to a base model checkpoint they can’t easily upgrade, and still not behaving the way they wanted.
What Fine-Tuning Actually Does (and Doesn’t Do)
Fine-tuning adjusts the weights of a model by training it on additional examples. It changes how the model behaves by internalizing patterns from your data, rather than reading instructions at inference time.
That sounds powerful, and it is, but the framing people get wrong is this: fine-tuning is for form, not facts.
You fine-tune to change how the model responds, the structure, the style, the formatting, the refusal patterns, the consistent application of a tone across thousands of interactions. You do not fine-tune to inject knowledge that changes. If your business logic, your product catalog, your policy documents shift regularly, fine-tuning doesn’t help. The information you baked in during training is frozen at that checkpoint. You’d need to retrain every time anything substantive changes.
That’s the first misconception. The second is that fine-tuning replaces retrieval. It doesn’t. A fine-tuned model that also pulls from a live knowledge base outperforms either approach on its own. The two are not competitors.
The Order of Operations Most Teams Skip
There’s a sequence that the practitioners who get this right follow. It goes: Prompt, then RAG, then fine-tuning, then distillation. The teams that jump to step three without working through one and two are almost always solving the wrong problem.
Prompting first. Before you conclude that the model can’t do what you need, ask whether you’ve actually told it precisely what you need. This sounds obvious. It is not, in practice. A system prompt that grew organically over three months of iteration is often a document in conflict with itself, full of instructions that were added to fix one problem and accidentally created another. A 4,000-token prompt with fifteen rules is a Jenga tower. Rule twelve quietly stops being obeyed once you add rule sixteen.
The fix, before anything else, is to audit and rebuild the prompt. Separate your constraints by category. Test with specific failure cases. Most of what looks like a model problem is a prompt problem.
RAG second. If the model is failing on factual accuracy, domain knowledge, or anything that changes over time, retrieval-augmented generation is the answer, not fine-tuning. Your model doesn’t need to memorize your knowledge base. It needs to be able to read the relevant parts of it at inference time. RAG infrastructure has matured significantly. It’s commodity at this point, not an experiment.
Fine-tuning third. Once you’ve genuinely exhausted the first two approaches and you still have consistent behavioral failures that no amount of prompting holds reliably, now you can have an honest conversation about fine-tuning. The clearest signal: you need deeply consistent tone or structure across high volumes of interactions, and you have a stable, narrow use case with enough labeled examples to train on properly.
What counts as enough? The honest number is 10,000+ carefully labeled examples. Not 200. Not 1,000. Teams that go into fine-tuning with thin datasets spend the run learning that they don’t have enough data, which is an expensive way to learn it.
Our AI consulting work with startup teams consistently surfaces this pattern: the conversation about whether to fine-tune only becomes useful once you’ve established that prompting has been genuinely attempted, not just tried once and declared insufficient.
The Hidden Costs Nobody Puts in the Slide Deck
The vendor quote the CTO received had a number: $45,000. What it didn’t have was the full cost picture.
Fine-tuning costs break down differently than most engineering costs. GPU hours, the compute cost of running the actual training, are typically the smallest line item. The real spend is in data preparation, evaluation, iteration runs when the first attempt doesn’t work, and ongoing maintenance.
That last one is worth saying clearly: a fine-tuned model is a recurring dependency, not a one-time asset. When the base model you trained on gets updated, your fine-tune doesn’t automatically benefit from the improvements. You’re locked to your checkpoint. If you want to move to a better base model, you restart the process. Budget for that before you start.
There’s also the evaluation cost that teams consistently undercount. One practitioner framing I find accurate: most teams spend 80 percent of their fine-tuning budget on training and 20 percent on evaluation. The ratio should be reversed. A poor eval is how you discover, after the training run, that your model learned the wrong patterns. That discovery costs you the entire run. We’ve written about why evaluation is the work most teams skip before making the fine-tuning commitment.
For LoRA and QLoRA approaches on open-source models, the numbers look more manageable: a 7B model with LoRA runs roughly $1,000 to $3,000 in compute. But that’s still not the total cost. Add dataset creation, evaluation infrastructure, and the time of whoever is doing this, and the number climbs.
The question is not “can we afford the fine-tuning run?” The question is “can we afford the full system, on an ongoing basis, across model upgrades?”
When Fine-Tuning is Actually the Right Call
To be clear about this: fine-tuning is not a mistake. It’s a powerful tool used at the wrong stage by most of the teams I see reach for it. Used correctly, the economics can be compelling.
A fine-tuned 7B model running narrow classification tasks can cost less than $800 per million inferences. The equivalent prompted model on a frontier API, paying for all those instruction tokens on every single call, can run over $11,000 per million. At production scale and volume, that gap is not academic.
Fine-tuning makes sense when your use case is narrow and stable, when the behavior you need is consistently reproducible from your examples, when you have the labeled data to support a proper training run, and when your volume is high enough that the per-inference token savings justify the upfront cost and the ongoing maintenance commitment.
It also makes sense when latency is a hard constraint. A smaller fine-tuned model is faster than a frontier model reading a 4,000-token prompt on every call. If you’re building something where response time matters, that tradeoff becomes real.
The checklist I’d work through before recommending fine-tuning to any team: Have you rebuilt and audited your prompt? Have you tried RAG for anything knowledge-dependent? Do you have 10,000+ high-quality labeled examples? Is your use case narrow and stable enough that training data won’t be obsolete in six months? Do you have a plan for maintaining the fine-tune across base model updates?
If you can answer yes to all of those, fine-tuning is probably the right conversation. If you can’t, it isn’t. Not yet.
What the CTO Did
We rebuilt the system prompt from scratch. Forty-seven rules became twelve, organized by category, tested against the failure cases that had prompted the original vendor conversation. The model started behaving consistently within two weeks. Cost: a few dozen engineering hours and the inference tokens for testing.
The $45,000 stayed in the budget. More importantly, the team didn’t take on a six-month maintenance commitment for a problem that didn’t require it.
This is what AI-powered development done properly looks like, not the most technically impressive solution, but the right solution for where the product actually is. The impressive solution comes later, when the use case is stable enough to justify it.
Fine-tuning will probably be the right call for this team eventually. When their volume scales, when their use case has narrowed and hardened through real usage, when they’ve accumulated the labeled data to do it properly. That’s not now. Now, there’s a working product.
The most expensive way to learn that you picked the wrong tool is to finish building with it.