7 Mistakes Businesses Make Fine-Tuning Their First LLM

Researchers behind the Orthogonal Subspace Fine-Tuning paper presented at ICLR 2026 fine-tuned a model that could solve 29 percent of grade-school maths problems across eight new tasks. Afterwards, its maths score dropped to 2 percent. The fine-tuning succeeded on every target task and destroyed a capability the model already had. That single result captures the central problem with first fine-tuning projects: the thing that goes wrong is usually not the thing the team was measuring.

Fine-tuning is the technique businesses reach for when a general model does not behave the way they want, and it is frequently the wrong tool for the problem they actually have. These are the seven mistakes that most reliably sink a first custom AI model training project, drawn from the failure patterns that show up repeatedly in production postmortems, and what to do instead of each one.

This is the most common and most expensive mistake, and it comes from a reasonable-sounding assumption: train the model on your documents and it will know your business. In practice, fine-tuning adjusts style and pattern recognition, but factual recall from training data stays unreliable, particularly for specific figures, dates and policies. A model fine-tuned on medical literature does not know the facts in that literature the way a retrieval system does. The rule most experienced teams work to is that fine-tuning is for form, not facts.

The practical consequence is that fine-tuning bakes information into weights that goes stale the moment the underlying data changes, and there is no way to update a single policy without retraining. If the answer depends on data that changes, prices, product specifications, internal policies, support tickets, the correct architecture is retrieval, which is why most enterprise generative AI development services engagements start by building a retrieval pipeline rather than a training run.

The recommended order of operations across current practice is consistent: fix the prompts, build a real retrieval pipeline, write evaluations, and only then consider fine-tuning. Teams that skip to fine-tuning because it sounds like the more serious engineering option usually discover months later that their real problem was retrieval, and no volume of additional training data would have fixed it. Retrieval-augmented generation is the correct first choice for roughly 80 percent of enterprise language model applications in 2026, according to consultancy Winder.AI's decision framework.

The test that separates the two cases cleanly is whether the model already produces the right answer but in the wrong format, tone or reasoning pattern. If it does, and prompting cannot fix it, fine-tuning is warranted. If the model is simply wrong about your business, more training data is not the answer; better retrieval is.

Full fine-tuning updates every weight in the model and is rarely the right call in 2026. LoRA and QLoRA, which adapt a small set of additional parameters instead, cover the overwhelming majority of legitimate business use cases at a fraction of the compute and storage cost, and QLoRA improvements now allow an eight-billion-parameter model to be fine-tuned on a single twelve-gigabyte consumer graphics card. Full fine-tuning also carries the highest risk of catastrophic forgetting, so the expensive option is frequently also the more fragile one.

Beyond the parameter-efficiency question, the technique should follow the data available: supervised fine-tuning for labelled input-output pairs, preference methods such as DPO or ORPO when you have comparative judgements rather than single correct answers, and reinforcement-style approaches for tasks with a verifiable reward. Choosing between these is its own scoping exercise, and the model training approaches post covers how the choice shifts with budget and data volume.

The evaluation harness must exist before training starts. Without it, there is no way to tell whether a new checkpoint is better than the last one, and teams end up shipping on the strength of a few impressive-looking sample outputs. A defensible evaluation scores the base model on the same examples, the same rubric and the same judge as the fine-tuned version, because the fine-tune's value is the delta between them, not its absolute score. A 0.3-point lift measured on 50 examples is judge noise, not a result.

Building and maintaining that harness is ordinary engineering discipline rather than research work, which is why it belongs in the scope of the machine learning development services engagement rather than being treated as an optional extra. The real cost of fine-tuning is not the training compute; it is the data curation, evaluation and twelve-month lifecycle ownership that sits around it.

This is the mistake the ICLR 2026 maths result illustrates. A fine-tune posts a strong gain on the held-out support-ticket set, the team celebrates, and two weeks later the on-call thread reports that the assistant handles refunds beautifully but can no longer do arithmetic and now complies with a jailbreak the base model refused. The holdout win was real; it was simply the only thing anyone measured. Catastrophic forgetting means the model loses prior capabilities as it acquires new ones, and it does not announce itself on the metric the team is watching.

The mitigation is to test for regression on general capability alongside the target task, and to red-team the checkpoint for safety behaviour the base model handled correctly. Where forgetting does appear, the standard remedies are reducing epochs, lowering the learning rate, reducing LoRA rank, and mixing roughly 5 to 10 percent general instruction-following data into the training set to preserve broad ability.

A holdout set only earns its keep under strict train and evaluation isolation: no overlap, no near-duplicates, and no leakage through synthetic generators that saw the evaluation data. The common failure is subtler than copying rows between sets. A holdout drawn from the same source as the training data will report a lift that reflects how well the model learned that source's quirks rather than how well it will handle real production inputs, which is why fine-tunes that look excellent offline routinely disappoint after launch.

Overfitting to spurious patterns in the fine-tuning dataset is the underlying mechanism, and it is particularly damaging because it produces exactly the metric movement the team hoped to see. The check worth running is whether the model has learned the target behaviour or has learned surface-level pattern matching, which usually means probing edge cases in the domain rather than adding more examples from the same distribution.

The final mistake precedes all the others. Teams write a job specification around fine-tuning because that is the term they encountered first, then hire a specialist who delivers exactly what was asked for on a problem that needed retrieval, or hire a generalist who has run a training script but has never built an evaluation harness or diagnosed forgetting. The strongest signal in a technical interview is not whether a candidate can name LoRA and DPO; it is whether they will tell you not to fine-tune when the evidence points elsewhere.

Because generative AI work demos impressively and verifies poorly, portfolio review matters more here than in most AI hiring. The Gen AI portfolio red flags post covers how to spot demos that were never evaluated rigorously, and the Gen AI vetting questions post covers the broader questions worth asking before an engagement starts.

Every mistake on this list traces back to the same root cause: training started before anyone established what success would look like or whether training was the right intervention at all. The teams that get fine-tuning right treat the training run as the last step in a sequence, not the first, and they are willing to abandon it entirely when the evaluation shows the problem was retrieval all along.

If you are weighing a first fine-tuning project and are not certain the technique matches the problem, hire an AI developer who will build the evaluation harness before proposing a training approach. The most valuable thing a specialist can tell you at the start of this work is that you do not need it yet.