6 Questions Before You Let a Developer Fine-Tune Your LLM

BigDataBoutique's 2026 fine-tuning guide is blunt about the most common mistake teams make: most teams asking whether to fine-tune should not fine-tune yet, since the honest sequence in 2026 runs prompt engineering first, then a real retrieval-augmented generation pipeline, and only then fine-tuning, reserved for shaping behaviour, style, structured output, and refusal patterns rather than injecting facts that change weekly. Databricks' fine-tuning guide frames the decision test just as directly: fine-tuning should only happen when there is a concrete, demonstrated gap between what the base model delivers today and what production actually requires, not because fine-tuning sounds more sophisticated than prompting.

If a developer is proposing to fine-tune your company's LLM anyway, six specific questions separate a project scoped correctly from one that quietly burns budget on a model that degrades once it reaches production. Asking them before signing a custom AI model training contract costs nothing and can save months of rework.

BigDataBoutique identifies four situations where fine-tuning actually moves the needle: structured output reliability when prompt-only solutions fall short, a defined style or behaviour pattern, refusal handling, and cases where retrieval alone cannot shape how the model responds. Everything outside those four situations is usually better solved by fixing the prompt or building a proper retrieval pipeline first, since fine-tuning is expensive to reverse once it is baked into a model's weights.

A developer who proposes fine-tuning before demonstrating that prompting and generative AI development services built around retrieval have already been tried and found insufficient is skipping a step that exists specifically to save you money. Ask them directly what failure mode they observed that prompting and RAG could not fix, and expect a specific answer, not a general preference for fine-tuning as a default approach.

SitePoint's 2026 fine-tuning guide flags a data quality problem that quietly wrecks otherwise well-designed projects: label leakage, where information from the expected output bleeds into the input field, inflates evaluation metrics while producing a model that performs worse than it appears to in testing. The fix requires a held-out test set built specifically to avoid this contamination, not just a random split of the same dataset used for training.

The data provenance question matters just as much as data quality. Ask exactly which systems the training data is pulled from, who has access to the resulting dataset and the fine-tuned model's weights, and whether the developer or the base model provider retains any copy of your data after training completes. A vague answer here is one of the clearest signals to ask for a written data handling plan before development starts.

Catastrophic forgetting, where a model's general capabilities degrade as it specialises on narrow training data, is the single most cited risk across 2026 fine-tuning research. SitePoint recommends mixing 5 to 10 percent general instruction-following data into the training set specifically to preserve broad capabilities, a technique DigitalApplied's 2026 business guide calls a replay buffer. AgamiSoft's 2026 production guide is specific about testing: evaluate the fine-tuned model against general capability benchmarks, not just the domain-specific evaluation set, since a model can look excellent on its target task while quietly losing ground everywhere else.

A developer who cannot describe a specific held-out regression test for general capability, separate from the domain-specific accuracy metric, does not yet have a plan for catching this before it reaches production. This is worth asking about explicitly rather than assuming it is covered by standard testing.

BigDataBoutique states plainly that LoRA and QLoRA, lightweight adapter methods that train a small set of additional parameters rather than the full model, are the only realistic paths for most 2026 projects, reserving full fine-tuning for cases that truly need it. SitePoint's cost breakdown shows why: a 7 billion parameter model needs roughly 14 gigabytes just for the weights in full fine-tuning, with optimiser states pushing total VRAM requirements to 48 gigabytes or more, and full fine-tuning also carries the highest risk of catastrophic forgetting of any method.

The model training approaches comparison covers how LoRA, QLoRA, and full fine-tuning trade off against budget in more depth. A developer defaulting to full fine-tuning without a specific reason your project needs it is very likely proposing a more expensive and riskier approach than the task requires.

This is the question most quotes skip entirely. BigDataBoutique warns that when a hosted provider updates their base model, a fine-tuned adapter built on top of the previous version may degrade silently, with no error message or obvious signal that something has changed, and recommends planning quarterly revalidation as a standing commitment rather than a reactive fix. Treating training configurations, random seeds, and dataset snapshots as version-controlled artefacts, the same discipline applied to application code, is what makes that revalidation possible without starting from scratch each time.

Ask directly what the revalidation cadence will be, who owns it once the initial project ends, and what the rollback plan looks like if a base model update breaks the fine-tuned adapter in production. A developer without a clear answer here is not planning for a cost that is coming regardless.

DigitalApplied's 2026 business guide notes fine-tuned models typically carry inference costs two to four times base model pricing, a figure that rarely appears in an initial project quote focused only on the training run itself. BigDataBoutique goes further, recommending buyers budget three to five times the initial training cost for ongoing lifecycle ownership over the following 12 months: adapter versioning, rollback planning, retraining cadence, and base-model drift management are recurring costs, not one-time work that ends when the model first ships.

One cost-saving pattern worth asking about directly is model distillation: using a stronger frontier model to generate synthetic training data, then fine-tuning a smaller model on that data, which DigitalApplied reports can deliver roughly ten times cheaper inference with quality close to the frontier teacher model. The Gen AI vetting questions post covers the broader set of questions worth asking any generative AI developer alongside these six fine-tuning specific ones.

Fine-tuning a company LLM is not a one-time purchase, it is an ongoing operational commitment with a real maintenance tax that most initial quotes leave out entirely. The six questions above surface whether a developer has actually planned for that commitment or is scoping the project as if training the model is the whole job.

The portfolio red flags post covers what to look for in a candidate's past work before you even get to these six questions. Hire an AI developer who can answer all six with specifics rather than general reassurance, and who prices the lifecycle cost into the proposal from the start.