How to Fine-Tune LLMs in 2026 — A Practical Guide

When fine-tuning is the right choice

Fine-tuning is powerful but often unnecessary. Use it when you've exhausted simpler approaches:

• Fine-tune when: You need consistent style/tone that prompt engineering can't achieve. You need to teach domain-specific terminology or reasoning. You want to reduce prompt length (and cost) by baking instructions into the model. You need better performance on a specific task type.
• Don't fine-tune when: You need the model to know specific facts (use RAG instead). You haven't tried thorough prompt engineering first. Your task is already well-served by a base model. You don't have at least 100 high-quality training examples.

A useful mental model: fine-tuning changes how the model behaves (style, format, reasoning patterns). RAG changes what it knows (facts, documents, data). Most projects need RAG. Some also need fine-tuning.

LoRA and QLoRA: fine-tuning on a budget

Full fine-tuning updates all model parameters — requiring enormous GPU memory and compute. LoRA (Low-Rank Adaptation) changes this by training only a small set of adapter weights:

• Memory reduction — LoRA typically trains <1% of total parameters. A 7B model that would need 28GB+ for full fine-tuning can be LoRA-trained on 8GB VRAM.
• QLoRA — Combines LoRA with 4-bit quantisation. Further reduces memory requirements. A 7B model can be QLoRA-trained on 6GB VRAM — meaning a consumer GPU or even Google Colab free tier.
• Quality — For most tasks, LoRA/QLoRA fine-tunes perform within 1-3% of full fine-tuning. The quality gap is small enough to be irrelevant for most applications.
• Speed — Training completes in minutes to hours depending on dataset size, not days. A typical 1000-example fine-tune on a 7B model takes 15-30 minutes on a single GPU.

Cost: fine-tuning a 7B model with QLoRA on a cloud GPU (A100) costs approximately $2-10 per training run. This is not expensive — you can iterate quickly.

Preparing your training data

Training data quality is the single biggest determinant of fine-tuning success. Guidelines:

• Format — Most fine-tuning tools expect instruction/response pairs in JSONL format: {"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
• Quality over quantity — 500 carefully curated examples typically outperform 10,000 auto-generated ones. Every example should represent exactly the output quality you want the model to produce.
• Diversity — Cover the range of inputs you expect. Include edge cases, different phrasings, and varying difficulty levels. A training set that only covers the easy cases produces a model that fails on hard ones.
• Consistency — All examples should follow the same style, format, and quality bar. Inconsistent training data produces inconsistent model behaviour.
• Test set — Reserve 10-20% of your examples for evaluation. Never train on your test set. This is how you objectively measure whether fine-tuning helped.

The training process

A standard fine-tuning workflow using the most common tools:

1. Choose a base model — Start with a strong open-source model: Llama 3 8B for general tasks, CodeLlama for code, Mistral 7B as a strong all-rounder. See our open-source LLMs guide for options.
2. Set up training — Use Hugging Face's trl library (most popular) or Axolotl (configuration-driven, less code). Both support LoRA/QLoRA out of the box.
3. Configure hyperparameters — Key settings: learning rate (start with 2e-4 for LoRA), epochs (1-3 for most datasets), LoRA rank (16-64, higher = more capacity but more parameters), batch size (4-16 depending on GPU memory).
4. Train — Run the training. Monitor loss curve — it should decrease steadily. If it spikes or flatlines, adjust learning rate.
5. Evaluate — Test on your held-out set. Compare against the base model with your best prompt. If fine-tuned quality isn't clearly better, iterate on data quality before adding more data.
6. Merge and deploy — Merge LoRA weights into the base model for inference. Deploy via Ollama, vLLM, or any standard inference tool.

Evaluation: is it actually better?

The hardest part of fine-tuning is knowing whether it worked. Automated metrics (loss, perplexity) tell you if the model learned the training data, not if it's actually useful. Practical evaluation approaches:

• Side-by-side comparison — For each test input, generate responses from both the base model (with your best prompt) and the fine-tuned model. Have a human rate which is better. You need at least 50 comparisons for statistical significance.
• Task-specific metrics — If your task has objective metrics (classification accuracy, extraction precision, format compliance rate), measure them directly.
• LLM-as-judge — Use a strong model (GPT-4, Claude) to evaluate responses against your quality criteria. Cheaper than human evaluation but less reliable for subtle quality differences.
• Regression testing — Check that fine-tuning didn't hurt performance on tasks the base model handles well. Models can "forget" general capabilities if fine-tuning data is too narrow.

A fine-tuned model that's only marginally better than a well-prompted base model may not be worth the maintenance overhead. The bar should be clear, measurable improvement.