How to Fine-Tune an LLM for Code Generation

To fine-tune an LLM for code generation, start from a model that was already trained on code, such as a Qwen2.5-Coder instruct model, rather than a general chat model. Build a dataset of task-and-solution pairs from code you are allowed to use, keep only examples whose code actually runs and passes tests, then train a LoRA adapter with TRL or Unsloth. Judge the result by running the generated code against held-out tests, not by training loss.

That is the whole recipe. The rest of this guide covers the decisions that make it work for code specifically: which base model, which data format, how to build and clean the dataset, and how to measure whether the fine-tune is any better. For the general mechanics of local fine-tuning (hardware paths, tool installation, GGUF export), see our step-by-step guide to fine-tuning an LLM locally.

When fine-tuning for code generation is worth it

Code models are already good at mainstream languages and popular libraries, so a fine-tune has to earn its keep. It tends to pay off when:

• You have internal APIs, frameworks or a DSL the model has never seen, and you want it to use them the way your team does.
• You need a house style, such as your error-handling pattern, logging conventions, test layout or type-annotation rules, applied consistently.
• You want a small, fast model for autocomplete or a narrow job (writing SQL against your schema, generating config files, converting one format to another) that a big general model does slowly or expensively.
• You work in a niche language that is thin in public training data.

It is a poor fit for keeping up with changing facts, such as the latest version of a library's API. Feeding the current docs or the relevant files into the prompt (retrieval) handles that better and does not go stale. Many teams combine the two: a light fine-tune for style and conventions, plus retrieval for the specifics of the codebase.

Step 1: pick a code base model

Start with a model whose pretraining was code-heavy. Its model card tells you what it was trained on, its context length, and its licence.

According to the Qwen2.5-Coder-7B-Instruct model card, the family comes in six sizes from 0.5B to 32B, was trained on 5.5 trillion tokens including source code and synthetic data, and supports a context of 131,072 tokens, although the shipped config is set to 32,768 and longer inputs need the YaRN setting the card describes. The Qwen3-Coder-30B-A3B-Instruct card lists 262,144 tokens of native context and notes it runs in non-thinking mode only.

Two more choices matter:

• Instruct or base? Use the instruct version when you want "describe the task, get the code" behaviour in chat. Use the base version when you are training pure code completion inside an editor.
• Start small. Get the pipeline working on a 1.5B or 7B model, where a training run takes minutes to hours, then scale up. Our VRAM guide helps you size the jump, and the Qwen tool page links the other releases in the family.

Step 2: choose a data format for the task

"Code generation" covers a few different tasks, and each wants a different training format.

Instruction to code (chat)

The most common setup: a natural-language request in, working code out. TRL's SFT documentation supports a conversational prompt-completion format, where the prompt is the user's message and the completion is the assistant's reply:

{"prompt": [{"role": "user", "content": "Write a function that parses our ISO week strings like 2026-W42 into a date range."}], "completion": [{"role": "assistant", "content": "def parse_iso_week(s: str) -> tuple[date, date]:\n    ..."}]}

With prompt-completion data, TRL computes the loss on the completion only by default, so the model learns to write answers rather than to repeat requests. With a plain list of messages, set assistant_only_loss=True in SFTConfig for the same effect; the docs note it needs a chat template with generation markers, which TRL patches automatically for known families such as Qwen3.

Fill in the middle (autocomplete)

Editor autocomplete is a different task: the model sees the code before and after the cursor and fills the gap. The Qwen3-Coder repository documents its fill-in-the-middle prompt as <|fim_prefix|> plus the code before the gap, <|fim_suffix|> plus the code after it, then <|fim_middle|>, with the model generating the missing piece. To teach completion in your own codebase, cut real files at random points into prefix, middle and suffix, and train on that format with the special tokens your model expects.

Bug and fix

Pairs of broken code plus an error message, with the corrected code as the answer, teach repair. Your commit history is a natural source: a failing test, the diff that fixed it, and the commit message describing why.

Step 3: build the code generation dataset

For code, a dataset you can verify beats a bigger one you cannot.

1. Start from code you have the right to use. Your own repositories are the best source for internal APIs and style. If you pull in public code or datasets, check their licences and any usage terms first.
2. Write the task side. For each function, class or snippet, write the request a developer would actually type. Docstrings, issue titles and commit messages are good raw material.
3. Consider synthetic data, carefully. The Magicoder paper introduced OSS-Instruct, which seeds a strong model with real open-source code snippets so that the generated instruction data is more varied and realistic. Its public 75K dataset was generated with an OpenAI model, and its card asks users to follow OpenAI's usage policy, so read the terms of any model you use to generate training data.
4. Execute everything. Run each solution against tests, keep only the ones that pass, and drop anything that does not compile, times out or needs network access. This one filter removes most of the noise that makes code fine-tunes worse.
5. Deduplicate and decontaminate. Remove near-duplicate examples, and remove anything that overlaps with the benchmark problems you plan to test on, or your scores will flatter you.
6. Hold out a test set. Keep a slice of real tasks, with their tests, that never goes into training.
7. Keep secrets out. Strip API keys, credentials and customer data from every example. A fine-tuned adapter can reproduce what it was trained on.

Hugging Face Datasets loads local JSONL files directly, so the cleaned set can stay on your machine.

Step 4: train a LoRA adapter

LoRA, which trains small adapter matrices on top of frozen weights, is the default for code fine-tunes on one GPU; QLoRA does the same on a 4-bit base to save memory. Our LoRA vs QLoRA guide compares them, and full fine-tuning is rarely worth it below a multi-GPU budget.

Here is a minimal TRL and PEFT script for a prompt-completion dataset:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-Coder-7B-Instruct",
    train_dataset=data["train"],
    eval_dataset=data["test"],
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                           task_type="CAUSAL_LM"),
    args=SFTConfig(output_dir="coder-lora", max_length=4096,
                   num_train_epochs=2, learning_rate=2e-4,
                   per_device_train_batch_size=1, gradient_accumulation_steps=8,
                   eval_strategy="steps", eval_steps=50, bf16=True),
)
trainer.train()

A few code-specific settings deserve attention:

• Sequence length. Code examples are long. Set max_length high enough that solutions are not cut off mid-function, and check how many examples get truncated.
• Epochs. One to three passes is usually enough. More tends to memorise your repository instead of learning its patterns.
• Tool choice. Unsloth wraps the same TRL trainers with memory savings on a single GPU, and Axolotl suits YAML-driven, multi-GPU runs. Older tutorials use torchtune, but its repository says it is no longer actively maintained.

After supervised fine-tuning, some teams add a reinforcement learning stage that rewards code for passing tests. Our GRPO vs PPO explainer covers how that works.

Step 5: evaluate by running the code

Training loss tells you the model is fitting the data; it does not tell you the code works. Code evaluation is execution-based.

• pass@k. The HumanEval paper from OpenAI introduced a benchmark that measures functional correctness, whether generated programs pass unit tests, and reported that sampling many attempts per problem solves far more problems than one. pass@1 is the share of tasks solved by a single attempt; pass@k counts a task as solved if any of k attempts passes.
• Stricter public tests. EvalPlus adds HumanEval+ and MBPP+, which its README says carry 80 and 35 times more tests than the originals, so fragile solutions get caught. One command, such as evalplus.evaluate --model <your-model> --dataset humaneval --backend vllm --greedy, generates and scores answers, using vLLM as the backend in that example.
• Your own test suite matters most. Public benchmarks check general skill and catch regressions. Your held-out tasks with real tests tell you whether the fine-tune learned your codebase. Run the base model and the fine-tune on the same tasks and compare.
• Agentic work needs repo-level tests. If the model will edit whole repositories, look at SWE-bench-style tasks, where a patch must make a project's tests pass.

Always run generated code in a sandbox, such as a container with no network and no access to your files.

Step 6: run it in your editor

Merge the adapter or export it to GGUF, load it in Ollama, and point your editor at it. Our guide to using a local LLM in VS Code walks through the setup, and the GGUF quantization guide helps you pick a quant that keeps code quality. If you trained fill-in-the-middle, make sure your editor extension sends the same FIM format.

Common mistakes

• Training on code that does not run. Unverified examples teach the model to produce plausible bugs.
• Testing on what you trained on. Leaked benchmark or repository examples inflate every score.
• Mixing formats. Instruction data, FIM data and chat templates must match how you will actually prompt the model.
• Expecting new library knowledge to stick. Use retrieval for docs that change.

FAQ

What is the best model to fine-tune for code generation?

For most local projects, a Qwen2.5-Coder instruct model in the 1.5B to 14B range is a practical start: it is code-trained, Apache 2.0 licensed (apart from the 3B size) and loads in standard Transformers-based tools such as TRL. Larger mixture-of-experts coders like Qwen3-Coder-30B-A3B need much more memory to train.

What should an LLM code generation dataset look like?

Task-and-solution pairs in the format you will use at inference: chat prompt and completion for instruction following, prefix-suffix-middle for autocomplete, or broken code plus error and fix for repair. Every solution should pass tests, and a held-out slice should never be trained on.

Which LLM fine-tuning methods work for code?

LoRA and QLoRA cover most single-GPU projects. Full fine-tuning updates every weight and needs far more memory. A test-based reinforcement learning stage after supervised fine-tuning can improve correctness further but adds complexity.

Is LLM fine-tuning the same as training?

No. Training from scratch builds a model's knowledge from raw data and costs far more compute. Fine-tuning adapts an existing model with a much smaller, targeted dataset. If you want to see the from-scratch side, read our guide to training a small LLM from scratch.

How do I know my code fine-tune actually improved?

Run the base model and the fine-tuned model on the same held-out tasks with real tests and compare pass rates, then check a public benchmark such as HumanEval+ to make sure general coding did not get worse.

Fine-tuning or RAG for my codebase?

Use fine-tuning for conventions, style and narrow tasks that repeat. Use retrieval for facts that change, like current APIs and file contents. Many setups use both.