How to Fine Tune an LLM with Reinforcement Learning

To fine tune an LLM with reinforcement learning, start from an instruct model, collect prompts whose answers you can check, write reward functions that score each answer, and train with an online method such as GRPO, which samples several answers per prompt and pushes the model toward the ones that beat the group average. If you only have pairs of good and bad answers instead of a checker, use DPO, an offline preference method that needs no sampling during training. Both run as LoRA jobs in TRL or Unsloth on a single GPU for small models.

The rest of this guide is the practical order of work: deciding whether RL is the right tool at all, choosing the method, preparing data, writing rewards that the model cannot game, running the job and reading the logs. If you want the theory behind GRPO and PPO, our explainer on GRPO vs PPO covers it. For general fine tuning setup (hardware, LoRA settings, export), see our guide to fine tuning an LLM locally.

When reinforcement learning beats supervised fine tuning

Supervised fine tuning (SFT) copies examples. You show the model the exact answer you want and it learns to produce text like it. Reinforcement learning works differently: the model writes its own answers, something scores them, and training makes high scoring answers more likely. The Unsloth reinforcement learning guide puts the difference plainly: regular fine tuning maximizes next word prediction, while GRPO optimizes for a reward.

That makes RL the better choice when:

• You can check an answer but cannot easily write the ideal one. Maths with a known result, code with unit tests, JSON that must match a schema, SQL that must return the right rows.
• You want the model to find its own reasoning. DeepSeek showed in the DeepSeek-R1 paper that reasoning behaviors such as self reflection and verification can emerge from reinforcement learning on verifiable tasks, without human written reasoning traces.
• You have preferences rather than answers. Human or model judgments of "this reply is better than that one" fit preference methods like DPO.

SFT is still the right first step when the model cannot do the task at all or does not know your output format. RL only improves behaviors the model already produces some of the time. The Unsloth guide makes the same point: if the chance of a correct answer is zero, RL never works, which is why people start from a model that already follows instructions reasonably well.

The usual order: SFT, then preferences, then verifiable rewards

Open post training recipes tend to stack these stages. AI2's Tulu 3 report describes its pipeline on Llama 3.1 base models as supervised fine tuning, then Direct Preference Optimization, then a method the authors named Reinforcement Learning with Verifiable Rewards (RLVR). You will rarely need all three for one task, but the order is a good default:

1. SFT teaches the task and the output format.
2. DPO nudges style and quality using chosen and rejected pairs.
3. RLVR, usually with GRPO, raises accuracy on answers a program can check.

If you start from a good instruct or thinking model, you can often skip the first two and go straight to RL. Our guide to fine tuning a thinking or instruct model covers how to pick the variant and keep its chat template intact.

Pick the method: GRPO, DPO or PPO

For most people fine tuning an open model at home, the answer is GRPO for checkable tasks and DPO for preference data. PPO is the classic RLHF method, but it trains a separate value model, which costs memory and tuning time. TRL lists GRPO, DPO, RLOO, KTO and reward model trainers in its main docs, and its GRPO trainer docs and DPO trainer docs list every option.

Step 1: build the prompt set

For GRPO you need prompts and whatever your reward functions use to check answers, not finished answers. A typical row has a prompt column and an answer column holding the reference result. TRL passes every extra dataset column to your reward functions as a keyword argument, so you can also add a task column if one dataset mixes maths and code, as the TRL docs show in their multitask example.

Some practical rules:

• Match the difficulty to the model. If the model already gets a prompt right on all eight samples, or wrong on all eight, every answer gets the same score, the group advantage is zero, and that prompt teaches nothing. Prompts the model solves some of the time carry the signal.
• Hold out a test set of prompts the model never trains on, with the same checker.
• Have enough rows. Unsloth's guide suggests at least 500 rows for good results, while noting that small sets can still work.

For DPO, each row needs a prompt, a chosen answer and a rejected answer. TRL accepts both plain text and conversational message lists, and recommends an explicit prompt field.

Step 2: write reward functions the model cannot game

The reward function is the whole job. In TRL, a reward function receives the prompts and completions (plus any dataset columns) and returns one float per completion. You can pass a list of functions; TRL sums them, with optional reward_weights, and logs each one separately.

Split the reward into small checks instead of one big score. Unsloth's guide calls this a rubric: a list of smaller verifiable rewards rather than one final reward. A typical set for a maths task:

import re

def format_reward(completions, **kwargs):
    # 0.5 if the answer sits inside <answer> tags, else 0
    return [0.5 if re.search(r"<answer>.*?</answer>", c, re.S) else 0.0 for c in completions]

def correctness_reward(completions, answer, **kwargs):
    rewards = []
    for c, ref in zip(completions, answer):
        m = re.search(r"<answer>(.*?)</answer>", c, re.S)
        rewards.append(2.0 if m and m.group(1).strip() == str(ref).strip() else 0.0)
    return rewards

Keep the correctness reward clearly larger than the format reward. Otherwise the model learns to produce perfect tags around wrong answers. Other traps to watch for:

• Length rewards. Anything that pays for longer output gets long output. TRL's docs describe how the original GRPO loss carried a response length bias and why later variants changed the normalization.
• Loose parsing. If "the answer is 4, or maybe 5" passes your regex, the model will learn to hedge. Parse strictly and score only one final answer.
• Judge models as rewards. An LLM judge can score open ended tasks, but the policy may learn to please the judge rather than the user. Spot check samples by hand.

For code, run the generated function against unit tests in a sandbox and reward the pass rate. Our guide to fine tuning an LLM for code generation covers execution based checks in more detail.

Step 3: run a GRPO job with LoRA

Here is a minimal TRL run on a small instruct model with a LoRA adapter from PEFT. It assumes a JSONL file with prompt and answer fields and the two reward functions above.

from datasets import load_dataset
from peft import LoraConfig
from trl import GRPOConfig, GRPOTrainer

train = load_dataset("json", data_files="rl_train.jsonl", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    reward_funcs=[format_reward, correctness_reward],
    train_dataset=train,
    args=GRPOConfig(
        output_dir="grpo-lora",
        num_generations=8,
        max_completion_length=512,
        learning_rate=1e-5,
        logging_steps=10,
    ),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()

A few defaults worth knowing, all from the current GRPOConfig reference: eight generations per prompt, a 512 token completion limit, a learning rate of 1e-6 (LoRA runs usually go higher, as above), and a KL penalty (beta) of 0.0, because recent studies found the KL term not essential. The default loss_type is "dapo", a token level normalization; "dr_grpo" is also available. Set use_vllm=True to generate samples with vLLM, which is much faster than plain Transformers generation for long completions.

For scale, TRL's own quickstart trains Qwen2.5 0.5B Instruct on the DeepMath-103K dataset with one accuracy reward, and reports that distributed across eight GPUs the run takes about a day.

Run it on one GPU with Unsloth

Unsloth wraps the same TRL trainers with lower memory use and built in vLLM generation. Its RL guide states that a model of 1.5B parameters or less can be trained with about 5 GB of VRAM, and that roughly 15 GB covers models up to 17B. Its rule of thumb for QLoRA 4 bit GRPO is that a model's parameter count in billions roughly equals the GB of VRAM you need, with longer context needing more, and 16 bit LoRA needing at least four times as much. It also recommends a model of at least 1.5B parameters if you want it to produce thinking tokens.

Unsloth publishes free GRPO notebooks for several models, so a Colab T4 is enough to try the workflow. Our Google Colab fine tuning guide explains the free tier limits, and our LoRA vs QLoRA comparison explains the memory tradeoff. Axolotl and torchtune are alternatives if you prefer YAML configs.

Or run DPO when you have preference pairs

DPO, from the paper Direct Preference Optimization: Your Language Model is Secretly a Reward Model, skips the reward model and the sampling loop. It compares the model's likelihood of the chosen and rejected answers against a frozen reference copy and trains directly on that difference. In TRL you swap in DPOTrainer and DPOConfig; the default beta is 0.1 and the default learning rate is 1e-6. Because nothing is generated during training, DPO is cheaper per step than GRPO, but it can only learn from the answers already in your dataset.

Read the logs: what good training looks like

TRL logs the mean and standard deviation of each reward function, the combined reward, completion lengths and more. Watch these:

• Reward rising slowly. Unsloth's guide says to wait at least 300 steps before expecting the reward to climb, and that good results can take 12 hours or more.
• Reward standard deviation near zero. Every sample scores the same, so there is no learning signal. Your prompts are too easy or too hard, or a reward is broken.
• Completion length exploding or collapsing. Often a sign the model found a shortcut in your reward.
• Format reward maxed while correctness stays flat. The model learned the tags, not the task. Reweight the rewards.

Always read a sample of raw completions every few hundred steps. A rising reward curve with nonsense outputs means the model beat your checker, not your task.

Test the result

Score the base model and the tuned model on your held out prompts with the same checker, then on a few general prompts to see whether it lost other skills. Compare against a simple baseline: the base model with a better prompt, or an SFT run on correct answers sampled from the base model. If RL does not beat those, keep the cheaper option. When you are happy, merge the adapter and export it for Ollama or llama.cpp as described in our local fine tuning guide.

FAQ

Is GRPO better than DPO for fine tuning?

They solve different problems. GRPO needs a way to score answers automatically and learns from the model's own samples, so it suits maths, code and structured output. DPO needs pairs of better and worse answers and trains offline, so it suits tone and preference tuning. Many recipes use DPO first and an RL stage after it.

Do I need SFT before reinforcement learning?

Only if the model cannot produce correct answers at least some of the time, or does not know your output format. RL improves behaviors the model already shows occasionally. A short SFT stage on a few hundred formatted examples is a common way to give it that start.

How much VRAM do I need for GRPO?

It depends on model size, completion length and the number of generations per prompt. Unsloth's guide gives about 5 GB for models up to 1.5B parameters and about 15 GB for models up to 17B with its QLoRA setup. Plain TRL without those optimizations needs considerably more.

Can I fine tune with reinforcement learning on a Mac?

GRPO generates several full answers for every prompt at every step, so it is slow without a fast GPU, and the vLLM generation path in TRL targets GPU servers. On a Mac, DPO is the lighter option because it generates nothing during training. For GRPO, a free Colab notebook or a rented GPU is usually the quicker route.

What is RLVR?

Reinforcement Learning with Verifiable Rewards: RL where the reward comes from a program that checks the answer, such as a maths checker or unit tests, rather than from a learned reward model. The Tulu 3 report named it, and GRPO is the most common way to run it.

How many training steps does GRPO need?

There is no fixed number. Unsloth suggests at least 300 steps before judging a run and notes some tasks need 1,000 or more. Stop when held out accuracy stops improving, not when the training reward looks good.