How to Fine Tune a Thinking LLM or Instruct Model

To fine tune an LLM thinking or instruct model, first decide which behavior you want at the end. If you want short, direct answers, start from an instruct model and train on plain question and answer pairs. If you want the model to keep reasoning before it answers, start from a thinking or hybrid model and put a reasoning trace in every assistant turn, in the exact field its chat template expects. Unsloth's advice for hybrid Qwen models is to keep at least 75% of your examples as reasoning examples, or the model will start skipping its thinking. Then train only on the assistant tokens, and test with the sampling settings the model card recommends, never greedy decoding.

The rest of this guide covers the parts that go wrong in practice: telling the variants apart, formatting the traces, the chat template traps that silently delete reasoning from your training data, and how to check the result. For the general setup (hardware, LoRA settings, exporting to GGUF), read our guide to fine tuning an LLM locally first.

Thinking, instruct and hybrid models: which one you have

Open weight model families now ship in three shapes, and each one is fine tuned a little differently.

The Qwen3-4B-Thinking-2507 model card says plainly that the model supports only thinking mode, and that it is normal for output to contain only a closing </think> tag because the default template already inserts the opening one. The Instruct-2507 versions are the opposite: TRL's documentation describes them as a simpler Qwen3 template without reasoning_content parsing or the enable_thinking flag. Hybrid Qwen models sit in between. The Qwen3.6 model cards say the model thinks by default and that the old /think and /nothink soft switches from Qwen3 are no longer officially supported, so you control it through the chat template instead. Small Qwen3.5 models differ again: TRL notes that the 2B and smaller sizes default to thinking off, while 4B and larger default to thinking on.

Check the model card and the chat_template in tokenizer_config.json (or chat_template.jinja) before you write a single training example. The template is the ground truth for how a trace must look.

Which starting point fits your task

Pick the variant by the output you want, not by which one scores higher on a leaderboard.

• Extraction, classification, formatting, tone or style tasks: start from an instruct model. A thinking trace adds tokens and latency without helping a task that has no multistep logic. Our guide to fine tuning an LLM for a classification task covers that route.
• Maths, code, planning, diagnosis or any task where the steps matter: start from a thinking or hybrid model and keep the traces.
• One model that does both: use a hybrid model and train on a mix, so it still answers directly when you turn thinking off.
• A base model you want to teach to reason: supervised traces alone can work, but this is where reinforcement learning with GRPO earns its place (see below).

What a training example looks like

Use the conversational format (a messages list) so the trainer applies the model's own chat template. For hybrid and thinking Qwen models, the chat template reads a reasoning_content field on the assistant message. You can see this in the template shipped with Qwen3-8B: if reasoning_content is a string it is used, and otherwise the template tries to split the content on </think>.

{"messages": [
  {"role": "user", "content": "A tank holds 240 liters and drains 8 liters a minute. How long until it is empty?"},
  {"role": "assistant",
   "reasoning_content": "240 divided by 8 is 30, so 30 minutes.",
   "content": "It empties in 30 minutes."}
]}

A direct answer example for the same model simply leaves reasoning_content out, or sets it to an empty string, which renders as an empty <think> block. That empty block is exactly what Qwen's nonthinking mode produces at inference time, so the model learns that "empty thinking, then answer" is a valid pattern.

gpt-oss uses a different layout. OpenAI's cookbook on fine tuning gpt-oss with Hugging Face explains that each assistant turn in its example dataset has a thinking field for the reasoning and a content field for the final reply, and the Harmony chat template maps those to the analysis and final channels.

Keep traces honest and consistent. A trace that reaches the wrong intermediate number and then states the right answer teaches the model that its reasoning does not need to support its conclusion.

The 75% rule for keeping reasoning

The most useful single number here comes from Unsloth. Its Qwen3 guide says you can fine tune a hybrid model on nonreasoning data, but this may affect its reasoning ability, and recommends 75% reasoning and 25% nonreasoning examples to retain it. Its own conversational notebook follows that split, using NVIDIA's open math reasoning data for the reasoning share and a general chat dataset for the rest. The Qwen3.5 fine tuning guide repeats the advice: mix reasoning examples with direct answers and keep a minimum of 75% reasoning, or drop reasoning entirely if you don't need it.

Treat 75% as a starting point that Unsloth recommends, not a measured law for every task. If your evaluation shows the model has stopped thinking on hard prompts, raise the reasoning share. If it overthinks simple prompts, add more direct answer examples.

Where reasoning traces come from

You have three realistic sources.

1. Existing reasoning datasets on the Hugging Face Hub, such as FreedomIntelligence/medical-o1-reasoning-SFT (used as an example in the TRL SFT documentation) or HuggingFaceH4/Multilingual-Thinking, which the gpt-oss cookbook uses. Check each dataset's licence before commercial use.
2. Traces you generate with a larger reasoning model on your own prompts, then filter so only traces that reach a verified correct answer stay. Read the teacher model's licence and terms, since some restrict using outputs to train other models.
3. Traces you write by hand for a few hundred hard cases. Slow, but the quality is under your control.

Small datasets can be enough for style changes. The gpt-oss cookbook trains on 1,000 examples and says that is usually more than sufficient for a model that has already had extensive post training.

Chat template traps that delete your reasoning

This is where most thinking model fine tunes quietly fail. The default inference templates are written to throw old reasoning away, which is right for chatting and wrong for training.

• Qwen3 drops thinking from earlier turns. The Qwen3 model card lists "No Thinking Content in History" as a best practice, and the template only renders a <think> block for assistant turns after the last user message. In a multiturn training example, every earlier assistant trace vanishes.
• DeepSeek-R1-Distill strips the trace. TRL's chat templates page says the original template "drops everything up to and including </think>, which would silently remove the reasoning from the training target."
• Single tag parsing. Qwen's stock template splits on </think> alone, so content with only a closing tag can be cut in the wrong place. TRL's patched Qwen templates require both tags first.

The fix in TRL is one flag. Set assistant_only_loss=True in SFTConfig and TRL computes loss only on assistant tokens and, for known families, swaps in a patched training template that always emits the thinking block and marks the assistant output for masking. The families listed include Qwen3 (and its Instruct-2507 variant), Qwen3.5, Qwen3.6, Qwen3.8, GPT-OSS and DeepSeek-R1-Distill. For any other model, check that its template contains the {% generation %} markers TRL needs.

Before training, print one rendered example with tokenizer.apply_chat_template(messages, tokenize=False) and read it. If you can't see your trace in the output, the model won't learn it.

A minimal setup with TRL and LoRA

Here is the shape of a small supervised run on a hybrid Qwen model with PEFT LoRA. Replace the model id and the dataset path with your own.

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files="mixed_traces.jsonl", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen3-8B",
    train_dataset=dataset,
    args=SFTConfig(
        assistant_only_loss=True,
        max_length=4096,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        num_train_epochs=1,
        output_dir="qwen3-8b-thinking-lora",
    ),
    peft_config=LoraConfig(r=16, lora_alpha=16, target_modules="all-linear"),
)
trainer.train()

Reasoning traces are long, so the sequence length matters more than it does for plain chat. Examples cut off mid trace teach the model to stop thinking without an answer. Measure your longest rendered examples first and either raise max_length or drop the outliers.

Unsloth wraps the same trainer with lower memory use. Its Qwen3.5 guide lists bf16 LoRA memory of about 3 GB for 0.8B, 10 GB for 4B, 22 GB for 9B and 56 GB for 27B, and advises against 4 bit QLoRA on Qwen3.5 because of larger than normal quantization differences. For MoE models it disables router training by default. If you are deciding between LoRA and QLoRA for another family, our LoRA vs QLoRA guide covers the tradeoff.

For gpt-oss, the cookbook loads the model with Mxfp4Config(dequantize=True) and, because it is a mixture of experts model, adds expert projection layers through PEFT's target_parameters. It reports about 18 minutes of training on one H100 for its 1,000 example run.

When GRPO beats supervised traces

Supervised fine tuning copies the traces you give it. If you don't have good traces but you can check answers automatically (unit tests for code, exact numbers for maths, a schema for JSON), reinforcement learning is often the better tool. GRPO samples several answers per prompt and rewards the ones that score above the group average, so the model finds its own reasoning paths. Unsloth's Qwen3 guide describes a GRPO notebook that first runs a short supervised stage so the model learns the output format before the reward stage begins. Our explainer on GRPO vs PPO covers how the method works.

How to test the result

Evaluate on held out prompts in both modes if your model is hybrid.

• Use the recommended sampling. The Qwen3 model card says not to use greedy decoding in thinking mode because it can cause performance degradation and endless repetition, and recommends temperature 0.6, top p 0.95 and top k 20. Unsloth's Qwen3.6 page lists temperature 1.0 for general thinking tasks and 0.6 for precise coding.
• Test thinking off. Render prompts with enable_thinking=False (in llama.cpp, --chat-template-kwargs '{"enable_thinking":false}') and confirm the model still answers directly.
• Watch trace length. If average thinking length jumps after training, your traces were longer than the model's natural style.
• Check the exported model. Unsloth warns that a model that behaves worse in another runtime most often has the wrong chat template or end of sequence token. Use the same template you trained with when you export to GGUF for llama.cpp, Ollama or LM Studio.

For running the newest hybrid Qwen release with its thinking controls, see how to run Qwen 3.8 locally.

Common mistakes

FAQ

Should I fine tune the thinking or the instruct version?

Fine tune the instruct version when you want short, direct output and the task has no real multistep logic. Fine tune the thinking or hybrid version when you want the model to keep reasoning, and include a trace in your training examples.

Will fine tuning remove a model's reasoning?

It can. Unsloth says training a hybrid Qwen model on nonreasoning data may affect its reasoning ability. Mixing at least 75% reasoning examples with your direct answers is its recommended way to keep it.

How do I format reasoning traces for Qwen fine tuning?

Put the trace in a reasoning_content field on each assistant message and the final reply in content, then let the trainer apply the chat template. Print one rendered example to confirm the <think> block appears.

Can I fine tune gpt-oss reasoning?

Yes. OpenAI's cookbook fine tunes gpt-oss-20b with TRL and LoRA on a dataset whose assistant turns have a thinking field and a content field, which map to the analysis and final channels. Its example changes the language the model reasons in.

Do I need SFT or GRPO for a thinking model?

Use supervised fine tuning when you have good example traces. Use GRPO when you can score answers automatically but don't have traces, or when you want the model to improve its own reasoning on a checkable task.

Why does my fine tuned thinking model loop or repeat itself?

The most common causes are greedy decoding, a chat template at inference that differs from the one used in training, or traces cut off during training. Check those three before retraining.