How to Fine Tune an LLM for a Classification Task

To fine tune an LLM for a classification task, load a small open model with a classification head (AutoModelForSequenceClassification in Transformers), set your labels with num_labels, id2label and label2id, and train a LoRA adapter with PEFT using task_type=TaskType.SEQ_CLS so the new head is trained and saved too. Measure the result with macro F1 and a confusion matrix on a held out test set, and compare it against a cheap baseline before you call it a win.

That covers the core recipe for text classification. The rest of this guide explains when an LLM is the right tool at all, how the two main approaches differ, how to shape the data, and the mistakes that quietly ruin classifiers. For the general mechanics of training on your own hardware (tools, VRAM paths, export), see our guide to fine tuning an LLM locally.

Should you use an LLM for classification at all?

Classification means mapping a piece of text to one label (or several) from a fixed list: spam or not spam, which support queue a ticket belongs in, the sentiment of a review, the topic of a document. Large models are good at this, but they are not always the cheapest way to get there. Try these in order:

1. A prompt with no training. Give a chat model the label list and a few examples. If that already scores well on your test set, you may not need to fine tune.
2. A small encoder model. Encoder models such as ModernBERT have a ready made ModernBertForSequenceClassification class and train fast on a single GPU. The official Hugging Face text classification guide walks through the same pattern with DistilBERT on IMDb movie reviews.
3. Few labeled examples. The SetFit library fine tunes sentence transformer models for classification from very little data. Its documentation reports that with 8 labeled examples per class on the Customer Reviews sentiment dataset it is competitive with fine tuning RoBERTa Large on the full 3,000 example training set.
4. A decoder LLM with LoRA. This is the subject of this guide. It makes sense when labels depend on long context, subtle reasoning or domain language, when you already serve an open model like Qwen and want one stack, or when a smaller model has plateaued.

If you have never trained a classical baseline, do that too. A TF IDF plus logistic regression model in scikit-learn takes minutes and tells you how hard the problem really is.

Two ways to fine tune an LLM for text classification

There are two common designs, and they behave differently.

For most fixed label problems the classification head is the better default. Every prediction comes back as a probability for each label, so you can set a confidence threshold, route uncertain items to a person and plot precision against recall. Label generation is easier if you already have an SFT pipeline, and it lets the model add a short reason, but you pay for decoding and you must handle replies that are not one of your labels.

Step 1: define the labels before you collect data

Most failed classifiers fail here. Write a one paragraph definition for each label with two or three borderline examples, and decide what happens to items that fit no label (an explicit "other" class is often better than forcing a choice).

Then decide the problem type, because it changes the loss function:

• Binary classification: two labels, such as relevant or not relevant. Use num_labels=2.
• Multiclass: exactly one of several labels. Use num_labels equal to the label count.
• Multilabel: any number of labels per item, such as tagging a ticket with both "billing" and "bug". Transformers configs accept problem_type="multi_label_classification", which switches the loss from cross entropy to a per label sigmoid loss, and your labels become lists of 0s and 1s.

Step 2: build and split the dataset

A classification dataset is just text and a label. A CSV or JSONL file with text and label columns loads directly into Hugging Face Datasets.

• Label quality beats quantity. Have two people label the same 100 items and compare. If they disagree often, fix the label definitions before training.
• Split by source, not by row. If several messages come from the same customer or thread, keep them in the same split, or the test score will be inflated by near copies.
• Keep three splits: train, validation for choosing settings, and a test set you only touch at the end.
• Look at the class balance. Write down how many examples each label has. A model that always predicts the majority class can look accurate on unbalanced data.

Step 3: train a classification head with LoRA

Here is a minimal setup with a small open model. Qwen3 has a Qwen3ForSequenceClassification class in Transformers, so AutoModelForSequenceClassification picks it up automatically.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import LoraConfig, TaskType, get_peft_model

labels = ["billing", "bug", "feature_request", "other"]
id2label = {i: l for i, l in enumerate(labels)}
label2id = {l: i for i, l in enumerate(labels)}

model_id = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForSequenceClassification.from_pretrained(
    model_id, num_labels=len(labels), id2label=id2label, label2id=label2id
)
model.config.pad_token_id = tokenizer.pad_token_id

config = LoraConfig(task_type=TaskType.SEQ_CLS, r=16, lora_alpha=32,
                    lora_dropout=0.05, target_modules="all-linear")
model = get_peft_model(model, config)
model.print_trainable_parameters()

Three details matter:

• The pad token. Decoder models often ship without one, and the classification head needs a padding token id to find the last real token in each padded sequence. Set it on both the tokenizer and the model config.
• The warning about score.weight. When you load the model you will see a message that some weights were newly initialized. The PEFT troubleshooting guide explains that this is the randomly initialized classification head and is expected.
• Saving the head. That same guide warns that if the head is not trainable and saved, reloading gives you a different random head and completely different results. Passing task_type=TaskType.SEQ_CLS lets PEFT handle this for standard heads; for an unusual head name, list it in modules_to_save. The PEFT task types reference lists SEQ_CLS as text classification.

From there, tokenize with truncation, pass a DataCollatorWithPadding, and train with the standard Trainer, exactly as in the Hugging Face text classification guide. A learning rate around the usual LoRA range, two to four epochs and evaluation on the validation set each epoch is a sensible start. Pick the checkpoint with the best validation macro F1, not the lowest loss. If memory is tight, see QLoRA vs LoRA and our VRAM guide.

Step 4 (alternative): train the model to generate the label

If you prefer the generative route, format each example as a short conversation: the system or user message holds the instructions, the label list and the text, and the assistant reply is only the label. Train it with TRL's SFTTrainer the same way you would any chat fine tune, as described in our code generation fine tuning guide.

At inference, use greedy decoding, cap the output at a few tokens, and normalize the reply (lowercase, strip punctuation) before matching it to your label list. Anything that does not match should fall back to "other" or go to review. Keep the label names short and distinct so they do not share a first word.

Step 5: evaluate with the right metrics

Accuracy alone hides problems on unbalanced data. Use scikit-learn's classification_report, which prints precision, recall and F1 for each label plus a macro average (the unweighted mean across labels) and a weighted average (weighted by how many examples each label has).

• Macro F1 is the best single number when every class matters, because a rare class counts as much as a common one.
• A confusion matrix shows which labels the model mixes up, which usually points straight back to unclear label definitions.
• Per class recall matters when missing one label is costly, such as fraud or safety reports.
• The baseline comparison. Report the prompt only score and the classical baseline next to the fine tuned number. If the gain is small, the simpler system may be the better choice.

Handling class imbalance

If one label is rare, try these before collecting more data:

• Collect or oversample the rare class in the training split only. Never touch the test split.
• Use class weights in the loss so mistakes on rare labels cost more. This needs a small custom compute_loss in a Trainer subclass.
• Tune the decision threshold. With a classification head you get probabilities, so for binary classification you can move the cutoff away from 0.5 to trade precision for recall.

How to fine tune an LLM for a specific task beyond classification

The same thinking carries over to other narrow tasks: define the output exactly, build clean examples, hold out a test set, and measure the result with a metric that matches the job. Extraction tasks that tag spans of text use token classification (TaskType.TOKEN_CLS in PEFT), and tasks that produce free text use SFT with a generative model. If your task is writing code, our code generation guide covers execution based evaluation, and if you want to see how adapters are stored, our safetensors explainer covers the file format PEFT saves.

Common mistakes

• Forgetting the pad token, which causes errors on batches larger than one.
• Not saving the classification head, so the reloaded model predicts noise.
• Leaking near duplicates between train and test, which makes scores look better than reality.
• Choosing checkpoints by loss instead of the metric you care about.
• Skipping the baseline, so you cannot tell whether the LLM was worth it.

FAQ

Can you fine tune an LLM for text classification?

Yes. Transformers provides sequence classification classes for many decoder models, including Llama and Qwen, and PEFT supports LoRA for them through TaskType.SEQ_CLS. You add a classification head, train it together with a LoRA adapter, and get a probability for every label in one forward pass.

How do I fine tune an LLM for binary classification?

Use the same setup with num_labels=2 and two entries in id2label. Track precision and recall for the positive class, and tune the probability threshold on the validation set rather than assuming 0.5 is right.

Is a classification head better than generating the label?

For a fixed label list it is usually the better default: outputs are always valid labels, every label gets a probability, and inference needs one forward pass. Generating the label is useful when you also want a short explanation or one model that handles several tasks.

How many examples do I need?

There is no fixed number. Start with a few hundred well labeled examples per class, check the validation macro F1, and add data where the confusion matrix shows mistakes. With only a handful of examples per class, SetFit or a prompt with examples is often a better starting point.

Should I use BERT or an LLM for classification?

Try a small encoder like ModernBERT first if your texts are short and the labels are clear. It is faster and cheaper to train and serve. Move to a decoder LLM with LoRA when the encoder plateaus, the texts are long, or the labels need more world knowledge.

Why does my reloaded classifier give random predictions?

Most likely the classification head was not saved with the adapter, so a new random head was created on load. Set task_type=TaskType.SEQ_CLS in your LoRA config, or add the head's module name to modules_to_save, then retrain and save again.