How to Fine Tune an LLM to Output JSON

How do you fine tune an LLM to output JSON? Write down one JSON schema, build a dataset of real inputs paired with answers that pass that schema, and train a LoRA adapter on those pairs with supervised fine tuning (for example the SFTTrainer in TRL) so the loss falls only on the JSON answer. Then test the tuned model on held out inputs for parse rate and field accuracy, and serve it with constrained decoding switched on, so a malformed reply cannot reach your code. Fine tuning teaches the model what to put in each field. Constrained decoding guarantees the syntax.

That split is the main point of this guide, because many people fine tune when a schema setting in their inference server would have solved the problem, and others rely on a schema setting when the real issue is that the model fills the fields badly. Below you will find how to tell which problem you have, then a five step workflow. Library details come from the TRL, vLLM, llama.cpp and Ollama documentation, checked on November 13, 2026 (TRL 1.18.0, vLLM 0.33.0, PEFT 0.21.3 on PyPI). We did not run a training job for this guide.

Do you need to fine tune for JSON at all?

Start with the cheapest fix and move down the list only when the one above it falls short.

Constrained decoding is built into the common local servers. Ollama accepts a JSON schema in the format field of its chat API, llama.cpp accepts a schema with the --json-schema flag or a response_format field, and vLLM accepts one through its structured outputs options. If your only complaint is broken syntax, try one of those before you collect a single training example. Our Ollama vs vLLM comparison covers which server suits which setup.

What fine tuning fixes and what it does not

Constrained decoding works at sampling time. At each step the server masks out every token that would break the grammar, so the output always parses. It cannot make the model understand your domain. If the model does not know that "net 30" belongs in a payment_terms field, a grammar will still let it write a valid but wrong value.

Fine tuning works on the weights. After training on enough good pairs (there is no official minimum, so grow the set when your evaluation shows gaps), the model learns your field names, your conventions for missing data and your edge cases. It still has no hard guarantee of valid syntax, which is why the last step of this guide turns constrained decoding back on.

There is one more reason to keep the two jobs apart. The paper Let Me Speak Freely? (Tam and colleagues, 2024) reported a significant decline in reasoning performance when models were held to strict output formats, and stricter constraints tended to hurt more. If your task needs the model to think before it answers, give it room to do that, then extract or convert the answer into JSON, or add a reasoning field that comes before the final fields in your schema.

Step 1: Pin down the schema

Write the schema as code before you write any data, so the same definition drives your dataset checks, your evaluation and your server. A Pydantic model is the easiest route in Python, and both the Ollama and vLLM docs use one in their examples.

from typing import Literal
from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    total: float
    currency: Literal["USD", "EUR", "GBP"]
    due_date: str | None
    line_items: list[str]

schema = Invoice.model_json_schema()

Decide now how the model should handle missing information. A nullable field (str | None) with a rule such as "use null when the document does not say" is far easier to learn than a model that invents a date. Keep key names short and stable, because every rename later means relabeling your data.

Step 2: Build a clean training set

TRL's SFTTrainer accepts a conversational prompt and completion format, where the prompt holds the system and user messages and the completion holds the assistant reply. For JSON work this is the format to use, because by default TRL computes the loss on the completion tokens only, so the model is graded on the JSON and not on copying your input text.

Save one example per line in a JSONL file:

{"prompt": [{"role": "system", "content": "Extract the invoice as JSON matching the Invoice schema. Use null when a field is missing."}, {"role": "user", "content": "ACME Tools, invoice 4471, total 1,250.00 USD, due 2026-12-01. Items: drill, bits."}], "completion": [{"role": "assistant", "content": "{\"vendor\": \"ACME Tools\", \"invoice_number\": \"4471\", \"total\": 1250.0, \"currency\": \"USD\", \"due_date\": \"2026-12-01\", \"line_items\": [\"drill\", \"bits\"]}"}]}

A few rules make the difference between a model that learns the format and one that learns your mistakes:

• Validate every target. Run each completion through Invoice.model_validate_json() and drop any row that fails. One broken example teaches broken output.
• Use real inputs. Messy scans, odd date formats and half empty forms belong in the set, because that is what the model will see later.
• Include the hard cases on purpose. Add inputs with missing fields (so the model learns to write null), inputs with several line items, and inputs that look similar but belong to a different category.
• Keep the JSON plain. No Markdown code fences, no commentary before or after. The target should be exactly the string you want back.
• Keep the system prompt the same across training and serving, so the model links that instruction to the format.
• Hold out a test split of real examples that never appear in training.

If you build the set with the Hugging Face datasets library, our guide to using Hugging Face Datasets covers loading JSONL, splitting and pushing to the Hub. Labeling by hand is slow, so many teams draft labels with a larger model and then review them; if you do, the validation pass above is not optional.

Step 3: Train a LoRA adapter with TRL

A LoRA adapter is usually enough for a format and extraction task, and it keeps memory needs low. Our LoRA vs QLoRA guide explains when the 4 bit QLoRA route is worth it. This sketch follows the patterns in the TRL SFTTrainer documentation:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=SFTConfig(
        output_dir="qwen3-invoice-json",
        learning_rate=1e-4,
        num_train_epochs=3,
        max_length=1024,
    ),
    train_dataset=data["train"],
    eval_dataset=data["test"],
    peft_config=LoraConfig(),
)
trainer.train()
trainer.save_model()

Points from the docs that matter for JSON:

• With a prompt and completion dataset, completion_only_loss is on by default. If you use a plain messages dataset instead, set assistant_only_loss=True, which needs a chat template with generation markers (TRL patches known families such as Qwen3 for you).
• TRL suggests a higher learning rate, around 1e-4, when training adapters, compared with the 2e-5 default for full training.
• max_length defaults to 1024 tokens and longer sequences are cut from the end. If your inputs are long documents, raise it, or the model will train on JSON that has lost its closing braces.
• When you start from a base model rather than an instruct model, align the end of sequence token with the chat template (TRL documents an eos_token setting for this). A model that never learned to stop will append text after the closing brace.

Prefer a faster or lower memory path? Unsloth plugs into the same TRL trainer, and our torchtune vs Unsloth comparison and guide to fine tuning an LLM locally cover hardware and setup in more depth. The PEFT library supplies the LoraConfig used above.

Step 4: Measure parse rate and field accuracy

Loss going down does not tell you whether the JSON is usable. Run the tuned model on your held out test split with constrained decoding switched off, so you see what the weights alone produce, and score three things:

1. Parse rate: the share of replies that load with json.loads().
2. Schema pass rate: the share that pass Invoice.model_validate_json().
3. Field accuracy: for each key, how often the value matches the label.

import json
from pydantic import ValidationError

parsed = valid = 0
for reply in replies:
    try:
        json.loads(reply)
        parsed += 1
        Invoice.model_validate_json(reply)
        valid += 1
    except (json.JSONDecodeError, ValidationError):
        pass
print(f"parse {parsed / len(replies):.1%}, schema {valid / len(replies):.1%}")

Run the same test on the untuned base model with the same prompt. If the tuned model barely beats it, your data is the bottleneck, not the training settings. Per field accuracy usually points straight at the problem: one weak field means you need more examples of that case.

Step 5: Serve with constrained decoding as a safety net

Once the weights fill the fields well, let the server guarantee the syntax. Pass the same schema from Step 1.

• Ollama: set format to Invoice.model_json_schema() in the chat call. The Ollama structured outputs docs also suggest repeating the schema in the prompt and setting temperature to 0, and they note that Ollama's cloud does not support structured outputs yet.
• llama.cpp: start llama-server with --json-schema or --json-schema-file, or send response_format with a JSON schema to its OpenAI compatible endpoint. Under the hood the schema becomes a GBNF grammar, described in the llama.cpp grammar guide. Our llama.cpp how to covers building and serving.
• vLLM: pass extra_body={"structured_outputs": {"json": schema}} or use response_format. The vLLM structured outputs page uses xgrammar or guidance as backends, notes that the old guided_json style fields were removed in v0.12.0, and lists the LM Format Enforcer backend as deprecated from v0.33.0.

vLLM can also serve your LoRA adapter on top of the base model without merging. For Ollama and llama.cpp you merge the adapter and convert to GGUF first, the same route described in our local fine tuning guide.

Common mistakes

• Training on reasoning and JSON in one blob when you only want JSON back, or the reverse. Decide which, and make every target follow that choice.
• Mixing key orders and styles. Pick one order and keep it, including how you write numbers, dates and empty lists.
• Testing only with the grammar on. That hides how often the weights alone get it wrong, which is the number that tells you whether training worked.
• Letting truncation cut the targets. Check the longest tokenized example against max_length.
• Changing the system prompt after training. The model learned the format alongside that exact instruction.

If your output is a single label rather than an object, the simpler route in our guide to fine tuning an LLM for classification may suit you better.

FAQ

Can I get JSON output without fine tuning?

Yes, and you should try that first. Ollama, llama.cpp and vLLM all accept a JSON schema and constrain sampling so the reply always parses. Fine tune only when the values in the fields are wrong, when the prompt full of format rules has grown too long, or when a small model needs to match a larger one.

Does fine tuning guarantee valid JSON?

No. A well tuned model produces valid JSON on almost every input you tested, but nothing in the weights prevents a stray token on an input it has never seen. Constrained decoding at serving time is what gives the guarantee, so use both.

How many examples do I need?

No official number exists. Start with a set you can fully validate, measure parse rate and per field accuracy on held out data, and add examples for the specific cases that fail. A narrow schema with clean inputs needs far fewer examples than a wide schema built from messy documents.

Should I use LoRA or full fine tuning for structured output?

LoRA is the usual choice. Output format and field filling are narrow behaviors that an adapter learns well, training needs much less memory, and you can keep several adapters for different schemas on one base model. Full fine tuning makes more sense when you also need new domain knowledge across many tasks.

Does forcing JSON make the model less accurate?

It can. The Let Me Speak Freely? study found reasoning accuracy dropped under strict format restrictions, and stricter formats hurt more. For tasks that need reasoning, let the model reason first (in a separate step or an earlier field) and keep the final JSON simple.

Which base model should I start with?

Pick an instruct model small enough for your hardware that already follows instructions reasonably well, then test it with your schema before training. Our guide to open source LLM models to run locally lists current options and their licenses.