Best LLM Pretraining Dataset in 2026: Top Picks

The best LLM pretraining dataset for most people in 2026 is FineWeb-Edu: 1.3 trillion tokens of educational English web text from Hugging Face, free under the ODC-By licence, with ready made 10 billion token samples you can download on a normal connection. If you are training at scale and need more unique tokens, look at DCLM-Baseline (about 4 trillion tokens) or Nemotron-CC (6.3 trillion tokens). If your lawyers care about where every document came from, pick Common Pile v0.1, which contains only public domain and openly licensed text. Add FineMath for maths and The Stack v2 for code.

There is no single winner because pretraining data is a mix. Labs blend web text with code, maths, papers and books, and change the blend near the end of training. Below are the open datasets worth knowing, what each one is good for, and how to load them without downloading tens of terabytes.

Best LLM pretraining dataset picks at a glance

All figures come from each dataset's own card or paper. Token counts depend on the tokenizer, so treat them as rough sizes rather than exact matches.

1. FineWeb-Edu: the best default

FineWeb-Edu is the dataset to start with unless you have a reason not to. Hugging Face built it by asking Llama 3 70B Instruct to rate web pages for educational value, training a small classifier on those ratings, and running it over all of FineWeb. The FineWeb-Edu dataset card says pages scoring below 3 were dropped, which removed 92% of the data and left 1.3 trillion tokens. A looser version that keeps pages scoring 2 or more holds 5.4 trillion tokens.

According to the FineWeb paper, models pretrained on FineWeb-Edu show much better results on knowledge and reasoning benchmarks such as MMLU and ARC than models trained on other open web datasets. That matters most for small models, which have little capacity to waste on spam and boilerplate.

The practical win is the samples. The card lists sample-10BT, sample-100BT and sample-350BT configs, so you can grab 10 billion tokens instead of the whole thing. That is plenty for a model of a few hundred million parameters. Our guide to training a small LLM from scratch walks through a full run with nanoGPT style code.

2. FineWeb: when you need raw volume

FineWeb is the parent set. Its dataset card describes more than 18.5 trillion tokens (the first release had 15 trillion) of cleaned and deduplicated English text from Common Crawl, released under ODC-By 1.0. Version 1.4.0, dated July 2025, added six crawls covering January to June 2025, so it is reasonably fresh.

Pick plain FineWeb over FineWeb-Edu when your token budget is larger than the filtered set can supply, or when you want broader writing styles than educational pages. Hugging Face also publishes the processing code (the datatrove library) and the small ablation models it trained, which makes FineWeb unusually easy to audit.

3. DCLM-Baseline: strong English data at 7B scale

DCLM (DataComp for Language Models) is a benchmark for data curation. Its authors extracted a 240 trillion token pool from Common Crawl and tested filtering recipes on it. The winning recipe became DCLM-Baseline, which the DCLM-Baseline dataset card describes as a 4 trillion token, 3 billion document set under CC-BY-4.0.

The DCLM paper reports that a 7B model trained from scratch on 2.6 trillion DCLM-Baseline tokens reached 64% five shot accuracy on MMLU, and found that filtering with a trained model was the key step. Allen AI later used DCLM-Baseline as the web portion of its OLMo 2 pretraining mix: the olmo-mix-1124 card lists 3.70 trillion DCLM tokens out of 3.90 trillion in total.

4. Nemotron-CC: more unique tokens for long runs

Aggressive filtering has a cost. The Nemotron-CC paper points out that FineWeb-Edu and DCLM remove about 90% of the data, which is a problem if you plan to train for something like the 15 trillion tokens used for Llama 3.1. NVIDIA's answer combines several quality classifiers, rewrites some lower quality text with a model, and leans less on heuristic filters.

The paper says the full 6.3 trillion token set matches DCLM on MMLU while containing four times more unique real tokens, and that an 8B model trained for 15 trillion tokens (7.2 trillion of them from Nemotron-CC) beat Llama 3.1 8B by 5 points on MMLU. The data is hosted through Common Crawl rather than Hugging Face, so read the hosting terms before you use it. This is the pick for teams running Megatron-LM or DeepSpeed across many GPUs, not for a weekend project.

5. Dolma: the reproducible research mix

Dolma is Allen AI's open corpus, built to make pretraining research reproducible. The Dolma paper describes three trillion English tokens from web content, scientific papers, code, public domain books, social media and encyclopedic material, and the dataset moved to ODC-By in April 2024. Version 1.7 was used to train OLMo 7B v1.7.

Choose Dolma when you want a mixed corpus that already balances sources, plus the open toolkit used to build it. Because OLMo models publish their data and checkpoints, you can compare your run against a known baseline.

6. Common Pile v0.1: the cleanest licence story

Most web datasets are scraped text whose copyright status is unclear. The Common Pile v0.1 paper takes a different approach: an 8 TB collection of public domain and openly licensed text from 30 sources, including research papers, code, books, encyclopedias, educational materials and audio transcripts. The authors trained two 7B models, Comma v0.1-1T and Comma v0.1-2T, on 1 and 2 trillion tokens, and report performance competitive with Llama 1 and 2 7B at similar compute.

If your model will ship in a product and data provenance is a real business question, start here. Expect to give up some benchmark points compared with the best filtered web sets.

Specialist datasets: maths, code and multilingual

Maths. The best math dataset for LLM pretraining among open options is FineMath. Its dataset card describes 34 billion tokens in FineMath-3+ and a higher quality FineMath-4+ subset of 9.6 billion tokens, with samples overlapping GSM8k, MATH, MMLU and ARC test sets removed.

Code. The StarCoder 2 and The Stack v2 paper describes a corpus built with Software Heritage covering 619 programming languages, plus GitHub pull requests, Kaggle notebooks and documentation. StarCoder2 models were trained on 3.3 to 4.3 trillion tokens of it. For a smaller, ready to use slice, the SmolLM corpus includes python-edu, Python files scored as educational.

Multilingual. FineWeb 2 applies the FineWeb pipeline to more than 1,000 languages and provides filtered data for 1,868 language and script pairs, according to its dataset card.

Synthetic data and LLM dataset generators

Synthetic text is now a normal part of pretraining mixes. The SmolLM corpus card describes Cosmopedia v2 as over 39 million textbooks, blog posts and stories generated by Mixtral 8x7B Instruct, sitting next to a deduplicated 220 billion token FineWeb-Edu slice. If you want to build your own LLM dataset generator, the pattern is the same: write varied prompts (topics, audiences, formats), generate with an open weight model whose licence allows training on its outputs, then deduplicate and filter. Synthetic data works best as a supplement to real text, not a replacement.

How to choose and load a pretraining dataset

1. Match the dataset to your token budget. DeepMind's Chinchilla paper found that for compute optimal training, model size and training tokens should grow together: every time you double the parameters, double the tokens. A model of 100 to 200 million parameters needs a few billion tokens, which the FineWeb-Edu 10B sample covers.
2. Check the licence. ODC-By and CC-BY-4.0 need attribution. Datasets built on Common Crawl are also subject to Common Crawl's terms of use.
3. Stream before you download. With Hugging Face Datasets you can read records without storing the full set:

from datasets import load_dataset

ds = load_dataset("HuggingFaceFW/fineweb-edu", name="sample-10BT",
                  split="train", streaming=True)
for row in ds.take(3):
    print(row["text"][:200])

4. Mix, then anneal. Start with mostly web text, and raise the share of maths, code and high quality sources in the final stage. The Comma v0.1 training card shows exactly this pattern, with a separate cooldown stage.
5. Decontaminate. Remove documents that overlap your evaluation sets, as FineMath does, or your benchmark scores will flatter you.

If you only need to adapt an existing model rather than train one, skip pretraining entirely and read how to fine tune an LLM locally or our guide on fine tuning an LLM for code generation.

FAQ

What are the best free LLM training datasets?

FineWeb-Edu, FineWeb, DCLM-Baseline, Dolma, FineMath and Common Pile v0.1 are all free to download. They differ in licence terms (ODC-By, CC-BY-4.0 or openly licensed sources), so read each dataset card before commercial use.

Which dataset should I use for a small model on one GPU?

Use the FineWeb-Edu sample-10BT config, or TinyStories if you want a tiny story model that trains in hours. Both are covered in our small LLM training guide.

What is a golden dataset in LLM work?

A golden dataset is a small, carefully checked set of inputs with known correct outputs, used to evaluate a model or a pipeline. It is for testing, not pretraining, and it should never leak into your training data.

Is The Pile still a good choice?

The FineWeb team compared against The Pile, C4, SlimPajama, RedPajama2 and Dolma v1.6 and reports that FineWeb trained better models on its benchmark tasks. For new projects, the newer filtered sets above are the stronger starting point.

Do I need terabytes of storage?

Not to start. Streaming and the published samples let you work with a few hundred gigabytes or less. The FineWeb card lists its 10 billion token sample at 27.6 GB.