Dolma

github.com · Datasets

Open source

#2 in Datasets

AI2's open pretraining corpus and toolkit used to train OLMo, with tools for curating large text datasets.

#pretraining#ai2#corpus#toolkit

Visit Dolma ↗GitHub ↗

About the Datasets category

Open corpora and dataset tooling for pretraining and evaluation. Data quality and documentation matter as much as model architecture.

Dolma is one of 5 datasets tools indexed on Axiomi. Facts on this page come from the tool's official site and public APIs; prices and features change, so confirm details on the official website before you commit.

// related

Related tools

All Datasets →

Common Crawlcommoncrawl.orgNonprofit that maintains a free, open archive of web crawl data used in most large-scale pretraining corpora.DatasetsFreeHN 2

FineWebhuggingface.coLarge, carefully filtered and deduplicated English web dataset for LLM pretraining, documented with ablations.DatasetsOpen source

Hugging Face Datasetshuggingface.coHub of hundreds of thousands of datasets plus a library for loading and streaming them efficiently.DatasetsFree★ 22k

LAIONlaion.aiNonprofit producing open multimodal datasets, models and tools for democratising machine learning research.DatasetsFreeHN 2

Megatron-LMgithub.comNVIDIA's research codebase for training transformer models at scale with tensor, pipeline and sequence parallelism.Training & Fine-tuningOpen source★ 18.1k

nanoGPTgithub.comAndrej Karpathy's minimal repository for training and fine-tuning medium-sized GPTs, a classic learning resource.Training & Fine-tuningOpen source★ 63.6k