Skip to content

// category · 5 tools · 2 open source

Datasets

Open corpora and dataset tooling for pretraining and evaluation. Data quality and documentation matter as much as model architecture.

Hugging Face Datasetshuggingface.coHub of hundreds of thousands of datasets plus a library for loading and streaming them efficiently.DatasetsFree★ 22k

Dolmagithub.comAI2's open pretraining corpus and toolkit used to train OLMo, with tools for curating large text datasets.DatasetsOpen source★ 1.6k

Common Crawlcommoncrawl.orgNonprofit that maintains a free, open archive of web crawl data used in most large-scale pretraining corpora.DatasetsFreeHN 2

LAIONlaion.aiNonprofit producing open multimodal datasets, models and tools for democratising machine learning research.DatasetsFreeHN 2

FineWebhuggingface.coLarge, carefully filtered and deduplicated English web dataset for LLM pretraining, documented with ablations.DatasetsOpen source

More categories

Open-Weight Models ML Frameworks Training & Fine-tuning Interpretability Research Platforms