Common Crawl
Nonprofit web crawling and archive organization
From Wikipedia, the free encyclopedia
The Common Crawl Foundation (Common Crawl) is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public.[1][2] Access to the data is free on Amazon Web Services, but users may incur storage and compute costs. [3]
 | |
| Type of business | 501(c)(3) non-profit |
|---|---|
| Founded | 2007 |
| Headquarters | San Francisco, California; Los Angeles, California, United States |
| Founder | Gil Elbaz |
| Managing director | Rich Skrenta |
| URL | commoncrawl |
Content license | Apache 2.0 (software) [clarification needed] |
Common Crawl was founded by Gil Elbaz.[1][2]
The data had mostly been primarily used by researchers and some startups until the 2020s, when AI companies started training large language models using the data.[4] In November 2025, an investigation by The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and it was not honoring requests from publishers to have their content removed from its databases.[4]
History
Common Crawl was founded in 2007 in San Francisco.[5] It began publishing its crawls in 2011.[6][additional citation(s) needed]
By 2013, sites like TinEye were building their products off of Common Crawl.[7][8] The crawl reduces the reliance of companies and researchers on Google, which has the biggest dataset.[7][8] Common Crawl was designed to have more and fresher data that was more efficient to analyze and utilize than the Wayback Machine created by the Internet Archive.[8][7]
By 2015, 1.8 billion webpages were on the Common Crawl, which started by crawling a list of URLs donated by the search engine Blekko.[9] They use Amazon Web Services, which provides some of its services for free, allowing computing costs to average $2-4000/month.[9] The Common Crawl website listed 30 studies based on Common Crawl data.[9]
Before 2023, Common Crawl was not very well known outside of academic researchers who utilize the data.[5] Common Crawl received its first requests to redact information in 2023 and increasingly started seeing its crawler, CCBot, blocked.[5] In 2023, it began receiving significant financial support from AI companies, including Anthropic and OpenAI, each of which donated $250,000.[4] It was also used to train Google DeepMind's large language model Gemini.[10] By April 2023, Common Crawl was capturing 3.1 billion webpages, with an estimated 5% of pages before 2021 containing hate speech or slurs.[11]
As of 2024, Common Crawl had been cited in more than 10,000 academic studies.[12] By 2024, The Pile and Common Crawl had been the two main training datasets being used to train AI models.[13][14]
In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and when it said that it was honoring requests from publishers to have their content removed from its databases.[4] It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies.[4] As of 2025, Reisner found that CCBot was the most widely-blocked bot by the top 1000 websites.[4]
A 2026 article in LWN.net discussed an advantage to services like Common Crawl being that it can limit the scraping costs to websites by allowing companies and researchers to download the data from Common Crawl instead of scraping it themselves.[15]
In April 2026, Common Crawl experimentally began to distribute its data through Hugging Face Storage Bucket, in addition to its standard storage on Amazon S3. [16]
Organization
Peter Norvig and Joi Ito have served on the advisory board.[8] Rich Skrenta is the executive director.[4]
It has received funding almost exclusively from the Elbaz Family Foundation Trust until 2023 when it started receiving donations from the AI industry.[4]
Refined versions
A number of organizations take raw Common Crawl data and refine it into datasets that exclude edgy content or are otherwise higher-quality for their purposes, such as FineWeb, DCLM and C4.[4]
Colossal Clean Crawled Corpus
Google version of the Common Crawl is called the Colossal Clean Crawled Corpus, or C4 for short. It was constructed for the training of the T5 language model series in 2019.[17] As of 2023, there were some concerns over copyrighted content in the C4 as well as racist content.[18][17] A 2024 study found that 45% of content was explicitly restricted by websites' terms of service to be used for purposes like AI training by for-profit companies.[12]