Updated
Where AI data comes from
Where Do AI Companies Get Their Training Data
Short answer
Mostly from five places: public websites, content they pay publishers for, work they pay people to do, text written by other AI, and private records that businesses let them use for a fee. That last one is the hardest to get.
Where do AI companies get their training data? Most of it comes from five places. The first is public web pages that programs called crawlers copy. The second is content they pay publishers and sites to use. The third is work they pay people to do. The fourth is text that other AI models write. The fifth is private records that businesses license to them.
To license means you let a buyer use a copy for a fee, and you still own the records. The first four sources get most of the news. The fifth is the hardest to find, because only the company that made the records can say yes. This page explains each source in plain words. Then it shows why business records are in such demand. It also shows how an older company can get paid to license its own.
- Five sources, explained
- Checked public sources
- Estimate in 2 minutes
The short answer
What are the five main sources of AI training data?
OpenAI, for example, names three kinds of information its models learn from (OpenAI Help Center). One is what anyone can find on the internet. Another is what it gets through deals with other companies. The last is what users, human trainers and researchers give it or create. In practice, that breaks down into five groups.
1. Public web crawls
Programs called crawlers visit billions of web pages and save the text. The best known public collection is Common Crawl. It is a nonprofit that has given away a free archive since 2007. It holds more than 300 billion pages. It adds 3 to 5 billion new pages each month.
2. Paid content deals
AI companies pay sites and archives for the right to train on their material. In its 2024 filing to go public, Reddit listed data licensing deals worth $203M. Shutterstock earned $104M from AI makers in 2023 alone.
3. Paid human work
Companies hire people to write sample answers. These workers also rank a model's replies and fix its mistakes. This work costs a lot, and it is big business. One company that sells it got $14.3B from Meta. In return, Meta got a 49% stake in that company.
4. Synthetic data
Synthetic data is practice material that one AI model writes for another. Microsoft's Phi-4 report says it used synthetic data at every stage of training. It is cheap to make in large amounts. But it can only repeat patterns a model has already seen.
5. Private business records
These are email, chat, support tickets, contracts and work orders. A company licenses them to an AI maker. Google won a court auction for one airline's old staff email and chat with a $10M bid. One service that helps startups shut down reports close to 100 such archive deals in a year.
Source by source
Why are the free sources getting harder to count on?
For years, the public web did most of the work. OpenAI described GPT-3 in 2020. At that time, a filtered copy of Common Crawl made up 60% of its training mix (Brown et al., 2020). Books, Wikipedia and other web text made up the rest.
That free supply is shrinking in two ways. First, more sites are saying no. Common Crawl's crawler follows each site's robots.txt file. That is a small file that tells crawlers which pages they may visit. So any site owner can block it with two lines.
A 2024 study looked at C4, a popular web dataset (Longpre et al., 2024). A group called the Data Provenance Initiative did the study. It looked at the most active, most important sources in C4. Within about a year, rules like robots.txt had fully blocked AI crawlers from more than 28% of them.
Second, there is only so much of it. Epoch AI is a research group. In 2024 it estimated that, if trends hold, models could use up the public text people have written. It put the date between 2026 and 2032. Our page on AI companies running short of training data explains that forecast and its limits.
Synthetic data helps fill some gaps, but it has a known weak spot. A 2024 paper in Nature tested what happens when models keep training on text that AI wrote. Over time, those models lost the rare details found in real human writing. So labs still need fresh material that people wrote.
The scarce part
Why are private business records the hardest data to get?
Money and computers can grow every other source on the list. Crawlers can run longer. Labs can sign more content deals and hire more workers. They can also make more synthetic text. Private business records do not work that way.
These records exist only inside the companies that made them. They sit behind logins that no crawler can get past. They also hold something the other sources lack. A support ticket shows a real customer, a real problem and the steps that solved it.
A ten-year email archive shows how a business handles orders, delays, fights and hires. Paid workers can copy that style. But they cannot make up ten years of real history.
That puts owners of older, established companies in a strong spot. Say you have 20 or more people and at least 3 years of records. Then you likely hold material no lab has trained on. You can license a copy and still own the records. You also decide what stays out. Our guide to licensing your company's records to AI developers covers the basics in more depth.
Business records in the news
What companies have been paid for their records
These are reported deals for private business records, the fifth source above. They do not promise any price for your company.
Google's top bid at a court auction held in August 2026 for Spirit Airlines' work archive. It holds roughly 100 million emails. It also holds some 500 million Teams messages. Another AI data firm was next in line at $7.5M. The sale still needs a judge's approval.
Chapter11Cases, Oct 2026What a second AI data company offered for the Spirit records after the auction, 25% above Google's bid.
Fortune, Sep 2026Dollars cielo24 got for 13 years of Jira tickets, email and Slack after it shut down, Forbes reported.
Forbes, Apr 2026A buying pledge the firm behind that offer made in October 2026. It covers 12 months of buying and licensing company records.
Company announcement, Oct 2026Share of GPT-3's training mix that came from filtered Common Crawl web pages in 2020. That was before many sites began to block crawlers.
Brown et al., 2020Your part of the supply chain
Which of your records AI companies license
If your company runs on common business software, you likely have several of these already.
| What you have | Usual software | Why buyers ask for it |
|---|---|---|
| Years of company email | Microsoft 365, Gmail, Exchange archives | Shows how business choices get made in writing |
| Work chat logs | Slack, Microsoft Teams | Shows staff solving problems in casual talk |
| Customer support tickets | Zendesk, Freshdesk, Intercom | Pairs real questions with real answers |
| Deal and account notes | Salesforce, HubSpot | Tracks a sale from first contact to close |
| Contracts, bids and reports | SharePoint, Google Drive, Box | Detailed writing from people who know the trade |
| Work orders and job logs | Business planning (ERP) or field service software | Shows tasks done in order, with results |
| Phone and meeting recordings | Zoom, RingCentral, Dialpad | Real spoken back and forth from work |
| How-to docs and source code | Confluence, Notion, GitHub | Teaches steps and how software gets built |
Put a rough price on your part of the supply chain
The number comes from public buyer pricing. We average the programs that match you best. Answer a short quiz. No uploads, no logins.
Stay in control
What to leave out, and how that works
A license covers only what you list. The contract names anything you leave out, and the buyer must follow it.
Personal files about staff
HR records, payroll, benefits and ID scans add risk and bring little value. Leave them out.
Records with special legal rules
Medical records fall under HIPAA, a federal law that protects health privacy. Private legal files need care too. Our page for law firms licensing their records shows how one such field handles it.
Other people's documents
Files you store for clients may belong to them. Check your agreements before you include them.
Mailboxes full of private talk
You can drop whole mailboxes, channels or years. One example is the owner's inbox. Another is a folder about a legal dispute.
Current trade secrets
You can name live pricing, formulas and plans and keep them out. Older material is often worth just as much to buyers. It is also far less sensitive to you.
How we work
From estimate to payment in four steps
DillisData is a data agency. We match you with programs that pay for records like yours. The program pays our fee. We never receive, store or open your records.
Two-minute estimate
Tell us a few quick things about your company. You need no files and no account.
2 minutesProgram match
We test your record profile against each AI data program that needs records like yours. Then we show the top 3 by likely payout.
SecondsSign-up
Each match warns you about any permanent rights before you agree to anything. Joining takes one click.
One clickTransfer and payment
Files move straight from your own systems to the buyer you chose. We are never in the middle. The buyer removes personal details and then pays you. Published payment times range from one week to roughly three months.
Per contractTiming
Why the best prices go to companies that move early
No one can say how long labs will keep paying at today's level. Here is what we do know.
Gaps get filled
Labs pay top prices to cover a blind spot. Once a few companies in your field have covered it, your records matter less to them.
Other sources keep getting better
Better synthetic data or new training methods could lower demand for some records. Nobody knows when.
Stored records cost you money
Old archives take up paid storage. They earn nothing while they sit there.
Old accounts get closed
When you cancel software or retire a server, years of records can go with it. What is deleted can never be licensed.
Questions
Questions owners ask
Do AI companies train on my company's website?
Maybe. Crawlers such as Common Crawl's can collect public pages. You can block many crawlers with your robots.txt file. Crawlers cannot reach private email, tickets and files behind a login.
Is it legal for AI companies to use web pages without paying?
Courts are still arguing about that, and it is not settled. One thing is clear. Licensed data comes with written permission, and that is one reason labs pay for it.
What is synthetic data, in plain words?
It is practice text that an AI model writes instead of a person. It is cheap to make. But research shows it works best mixed with real human writing. It does not fully replace it.
Why would a lab pay for records instead of just making synthetic ones?
Synthetic text can only reflect what a model already knows. Real records show new cases, real mistakes and real fixes. The model has not seen those before.
What happens to our records after we license them?
The buyer copies the records you approved. It removes names and personal details. It uses that copy only in the ways the contract allows. You keep the originals, and you still own them.
Who pays DillisData?
The programs pay our fee. Owners never pay DillisData, and with no signed deal there is no fee at all.
Which companies qualify?
Programs differ. Several want 20 to 50 staff and a few years on file. Others name no size minimum. Larger staffs and more years of records tend to bring bigger offers. Closed or closing companies can still qualify.
Get started
Add your company to the supply chain, on your terms
You now know where AI training data comes from. You also know why private records are the scarce part. Find out what a license to yours might bring in.
Free to your company. We never hold your records. Ranked matches, explained.