DillisData Get my estimate

Updated

Where AI data comes from

Where Do AI Companies Get Their Training Data

Short answer

Mostly from five places: public websites, content they pay publishers for, work they pay people to do, text written by other AI, and private records that businesses let them use for a fee. That last one is the hardest to get.

Where do AI companies get their training data? Most of it comes from five places. The first is public web pages that programs called crawlers copy. The second is content they pay publishers and sites to use. The third is work they pay people to do. The fourth is text that other AI models write. The fifth is private records that businesses license to them.

To license means you let a buyer use a copy for a fee, and you still own the records. The first four sources get most of the news. The fifth is the hardest to find, because only the company that made the records can say yes. This page explains each source in plain words. Then it shows why business records are in such demand. It also shows how an older company can get paid to license its own.

  • Five sources, explained
  • Checked public sources
  • Estimate in 2 minutes

See what your data is worth

Progress

The short answer

What are the five main sources of AI training data?

OpenAI, for example, names three kinds of information its models learn from (OpenAI Help Center). One is what anyone can find on the internet. Another is what it gets through deals with other companies. The last is what users, human trainers and researchers give it or create. In practice, that breaks down into five groups.

1. Public web crawls

Programs called crawlers visit billions of web pages and save the text. The best known public collection is Common Crawl. It is a nonprofit that has given away a free archive since 2007. It holds more than 300 billion pages. It adds 3 to 5 billion new pages each month.

2. Paid content deals

AI companies pay sites and archives for the right to train on their material. In its 2024 filing to go public, Reddit listed data licensing deals worth $203M. Shutterstock earned $104M from AI makers in 2023 alone.

3. Paid human work

Companies hire people to write sample answers. These workers also rank a model's replies and fix its mistakes. This work costs a lot, and it is big business. One company that sells it got $14.3B from Meta. In return, Meta got a 49% stake in that company.

4. Synthetic data

Synthetic data is practice material that one AI model writes for another. Microsoft's Phi-4 report says it used synthetic data at every stage of training. It is cheap to make in large amounts. But it can only repeat patterns a model has already seen.

5. Private business records

These are email, chat, support tickets, contracts and work orders. A company licenses them to an AI maker. Google won a court auction for one airline's old staff email and chat with a $10M bid. One service that helps startups shut down reports close to 100 such archive deals in a year.

Source by source

Why are the free sources getting harder to count on?

For years, the public web did most of the work. OpenAI described GPT-3 in 2020. At that time, a filtered copy of Common Crawl made up 60% of its training mix (Brown et al., 2020). Books, Wikipedia and other web text made up the rest.

That free supply is shrinking in two ways. First, more sites are saying no. Common Crawl's crawler follows each site's robots.txt file. That is a small file that tells crawlers which pages they may visit. So any site owner can block it with two lines.

A 2024 study looked at C4, a popular web dataset (Longpre et al., 2024). A group called the Data Provenance Initiative did the study. It looked at the most active, most important sources in C4. Within about a year, rules like robots.txt had fully blocked AI crawlers from more than 28% of them.

Second, there is only so much of it. Epoch AI is a research group. In 2024 it estimated that, if trends hold, models could use up the public text people have written. It put the date between 2026 and 2032. Our page on AI companies running short of training data explains that forecast and its limits.

Synthetic data helps fill some gaps, but it has a known weak spot. A 2024 paper in Nature tested what happens when models keep training on text that AI wrote. Over time, those models lost the rare details found in real human writing. So labs still need fresh material that people wrote.

The scarce part

Why are private business records the hardest data to get?

Money and computers can grow every other source on the list. Crawlers can run longer. Labs can sign more content deals and hire more workers. They can also make more synthetic text. Private business records do not work that way.

These records exist only inside the companies that made them. They sit behind logins that no crawler can get past. They also hold something the other sources lack. A support ticket shows a real customer, a real problem and the steps that solved it.

A ten-year email archive shows how a business handles orders, delays, fights and hires. Paid workers can copy that style. But they cannot make up ten years of real history.

That puts owners of older, established companies in a strong spot. Say you have 20 or more people and at least 3 years of records. Then you likely hold material no lab has trained on. You can license a copy and still own the records. You also decide what stays out. Our guide to licensing your company's records to AI developers covers the basics in more depth.

Business records in the news

What companies have been paid for their records

These are reported deals for private business records, the fifth source above. They do not promise any price for your company.

$10M

Google's top bid at a court auction held in August 2026 for Spirit Airlines' work archive. It holds roughly 100 million emails. It also holds some 500 million Teams messages. Another AI data firm was next in line at $7.5M. The sale still needs a judge's approval.

Chapter11Cases, Oct 2026
$12.5M

What a second AI data company offered for the Spirit records after the auction, 25% above Google's bid.

Fortune, Sep 2026
Hundreds of thousands

Dollars cielo24 got for 13 years of Jira tickets, email and Slack after it shut down, Forbes reported.

Forbes, Apr 2026
$1B

A buying pledge the firm behind that offer made in October 2026. It covers 12 months of buying and licensing company records.

Company announcement, Oct 2026
60%

Share of GPT-3's training mix that came from filtered Common Crawl web pages in 2020. That was before many sites began to block crawlers.

Brown et al., 2020

Your part of the supply chain

Which of your records AI companies license

If your company runs on common business software, you likely have several of these already.

What you haveUsual softwareWhy buyers ask for it
Years of company emailMicrosoft 365, Gmail, Exchange archivesShows how business choices get made in writing
Work chat logsSlack, Microsoft TeamsShows staff solving problems in casual talk
Customer support ticketsZendesk, Freshdesk, IntercomPairs real questions with real answers
Deal and account notesSalesforce, HubSpotTracks a sale from first contact to close
Contracts, bids and reportsSharePoint, Google Drive, BoxDetailed writing from people who know the trade
Work orders and job logsBusiness planning (ERP) or field service softwareShows tasks done in order, with results
Phone and meeting recordingsZoom, RingCentral, DialpadReal spoken back and forth from work
How-to docs and source codeConfluence, Notion, GitHubTeaches steps and how software gets built

Put a rough price on your part of the supply chain

The number comes from public buyer pricing. We average the programs that match you best. Answer a short quiz. No uploads, no logins.

Estimate my records

Stay in control

What to leave out, and how that works

A license covers only what you list. The contract names anything you leave out, and the buyer must follow it.

Personal files about staff

HR records, payroll, benefits and ID scans add risk and bring little value. Leave them out.

Records with special legal rules

Medical records fall under HIPAA, a federal law that protects health privacy. Private legal files need care too. Our page for law firms licensing their records shows how one such field handles it.

Other people's documents

Files you store for clients may belong to them. Check your agreements before you include them.

Mailboxes full of private talk

You can drop whole mailboxes, channels or years. One example is the owner's inbox. Another is a folder about a legal dispute.

Current trade secrets

You can name live pricing, formulas and plans and keep them out. Older material is often worth just as much to buyers. It is also far less sensitive to you.

How we work

From estimate to payment in four steps

DillisData is a data agency. We match you with programs that pay for records like yours. The program pays our fee. We never receive, store or open your records.

01

Two-minute estimate

Tell us a few quick things about your company. You need no files and no account.

2 minutes
02

Program match

We test your record profile against each AI data program that needs records like yours. Then we show the top 3 by likely payout.

Seconds
03

Sign-up

Each match warns you about any permanent rights before you agree to anything. Joining takes one click.

One click
04

Transfer and payment

Files move straight from your own systems to the buyer you chose. We are never in the middle. The buyer removes personal details and then pays you. Published payment times range from one week to roughly three months.

Per contract

Timing

Why the best prices go to companies that move early

No one can say how long labs will keep paying at today's level. Here is what we do know.

Gaps get filled

Labs pay top prices to cover a blind spot. Once a few companies in your field have covered it, your records matter less to them.

Other sources keep getting better

Better synthetic data or new training methods could lower demand for some records. Nobody knows when.

Stored records cost you money

Old archives take up paid storage. They earn nothing while they sit there.

Old accounts get closed

When you cancel software or retire a server, years of records can go with it. What is deleted can never be licensed.

Questions

Questions owners ask

Do AI companies train on my company's website?

Maybe. Crawlers such as Common Crawl's can collect public pages. You can block many crawlers with your robots.txt file. Crawlers cannot reach private email, tickets and files behind a login.

Is it legal for AI companies to use web pages without paying?

Courts are still arguing about that, and it is not settled. One thing is clear. Licensed data comes with written permission, and that is one reason labs pay for it.

What is synthetic data, in plain words?

It is practice text that an AI model writes instead of a person. It is cheap to make. But research shows it works best mixed with real human writing. It does not fully replace it.

Why would a lab pay for records instead of just making synthetic ones?

Synthetic text can only reflect what a model already knows. Real records show new cases, real mistakes and real fixes. The model has not seen those before.

What happens to our records after we license them?

The buyer copies the records you approved. It removes names and personal details. It uses that copy only in the ways the contract allows. You keep the originals, and you still own them.

Who pays DillisData?

The programs pay our fee. Owners never pay DillisData, and with no signed deal there is no fee at all.

Which companies qualify?

Programs differ. Several want 20 to 50 staff and a few years on file. Others name no size minimum. Larger staffs and more years of records tend to bring bigger offers. Closed or closing companies can still qualify.

Get started

Add your company to the supply chain, on your terms

You now know where AI training data comes from. You also know why private records are the scarce part. Find out what a license to yours might bring in.

Free to your company. We never hold your records. Ranked matches, explained.