This content originally appeared on DEV Community and was authored by Denis Perov
We all know the current state of AI data scraping is broken. AI companies are consuming massive amounts of bandwidth (seen cases of 30TB/month from single bots), driving up infrastructure costs, and training on content without compensating creators. Meanwhile, content creators are stuck either blocking everything or watching their servers burn 🔥
There's Cloudflare's pay-per-crawl (when it releases) that just adds paywalls to web scraping which is essentially a cat-and-mouse game because 30-40% of AI traffic is now unidentified bots.
We need a completely different solution, which will motivate AI companies to pay for the content. And to achieve that, we must give them a better alternative to web scraping 💡
The Concept
I though of a different approach -- a data marketplace where AI companies pay content creators directly for structured, clean training data.
Instead of the current chaos of scraping, blocking, and legal battles, this creates a professional data exchange where:
- Content creators export their data to external data storage
- AI companies access structured data via MCP server using SQL-like queries
- Data is split into two chunks: public & private. Public chunk can be used for filtering but access to the private chunk is token-gated
- Smart contracts handle automatic payments in cryptocurrency and access control
- Everything is permissionless -- no middleman taking cuts or controlling access
This approach offers:
- Structured data that AI companies actually prefer over HTML scraping
- Zero infrastructure burden - data served from decentralized network
- Cryptographic proof of licensing for auditing
Development Plan
Planning to build a WordPress plugin as the first integration because:
- WordPress powers 43% of the web - massive potential impact
- Site owners are desperately seeking solutions (27+ blocking plugins exist)
- Non-technical users need simple tools to participate
The plugin would provide simple admin UI for data export configuration, public/private field mapping (e.g., titles free, full content paid), automatic sync of new content and earnings dashboard.
Seeking Feedback
This isn't about forcing anyone to monetize their data -- it's about giving creators the option to get compensated instead of just bearing the costs. If you want your data free for AI training, set the price to zero. Your choice.
Happy to dive into technical details. Also interested in connecting with WordPress site owners who want to "make money from the machines" and might want to beta test.
Feel free to roast the idea if you want! I think it's better to face a hard truth before investing a lot of personal resources into the development of something that will never work.
This content originally appeared on DEV Community and was authored by Denis Perov
Denis Perov | Sciencx (2025-08-27T20:07:59+00:00) What can make AI companies pay for data instead of scraping?. Retrieved from https://www.scien.cx/2025/08/27/what-can-make-ai-companies-pay-for-data-instead-of-scraping/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.