Internet publishers back strategy to get compensation for AI-scraped content

Top content platforms including Reddit, Medium, and Yahoo have adopted a new way to get fair compensation for AI scraping of their content.

According to a press release by the RSL Web Standard and Collective Rights Organization, the three platforms among many others have expressed support for the new Really Simple Licensing (RSL) standard that ensures fair Compensation for Millions of Publishers and Creators worldwide.

Ensuring content fairplay 

The world is tending towards AI in many ways, and one of the most popular is the search for information. Already, most people prefer AI when searching for information instead of traditional browsers.

AI search tools usually scrape platforms such as Reddit and Medium for content that it provides to its users, but there has been no predictable way to compensate those whose content is being shared.

Therefore, the platforms have today supported the launch of the RSL Standard licensing protocol to ensure fair, standardized compensation for publishers and creators, as well as provide simple, automated licensing for AI and technology companies.

An open, decentralized protocol based on the widely adopted RSS (Really Simple Syndication) standard that scales to millions of websites, RSL can be applied to any digital content, including web pages, books, videos, and datasets. 

CEO of O’Reilly Media, Tim O’Reilly said RSS was critical in giving early online publishers a simple and open way to reach readers all over the world, but AI has changed that.

“But today, as AI systems absorb and repurpose that same content without permission or compensation, the rules need to evolve. RSL builds directly on the legacy of RSS, providing the missing licensing layer for the AI-first Internet. It ensures that the creators and publishers who fuel AI innovation are not just part of the conversation but fairly compensated for the value they create.”

How it works

The RSL Standard goes beyond simply blocking the robots.txt protocol to It bring on a new licensing infrastructure that empowers publishers to add machine-readable licensing and royalty terms to their robots.txt files. 

This then specifies how AI applications and agents must compensate them for using their content.

Any website can access it for free today to define licensing, usage, and compensation terms for AI crawlers and agents.