TECH - OpenAI And Anthropic Allegedly Ignore Web Scraping Rules Stirring Controversy | Benzinga
Two of the world’s leading AI startups, OpenAI and Anthropic, are reportedly disregarding requests from media publishers to cease scraping their web content for free model training data.
What Happened: OpenAI and Anthropic are either ignoring or bypassing a well-established web rule, known as robots.txt, which is designed to prevent automated scraping of websites, reported Business Insider.
This has been brought to light by TollBit, a startup aiming to facilitate paid licensing deals between publishers and AI companies.
Despite public statements from OpenAI and Anthropic that they respect robots.txt and blocks to their specific web crawlers, TollBit’s findings suggest otherwise.
See Also: Nvidia Is ‘At The Center’ Of Something ‘Really Big,’ Says Expert
A report ...