On July 31, the Financial Times (FT) published a blog post stating that AI Companies Anthropic Although it claims to "develop AI responsibly",But through ClaudeBot robotOver-fetchingWebsite data, used to train the Claude large language model.
Although using web crawlers to scrape data is a common practice in the artificial intelligence industry, Anthropic has been criticized for its aggressiveness.
Freelancer, a freelance website, also said that ClaudeBot visited 3.5 million times in four hours and was forced to block it. Critics pointed out that Anthropic ignored the website's robots.txt protocol and forcibly obtained data, which was contrary to its declared concept of "responsible AI".
Kyle Wiens, CEO of repair team iFixit, tweeted on July 24, translated by IT Home as follows:
@AnthropicAI, I know you're hungry for data, and Claude is a very smart model, but is it really necessary to hit our servers 1 million times in 24 hours?
This traffic does not pay us and takes up our development resources, which is really unfair.
Our Terms of Service clearly prohibit the use of our content in this way, but you quietly @AnthropicAI did it.
If @AnthropicAI wants to communicate about commercial use licenses for our content, we’re open to communicating.