Cloudflare on Wednesday offered web hosting customers a way to block AI RobotsMethods of crawling website content and using the data without permission to train machine learning models.
It is based on customers' distaste for AI bots and says that to help keep content creators safe on the internet, the company said in a statement.
Source Note: The image is generated by AI, and the image is authorized by Midjourney
"We've heard loud and clear that customers don't want AI bots accessing their sites, especially those that do so dishonestly. To help, we've added a new one-click block all AI bots feature."
There is already a relatively effective way for website owners to block bots, and that is the widely available robots.txt file. When placed in the root directory of a website, the automaticWeb crawlerShould notice and follow the instructions in the document telling them not to enter.
Given the widespread belief that generative AI is based on theft and the many lawsuits attempting to hold AI companies accountable, companies engaged in "scrubbing" content generously allow web publishers to opt out of theft.
Last August, OpenAI released guidance on how to block its GPTbot web crawler using the robots.txt directive, probably because it realized that people were concerned about content being crawled without permission and used for AI training without consent. Google took similar steps the following month. Also last September, Cloudflare began offering a way to block AI bots that follow the rules, with 85% customers reportedly enabling this blocking feature.