BETA
THIS IS A BETA EXPERIENCE. OPT-OUT HERE

More From Forbes

Edit Story

Cloudflare Enables Websites To Block AI Bots With One-Click Solution

Following
Updated Jul 9, 2024, 12:59am EDT

A new problem for website owners in this era of artificial intelligence changing the digital landscape is AI bots scraping their content without permission. To address this growing concern, Cloudflare, the connectivity cloud company, has introduced a feature that allows customers to block AI bots with just a single click.

AI bots, also known as AI crawlers or scrapers, are automated programs designed to systematically browse the internet and collect vast amounts of data. Unlike traditional web crawlers used by search engines to index content, AI bots often gather information to train large language models or power AI-driven applications. While search engine crawlers typically follow established protocols like respecting robots.txt files and identifying themselves clearly, some AI bots may not adhere to these courtesies.

The rise of generative AI has dramatically increased the demand for training data, making original web content more valuable than ever. This has led to concerns about the unauthorized use of copyrighted material, personal information and intellectual property. Notable incidents have highlighted these issues, such as Google's reported $60 million annual payment to license Reddit's user-generated content and allegations of AI companies using celebrity voices without permission.

Recognizing the growing need for better control over AI bot access, Cloudflare has launched a new feature that allows customers to block all AI bots with a single click. This option is available to all Cloudflare users, including those on the free tier. To enable this protection, customers simply navigate to the Security section of the Cloudflare dashboard and toggle the "AI Scrapers and Crawlers" switch.

This feature is designed to be dynamic, with Cloudflare continuously updating it to address new fingerprints of offending bots identified as widely scraping the web for model training. By leveraging its vast network, which processes an average of 57 million requests per second, Cloudflare can quickly detect and respond to emerging AI bot activities.

Cloudflare's analysis of AI bot traffic across its network revealed some interesting insights:

1. The most active AI bots in terms of request volume are Bytespider, Amazonbot, ClaudeBot and GPTBot.

2. Bytespider, operated by ByteDance (TikTok's parent company), leads in both request volume and the extent of internet property crawling.

3. GPTBot, managed by OpenAI, ranks second in both crawling activity and frequency of being blocked by website owners.

4. Despite AI bots accessing 39% of the top one million internet properties using Cloudflare, only 2.98% of these properties actively block or challenge AI bot requests.

5. More popular websites are more likely to be targeted by AI bots and, correspondingly, more likely to implement blocking measures.

One of the challenges in managing AI bot traffic is that some operators attempt to disguise their bots as legitimate web browsers by using spoofed user agents. Cloudflare has developed sophisticated machine learning models to identify these deceptive practices. Their global bot score system can accurately flag traffic from evasive AI bots, even when they change their user agents or employ other obfuscation techniques.

Cloudflare's approach leverages global machine learning models and aggregates data across numerous indicators to understand the trustworthiness of various bot fingerprints. This allows them to detect new scraping tools and behaviors without needing to manually fingerprint each bot, ensuring that customers remain protected against the latest waves of bot activity.

By providing this easy-to-use blocking feature, Cloudflare aims to empower website owners to maintain control over their content and decide how it may be used in AI training or applications. This move also sends a clear message to AI companies about the importance of respecting content creators' rights and obtaining proper permissions for data usage.

Cloudflare has also introduced mechanisms for users to report misbehaving AI crawlers. Enterprise Bot Management customers can submit false negative feedback reports through Bot Analytics, while all Cloudflare customers can use a dedicated reporting tool to flag AI bots scraping their websites without permission.

As AI technology continues to evolve, Cloudflare anticipates that some AI companies may persistently adapt their methods to evade detection. In response, Cloudflare is promising to continually update their AI Scrapers and Crawlers rules and refine their machine learning models. Their goal is to ensure that the internet remains a place where content creators can thrive and maintain full control over how their work is used in AI training and applications.

This initiative by Cloudflare represents a significant step in the ongoing dialogue about AI ethics, data rights and the future of content creation in the digital age. By providing tools to manage AI bot access, Cloudflare is helping to shape a more transparent and consensual relationship between content creators and AI developers, potentially influencing the direction of AI development towards more responsible and ethical practices.

Follow me on Twitter or LinkedInCheck out my website

Join The Conversation

Comments 

One Community. Many Voices. Create a free account to share your thoughts. 

Read our community guidelines .

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's Terms of Service.  We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

  • False or intentionally out-of-context or misleading information
  • Spam
  • Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
  • Attacks on the identity of other commenters or the article's author
  • Content that otherwise violates our site's terms.

User accounts will be blocked if we notice or believe that users are engaged in:

  • Continuous attempts to re-post comments that have been previously moderated/rejected
  • Racist, sexist, homophobic or other discriminatory comments
  • Attempts or tactics that put the site security at risk
  • Actions that otherwise violate our site's terms.

So, how can you be a power user?

  • Stay on topic and share your insights
  • Feel free to be clear and thoughtful to get your point across
  • ‘Like’ or ‘Dislike’ to show your point of view.
  • Protect your community.
  • Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's Terms of Service.