Cloudflare is cracking down on multi purpose crawlers – Breaking News & Latest Updates 2026
Skip to main content
The homepageThe VergeThe Verge logo.
The homepageThe VergeThe Verge logo.
Posted
E
External Link
Emma Roth
Cloudflare is cracking down on “multi-purpose” crawlers.

Starting September 15th, Cloudflare will block bots that scrape ad-supported websites for search indexing and AI training at the same time. The goal is to encourage AI companies to separate their crawlers for different purposes, giving publishers more control over which ones to allow into their websites.

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
Comments
Loading comments
Getting the conversation ready...

Most Popular

Most Popular
  1. OpenAI pauses training of its ‘most capable models’
  2. Leaks reveal a new Apple HomePod mini, iPad mini, and Apple TV 4K
  3. Can Apple Home’s AI camera features outsmart Amazon’s and Google’s? I put them to the test
  4. Can ‘eSUV’ e-bikes really go from trail to town?
  5. Gemini 4 is almost ready, says new Google DeepMind chief

More in AI

OpenAI pauses training of its ‘most capable models’OpenAI pauses training of its ‘most capable models’
Can Cloudflare CEO Matthew Prince save the web from AI?Can Cloudflare CEO Matthew Prince save the web from AI?
Play
Meta makes the Muse filesystem even more accessibleMeta makes the Muse filesystem even more accessible
Sony and UMG are suing Suno againSony and UMG are suing Suno again
Terrence O'Brien
One company is at the center of a wave of rogue AI attacksOne company is at the center of a wave of rogue AI attacks
Can Apple Home’s AI camera features outsmart Amazon’s and Google’s? I put them to the testCan Apple Home’s AI camera features outsmart Amazon’s and Google’s? I put them to the test

Top Stories