Cloudflare’s new “pay per crawl” initiative has sparked a debate among SEO professionals and digital marketers.
The company has introduced a default AI crawler-blocking system alongside new monetization options for publishers.
This enables publishers to charge AI companies for access, which could impact how web content is consumed and valued in the age of generative search.
The system, now in private beta, blocks known AI crawlers by default for new Cloudflare domains.
Publishers can choose one of three access settings for each crawler:
Crawlers that attempt to access blocked content will receive a 402 Payment Required response. Publishers set a flat, sitewide price per request, and Cloudflare handles billing and revenue distribution.
Cloudflare wrote:
“Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.
The system integrates directly with Cloudflare’s bot management tools and works alongside existing WAF rules and robots.txt files. Authentication is handled using Ed25519 key pairs and HTTP message signatures to prevent spoofing.
Cloudflare says early adopters include major publishers like Condé Nast, Time, The Atlantic, AP, BuzzFeed, Reddit, Pinterest, Quora, and others.
While the current setup supports only flat pricing, the company plans to explore dynamic and granular pricing models in future iterations.
While Cloudflare’s new controls can be changed manually, several SEO experts are concerned about the impact of making the system opt-out rather than opt-in.
“This won’t end well,” wrote Duane Forrester, Vice President of Industry Insights at Yext, warning that businesses may struggle to appear in AI-powered answers without realizing crawler access is being blocked unless a fee is paid.
Lily Ray, Vice President of SEO Strategy and Research at Amsive Digital, noted the change is likely to spark urgent conversations with clients, especially those unaware that their sites might now be invisible to AI crawlers by default.
Ryan Jones, Senior Vice President of SEO at Razorfish, expressed that most of his client sites actually want AI crawlers to access their content for visibility reasons.
Some in the community welcome the move as a long-overdue rebalancing of content economics.
“A force is needed to tilt the balance back to where it once was,” said Pedro Dias, Technical SEO Consultant and former member of Google’s Search Quality team. He suggests that the current dynamic favors AI companies at the expense of publishers.
Ilya Grigorik, Distinguished Engineer and Technical Advisor at Shopify, praised the use of cryptographic authentication, saying it’s “much needed” given how difficult it is to distinguish between legitimate and malicious bots.
Under the new system, crawlers must authenticate using public key cryptography and declare payment intent via custom HTTP headers.
As Cloudflare’s new default settings take effect, there’s concern around losing visibility in AI search tools. But you have options to regain control.
AI traffic may decline sharply if your domain blocks AI bots without you realizing it.
Himanshu Sharma, digital analytics consultant and founder of OptimizeSmart, warned on X:
“Expect a sharp decline in AI traffic reported by GA4 as Cloudflare blocks almost all known AI crawlers/bots from scraping your website content by default.”
Sharma advised site owners to proactively review their Cloudflare settings:
This option allows you to decide whether you want to maintain visibility on AI-driven platforms or limit the use of content for training and responses.
Cloudflare’s pay-per-crawl system formalizes a new layer of negotiation over who gets to access web content, and at what cost.
For SEO pros, this adds complexity: visibility may now depend not just on ranking, but on crawler access settings, payment policies, and bot authentication.
While some see this as empowering publishers, others warn it could fragment the open web, where content access varies based on infrastructure and paywalls.
If generative AI becomes a core part of how people search, and the pipes feeding that AI are now toll roads, websites will need to manage visibility across a growing patchwork of systems, policies, and financial models.
Featured Image: Roman Samborskyi/Shutterstock
Cloudflare’s new “pay per crawl” initiative has sparked a debate among SEO professionals and digital marketers.
The company has introduced a default AI crawler-blocking system alongside new monetization options for publishers.
This enables publishers to charge AI companies for access, which could impact how web content is consumed and valued in the age of generative search.
The system, now in private beta, blocks known AI crawlers by default for new Cloudflare domains.
Publishers can choose one of three access settings for each crawler:
Crawlers that attempt to access blocked content will receive a 402 Payment Required response. Publishers set a flat, sitewide price per request, and Cloudflare handles billing and revenue distribution.
Cloudflare wrote:
“Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.
The system integrates directly with Cloudflare’s bot management tools and works alongside existing WAF rules and robots.txt files. Authentication is handled using Ed25519 key pairs and HTTP message signatures to prevent spoofing.
Cloudflare says early adopters include major publishers like Condé Nast, Time, The Atlantic, AP, BuzzFeed, Reddit, Pinterest, Quora, and others.
While the current setup supports only flat pricing, the company plans to explore dynamic and granular pricing models in future iterations.
While Cloudflare’s new controls can be changed manually, several SEO experts are concerned about the impact of making the system opt-out rather than opt-in.
“This won’t end well,” wrote Duane Forrester, Vice President of Industry Insights at Yext, warning that businesses may struggle to appear in AI-powered answers without realizing crawler access is being blocked unless a fee is paid.
Lily Ray, Vice President of SEO Strategy and Research at Amsive Digital, noted the change is likely to spark urgent conversations with clients, especially those unaware that their sites might now be invisible to AI crawlers by default.
Ryan Jones, Senior Vice President of SEO at Razorfish, expressed that most of his client sites actually want AI crawlers to access their content for visibility reasons.
Some in the community welcome the move as a long-overdue rebalancing of content economics.
“A force is needed to tilt the balance back to where it once was,” said Pedro Dias, Technical SEO Consultant and former member of Google’s Search Quality team. He suggests that the current dynamic favors AI companies at the expense of publishers.
Ilya Grigorik, Distinguished Engineer and Technical Advisor at Shopify, praised the use of cryptographic authentication, saying it’s “much needed” given how difficult it is to distinguish between legitimate and malicious bots.
Under the new system, crawlers must authenticate using public key cryptography and declare payment intent via custom HTTP headers.
As Cloudflare’s new default settings take effect, there’s concern around losing visibility in AI search tools. But you have options to regain control.
AI traffic may decline sharply if your domain blocks AI bots without you realizing it.
Himanshu Sharma, digital analytics consultant and founder of OptimizeSmart, warned on X:
“Expect a sharp decline in AI traffic reported by GA4 as Cloudflare blocks almost all known AI crawlers/bots from scraping your website content by default.”
Sharma advised site owners to proactively review their Cloudflare settings:
This option allows you to decide whether you want to maintain visibility on AI-driven platforms or limit the use of content for training and responses.
Cloudflare’s pay-per-crawl system formalizes a new layer of negotiation over who gets to access web content, and at what cost.
For SEO pros, this adds complexity: visibility may now depend not just on ranking, but on crawler access settings, payment policies, and bot authentication.
While some see this as empowering publishers, others warn it could fragment the open web, where content access varies based on infrastructure and paywalls.
If generative AI becomes a core part of how people search, and the pipes feeding that AI are now toll roads, websites will need to manage visibility across a growing patchwork of systems, policies, and financial models.
Featured Image: Roman Samborskyi/Shutterstock
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using ‘Content here, content here’, making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for ‘lorem ipsum’ will uncover many web sites still in their infancy.
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using ‘Content here, content here’, making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for ‘lorem ipsum’ will uncover many web sites still in their infancy.
The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using ‘Content here, content here’, making
The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using ‘Content here, content here’, making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for ‘lorem ipsum’ will uncover many web sites still in their infancy.
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution
Copyright BlazeThemes. 2023
