Bottom line: Cloudflare enables granular control over AI crawlers instead of blanket blocking, distinguishing between different purposes such as training or research requests.
Cloudflare extends its offering for controlling AI crawlers, allowing website operators to selectively allow or block access based on the intended use case.
Cloudflare has expanded its capabilities for managing AI crawlers. Instead of blocking all requests from AI systems across the board, website operators can now exercise differentiated control and enable or disable individual crawler types according to their purpose.
This function targets website administrators who want to keep training data for large language models away from their content, while at the same time allowing legitimate research or indexing requests. The granular control enables distinction between different AI use cases and thus allows for a less restrictive approach than a complete blockade.
The feature contributes to the debate on controlling data use by AI systems and offers a pragmatic middle ground between complete openness and total blocking. Operators can thus better adapt their content strategies to their own requirements and business models.
Source: www.heise.de · Published 2 July 2026
Lumi AI News — AI-assisted curation in accordance with Art. 50 EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.7.2.