Cloudflare Introduces Continuous Trust Scoring for Verified Bots to Improve Web Accountability

Cloudflare Moves Beyond Bot Identity to Continuous Trust Assessment

Cloudflare has introduced a system designed to evaluate whether Verified bots continue to behave responsibly after their identities have been established. The approach assesses observed bot behaviour over time, focusing on whether automated crawlers respect website preferences and respond appropriately when website owners signal that traffic should slow down or stop.

The announcement addresses an increasingly important challenge for websites: identifying an automated visitor does not necessarily establish that its activity is welcome or responsible.

As search engines, AI crawlers and other automated agents interact with online content, website operators need ways to distinguish identifiable bots from those that consistently respect their access preferences.

Cloudflare's new approach aims to make that distinction clearer by combining bot identity with ongoing behavioural observations.

Why Verified Bot Identity Is Not Enough

Bot verification helps website operators understand who is accessing their websites. However, an identifiable bot can still generate excessive traffic, disregard crawling restrictions or continue making requests after a website has indicated that the activity is unwelcome.

Cloudflare explains that these problems do not always result from deliberate abuse. Misconfiguration and software bugs can also cause a bot to behave inappropriately.

The company therefore distinguishes between two concepts:

• Risk: The possibility that a particular request or action could cause harm at a specific moment.

• Trust: Confidence developed through consistent, responsible behaviour over time.

This distinction is central to the new system. Rather than treating verification as permanent evidence of good behaviour, Cloudflare wants to assess whether a bot continues to meet the expectations associated with its verified status.

For website owners, the principle is straightforward: a bot's identity matters, but its actions matter too.

How Cloudflare Evaluates Bot Behaviour

Cloudflare's new system examines observed patterns to determine whether Verified bots consistently follow website preferences. It initially focuses on two signals: whether bots respect instructions and whether they respond to direct feedback.

1. Respecting website crawling instructions

Website owners commonly use robots.txt to communicate which parts of a website crawlers should or should not access.

Cloudflare's system considers whether observed bot traffic aligns with these published preferences over time. It focuses on repeated patterns rather than treating every isolated violation as conclusive evidence of irresponsible behaviour.

The approach aligns with the Robots Exclusion Protocol described in RFC 9309. However, robots.txt should not be confused with an access-control mechanism. It communicates crawling preferences but does not, by itself, prevent unauthorised access.

Cloudflare also notes that responsible behaviour requires more than following the paths explicitly listed in a site's crawling instructions. A crawler may still need to respond appropriately when a website communicates that continued access is unwelcome.

2. Responding to rate limits and direct feedback

A second important signal is how a bot responds when a website asks it to reduce its activity.

Websites can return HTTP 429, commonly known as "Too Many Requests", when a visitor sends requests too frequently. This response can include guidance about when the visitor should retry.

Cloudflare's system observes whether a bot slows down, pauses or continues sending requests at the same rate after receiving this signal.

Consistently adjusting traffic in response to rate limits contributes to a more favourable trust assessment. Repeatedly ignoring these signals can reduce confidence in the bot's behaviour.

The objective is not simply to identify excessive traffic. It is to assess whether the bot responds appropriately when the website communicates its limits.

What Is a Bot Trust Band?

Instead of relying solely on a binary distinction between verified and unverified bots, Cloudflare describes an explainable trust band based on observed behaviour.

The trust band is intended to show how consistently a bot respects website preferences and which observed behaviours informed that assessment. It is not presented as a permanent judgement of a bot or its operator.

This distinction matters because bot behaviour can change over time. A crawler that previously followed site policies may later behave differently because of software changes, configuration errors or other issues.

A continuing assessment can help website operators understand whether the behaviour associated with a bot's verified identity remains consistent.

Cloudflare has not disclosed the numerical scale, exact scoring formula, weighting of individual signals or detailed thresholds used to determine the trust band. Additional details were not disclosed in the announcement.

Why explainability matters

For organisations managing automated traffic, an assessment is more useful when it provides context about the behaviour behind it.

A trust band that identifies relevant observations could help website owners distinguish recurring disregard for site preferences from isolated incidents. It could also give bot operators clearer reasons to investigate problems and demonstrate improved behaviour.

These are potential operational benefits of the announced approach, not measured outcomes reported by Cloudflare.

Cloudflare's Previous Example: Perplexity's Bot

Cloudflare points to its decision to remove Perplexity's bot from its Verified bot list in 2025 as an example of trust being withdrawn after observed behaviour raised concerns.

According to Cloudflare, the bot sought to access resources on websites that had opted out of scraping and subsequently attempted to circumvent controls after those requests were blocked.

The example illustrates the distinction between being able to identify a bot and deciding whether its activity merits continued trust.

For website owners, the broader lesson is that a bot's reputation should not be treated as an unconditional exemption from website policies. Verification can establish identity, but continued access should remain subject to appropriate controls and observed conduct.

The example is Cloudflare's account of the decision. It should not be interpreted as an independent assessment of every crawler or AI service.

How the System Fits Into Cloudflare's Existing Bot Controls

Cloudflare positions the new system as part of a broader approach to automated traffic management. It works alongside three relevant components: Verified bots, Bot Preference Sync and Bot Management.

Verified bots and BotBase

Establish and catalogue bot identities so website operators can better understand which automated services are making requests.

Bot Preference Sync

Helps website owners publish their preferences and align those preferences with their configured policies.

Continuous behaviour assessment

Observes whether Verified bots consistently honour website preferences and respond appropriately to feedback.

Bot Management

Allows website operators to decide how automated traffic is handled.

Together, these components connect identity, published website preferences, observed behaviour and traffic-handling decisions. The goal is to make trust depend on consistent conduct rather than identity alone.

What Website Owners and AI Developers Need to Know

The announcement has implications for organisations that publish content, operate web infrastructure or develop automated agents.

For website owners

Website operators should ensure that their crawling preferences are clear and that their traffic-management policies reflect their requirements.

They should also understand that publishing robots.txt is not a substitute for technical access controls. Where access must be restricted, appropriate enforcement mechanisms remain important.

Cloudflare's proposed approach could offer additional context when evaluating automated visitors, particularly where repeated crawling behaviour conflicts with site preferences.

For AI companies and bot developers

Developers operating crawlers and AI agents need to consider how their software responds to website policies and traffic feedback.

A verified identity alone may not be sufficient to maintain trust if the bot repeatedly ignores requests to stop or slow down.

Building reliable handling of crawling directives, rate limits and other feedback into automated systems could help operators maintain more consistent behaviour.

For security and infrastructure teams

The announcement also highlights the importance of monitoring automated traffic as an ongoing operational responsibility.

Security teams may need to distinguish between legitimate automated access, misconfigured crawlers and behaviour that warrants stronger restrictions. A behavioural trust assessment could complement existing controls, but it should not replace them.

These are practical implications of the announcement rather than specific implementation requirements mandated by Cloudflare.

When Will Customers Be Able to Use the New System?

Cloudflare says it is already using the system internally to evaluate how consistently Verified bots respect website preferences.

The company is using this initial phase to build a history of observations and refine how those observations translate into useful guidance. It plans to make the system available to all Cloudflare customers and allow them to select the trust threshold used to determine how Verified bot traffic is handled. Cloudflare also plans to publish trust assessments on Cloudflare Radar in the future.

The announcement does not specify a general availability date, the precise trust-threshold configuration options, the scoring methodology or a confirmed publication date for the Radar assessments.

Additional details were not disclosed in the announcement.

What This Development Means for the Web

Cloudflare's announcement highlights a broader industry shift towards evaluating automated systems by both their identity and their behaviour.

As AI-related crawlers and agents interact with websites, publishers face the challenge of making content accessible to appropriate services while retaining control over how their infrastructure is used.

Identity verification helps establish accountability, but it does not answer every question about responsible access. Ongoing behavioural observations offer another way to evaluate whether automated visitors respect the preferences of the websites they access.

For enterprises, this could mean gaining additional context when deciding how to handle verified automated traffic. For bot developers, it reinforces the importance of respecting crawling instructions, responding to rate limits and avoiding repeated attempts to bypass restrictions.

The effectiveness of Cloudflare's approach will depend on how its assessments are implemented, explained and incorporated into traffic-management decisions. The company has not yet published customer-wide results from the system.

The central principle is clear: a verified bot should continue to demonstrate responsible behaviour rather than rely on its identity alone.