Cloudflare Reveals GLM-5.3 Data Leak Affecting 155 AI Responses

Cloudflare has disclosed a tenant isolation issue in the GLM-5.3 model hosted on its Workers AI platform. The issue allowed some model responses to contain information from other customers' earlier requests to the same model.

The exposure began when GLM-5.3 launched on Workers AI on August 27, 2026. Cloudflare remediated the issue and restored the model on September 9.

According to Cloudflare, 155 responses were affected, and every affected response involved a prompt longer than 681,280 tokens. The company said its investigation found no evidence of deliberate exploitation. After excluding Cloudflare's own testing and the researcher's activity, the timing and volume of the requests were consistent with ordinary long-context usage.

Cloudflare traced the incident to two interacting problems: a mismatch between the model's configured context window and the available key-value (KV) cache capacity, and a bug in the open-source SGLang inference engine's handling of rejected requests in disaggregated prefill/decode mode.

The company corrected the configuration, patched the inference engine, and validated the changes before restoring public access. It also extended additional safeguards across production deployments using SGLang.

Cloudflare recommends that operators running SGLang in disaggregated prefill/decode mode upgrade to v0.5.22 or later, or backport the relevant fixes.

What Happened in the Workers AI Incident?

GLM-5.3 is a model from Z.ai designed for agentic coding workflows. Cloudflare makes it available through Workers AI under the model identifier @cf/zai-org/glm-5.3.

On September 8, security researcher Gadi Evron from Knostic reported through Cloudflare's HackerOne program that long prompts sent to the model could return text belonging to other users' sessions.

Cloudflare reproduced the issue the following morning. Its Product Security team confirmed the exposure using two independent accounts after declaring the incident. The company then made the model private while investigating.

Cloudflare corrected the configuration and deployed a patched inference engine before restoring the model on September 9.

The company confirmed that GLM-5.3 was the only Workers AI model affected. It said other disaggregated deployments had KV cache pools larger than their configured context lengths on both prefill and decode nodes. Those configurations did not reach the rejection path involved in this incident.

Cloudflare also applied the engine fix and additional hardening measures to every model it serves with SGLang.

The Two Technical Failures Behind the Exposure

Cloudflare traced the incident to two interacting issues. The first was a configuration mismatch between the model's advertised context window and the available prefill KV cache capacity. The second was a flaw in how SGLang handled rejected requests in disaggregated prefill/decode mode.

The exposure occurred because the undersized cache pool triggered a rejection path that SGLang did not handle safely in the disaggregated pipeline. The rejected request continued through processing and transfer, allowing the decode node to read stale cache pages that had not been correctly populated for the current request.

1. The KV Cache Pool Was Smaller Than the Context Window

Cloudflare configured GLM-5.3 with a context window of 1,048,576 tokens. However, the KV cache pool on its prefill GPUs could accommodate only 681,280 tokens.

The KV cache stores intermediate information used during model inference. In this deployment, it consisted of fixed-size pages in GPU memory. Requests receive pages as needed and release them after completion. The pages are not automatically zeroed between requests. Instead, subsequent requests overwrite the memory they use.

This reuse is common in inference engines, but it requires the engine to track page ownership and lifetime correctly. If a request reads pages that still contain data from another request, the system risks exposing information across request boundaries.

Cloudflare said it knew about the capacity mismatch before launch. Its pre-launch tests checked HTTP status codes and performance at long context lengths. The requests returned successful HTTP statuses and streaming responses without error frames, leading the team to believe the engine could handle the difference between the configured context window and the available cache capacity.

The gap was that these tests did not adequately validate the correctness and coherence of the generated output under the problematic conditions.

Cloudflare has since strengthened deployment checks to compare the actual KV pool capacity with the configured context length. The company also emphasized that actual cache capacity depends on the GPU memory remaining after model weights and other allocations. It can change with model versions, quantization, parallelism, and memory settings, so it should be measured rather than assumed.

2. SGLang Continued Processing a Rejected Request

The second issue involved SGLang's request handling in disaggregated prefill/decode mode.

In this architecture, prefill GPUs process the prompt and produce its KV cache. The cache is then transferred to separate decode GPUs, which generate the response. Cloudflare uses Mooncake and remote direct memory access (RDMA) for this transfer.

When a prompt exceeded the available KV cache capacity, SGLang's scheduler marked the request as rejected and recorded an error for the client. It also replaced the prompt with a one-token placeholder so the request could drain through the normal batch-processing machinery.

In disaggregated mode, however, the relevant admission queues checked whether a request was finished rather than whether it had been rejected. The rejected request therefore continued to be treated as active.

The prefill node processed the placeholder, wrote a small amount of KV cache data, and sent a transfer-complete message to the decode node. The decode node had already reserved cache pages based on the original prompt length and assumed the transfer had populated the expected cache state.

In reality, only the page written for the placeholder had been updated. Other reserved pages still held stale data from earlier requests.

The decode node then generated a response while attending to cache pages the current request had not written. This could cause the response to include content from another customer's earlier GLM-5.3 request.

The client received HTTP 200 instead of the expected HTTP 400 error. The response could appear coherent, which made checking HTTP status codes alone insufficient for detecting the issue.

Immediate Remediation: Configuration and Engine Changes

Cloudflare addressed both contributing issues before restoring public access to GLM-5.3.

Increasing KV Cache Capacity

Cloudflare adjusted its GLM-5.3 configuration to match its GLM-5.2 configuration, increasing the prefill KV cache pool to approximately 2.04 million tokens.

That capacity is roughly twice the configured 1,048,576-token context length. This removed the specific capacity mismatch that caused valid long-context requests to reach the problematic rejection path.

However, Cloudflare said relying on cache capacity alone would be fragile. Changes to model weights, quantization, tensor parallelism, CUDA graph sizes, or memory allocation can affect the amount of memory available for KV cache pages.

For that reason, the configuration change was only one part of the remediation. Cloudflare also fixed the inference engine's request-handling behavior.

Making the Rejection Path Safe

Cloudflare patched SGLang so requests carrying a pending rejection are retired at the disaggregation admission queue before bootstrap, allocation, or KV transfer takes place. The original error is preserved instead of allowing the rejected request to continue through the pipeline.

Cloudflare proposed this approach upstream as pull request #38704. An earlier change, #36988, had already been merged for a related failure path. SGLang maintainers subsequently merged #38935 to reject these requests earlier.

The updated handling prevents the rejected request from entering the handoff path that led to stale cache data being read by the decode node.

Cloudflare validated the configuration and engine changes using the researcher's reproduction script and its own canary-marker tests before restoring the model.

Working with the SGLang Team

Cloudflare said the incident was connected to broader work it had been conducting with SGLang maintainers on KV cache safety and request lifecycle handling.

The company had been reporting and addressing related issues since SGLang v0.5.10. During its investigation, Cloudflare also found that another user had reported the specific issue as SGLang issue #30233 on July 6, 2026.

After the incident, Cloudflare worked with SGLang maintainers to upstream fixes and strengthen handling of request rejection, cancellation, cache-page release, and transfer failures. The work addressed not only the precise failure path involved in the GLM-5.3 exposure, but also related conditions that could allow cache pages to be reused while operations were still in progress.

Key upstream changes included:

• Earlier rejection handling: Changes #36988 and #38935 improved the handling of rejected or aborted requests before they could proceed through the prefill-to-decode handoff.

• Request cancellation and failure coordination: Changes including #39461 and #41612 improved cancellation handling across prefill and decode when a client disconnects or prefill fails.

• Deferred KV cache release: Changes including #37454, #41023, #40645, #41402, and #41404 addressed situations in which cache pages could be released before transfers or related operations had safely finished.

• Transfer failure handling: Additional changes improved the handling of failed Mooncake sends, abort notifications, and transfer confirmations.

• KV cache integrity and monitoring: The upstream work also included optional checksum support, KV canary improvements, and metrics for deferred release.

The fixes were released across SGLang v0.5.20, v0.5.21, and v0.5.22. Cloudflare recommends operators using disaggregated SGLang upgrade to v0.5.22 or later, or backport the relevant fixes where an upgrade is not immediately possible.

Cloudflare also rolled out additional hardening across its production SGLang deployments between September 11 and September 19. The company's deployment checks now cover cache capacity relative to context length, deferred release on both prefill and decode roles, transfer-batch limits, and the presence of required patches.

Some work remained ongoing at the time of publication. Cloudflare identified additional checksum cleanup, zeroing KV pages on release, and extending integrity checks to every decode step as areas still under development.

SGLang Updates and Hardening Across Deployments

The upstream collaboration resulted in changes across several areas of the inference pipeline.

• Request admission: Rejecting affected requests before bootstrap, allocation, and KV transfer.

• Transfer coordination: Improving cancellation and failure handling across prefill and decode.

• Deferred KV release: Preventing cache pages from being released while transfers or other operations may still use them.

• Transfer integrity: Adding optional checksum verification for transferred KV data.

• Observability: Improving KV canary behavior and adding metrics for deferred release.

These changes address different parts of the KV cache lifecycle. A request must be rejected safely, transfer completion must reflect what actually happened, and cache pages must not be reused while another operation may still read or write them.

Cloudflare noted that not every integrity-checking feature supports every model architecture. Operators should confirm support for their models and inspect logs to verify that checks are running.

The company also identified follow-up work that remained in progress at the time of publication, including additional checksum cleanup, zeroing KV pages on release, and extending integrity checks to every decode step.

What Cloudflare Recommends to Other SGLang Operators

Cloudflare outlined several practical steps for teams running SGLang in disaggregated prefill/decode mode.

Check the Actual KV Pool Size

Measure KV pool capacity and compare it with the configured context length on every prefill and decode node. Use the engine's startup logs, and repeat the check after changes to weights, quantization, parallelism, or memory allocation.

If the pool is smaller than the context length, Cloudflare recommends reducing the context length to fit. Operators should not rely on a proxy to enforce the limit or assume the engine will always reject requests that exceed available capacity.

Upgrade or Backport the Relevant Fixes

Use a build containing fixes #36988 and #38935. Cloudflare recommends SGLang v0.5.22 or later. These changes provide an engine-side safeguard by rejecting affected requests before handoff, even if a capacity mismatch is missed.

Enable Deferred Release and Bound Transfers

Run with deferred KV release on both prefill and decode roles. Cloudflare notes that this behavior is enabled by default following #41023.

Operators should also bound Mooncake transfer batches so a timed-out transfer cannot leave an unbounded number of writes in flight.

Enable Corruption Detection Where Supported

Use --disaggregation-enable-kv-checksum and --kv-canary where the model architecture supports them. These mechanisms do not yet cover every architecture, so operators should check the relevant documentation and logs to confirm that requests are being verified.

Cloudflare said it was working with SGLang maintainers to extend these checks to more architectures, including GLM-5.3, GLM-5.3 Flash, and DeepSeek V4.

Avoid Unnecessary Request Aborts

Cloudflare also recommends configuring load balancers to avoid unnecessary mid-transfer aborts. Its approach keeps the prefill connection open until prefill finishes and the decode connection open until the first token is produced, then closes both.

This helps reduce cases in which a client disconnect interrupts the coordination between prefill and decode.

These recommendations are intended for operators of disaggregated SGLang deployments. They should not be interpreted as a claim that every SGLang installation or every model was affected by this incident.

Cloudflare's Longer-Term Plan: Integrity by Construction

Cloudflare said configuration checks and fixes to page lifetimes reduce the opportunities for KV cache misuse, but they do not make every possible failure impossible.

Its longer-term goal is for the inference engine to detect when a request is about to read a KV page it does not own and fail that request rather than return data from another session.

The company described a prototype in which each physical KV page carries a tag that changes when the page is reallocated. A request records the tags it expects, and the engine checks the mapping before decode reads the cache.

Cloudflare reported that its prototype tests incurred less than 1% overhead in throughput and tail latency. It tested the prototype on GLM-5.2 in mid-August but disabled it after encountering reliability issues. The prototype was also tied to particular attention backends and model architectures, which prevented it from serving as a general solution.

Cloudflare is working with the SGLang team on a production implementation that can work across models rather than relying on Cloudflare-specific patches.

The per-transfer checksum work in SGLang is an initial step. It validates the transfer from prefill to decode, while page-level tags would go further by checking ownership at read time, including reads that occur long after a transfer.

Formal Methods for Distributed KV Cache Safety

Cloudflare also plans to apply formal verification methods to KV cache transfer systems.

Because RDMA transfers data directly between the memory of separate machines, both sides must agree on where data belongs, how much data is transferred, and when the transfer has completed. Inconsistent state between the two nodes can cause the receiver to read incorrect data.

Cloudflare intends to model the safety of its production SGLang deployment and Mooncake-based transfer protocol, then identify and correct deviations from the required safety properties.

The company acknowledged that formal verification has limitations. Its stated goal is to prove as much of the system's behavior as possible and more closely review and monitor the areas that cannot be fully proven.

Timeline of the Incident and Response

Cloudflare provided the following timeline, with all times in UTC.

What This Incident Means for AI Infrastructure Security

The incident demonstrates how a configuration mismatch and unsafe error handling in a distributed inference engine can become a data isolation problem.

The issue was not simply that a prompt exceeded a limit. The critical failure was that a rejected request continued through the prefill/decode pipeline, allowing the decode node to assume cache data had been transferred when the expected cache state had not been written.

For AI infrastructure teams, Cloudflare's report highlights several practical lessons:

• Validate runtime capacity: Check actual KV pool capacity rather than relying only on advertised context limits or successful HTTP responses.

• Treat request rejection as a security boundary: A rejected request must not continue into downstream processing with incomplete state.

• Track memory ownership and lifetime: Cache pages should not be reused while transfers or other operations may still access them.

• Verify transfer completion: Distributed components need to distinguish a completed transfer from a failed or incomplete one.

• Test output correctness: Security testing should include output validation and cross-request isolation, not only response codes and performance.

• Apply upstream fixes: Operators should confirm that required patches are present and that supported integrity checks are active.

These are lessons from this incident and Cloudflare's stated recommendations. They are not evidence that all AI inference engines share the same flaw.

Conclusion

Cloudflare's GLM-5.3 incident showed how two interacting failures in configuration and SGLang request handling could cause AI responses to include data from other customers' earlier requests. The company reported 155 affected responses, limited the impact to GLM-5.3 on Workers AI, and found no evidence of deliberate exploitation.

Cloudflare corrected the configuration, patched the inference engine, restored the model on September 9, and extended safeguards across its production SGLang deployments. Its collaboration with SGLang maintainers also led to upstream changes addressing request rejection, cache-page release, transfer failures, and integrity checks.

Cloudflare recommends that operators using disaggregated SGLang upgrade to v0.5.22 or later, validate KV pool capacity against context length, and enable supported integrity checks.

Its longer-term work aims to make cache ownership verifiable at read time and strengthen the safety of distributed KV cache transfers. The incident underscores why reliable AI infrastructure requires not only capacity planning, but also strict handling of rejected requests, memory lifetimes, and data ownership across nodes.