A Successful AI Response Can Hide a Tenant Isolation Failure

Summary

A successful AI response can conceal a failure to isolate one customer’s data from another’s. Cloudflare’s October 26 disclosure describes how a capacity mismatch and an inference-engine error path combined to expose earlier requests in responses from one Workers AI model. The incident was remediated in September; the disclosure offers a practical lesson for teams operating shared inference infrastructure.

Key Updates

• A mismatch made the failing path reachable. Cloudflare configured GLM-5.3 for a 1,048,576-token context window, while its prefill KV cache pool held 681,280 tokens. Long requests could fit the advertised window but exceed the available pool. Pre-launch tests checked response codes and performance, not output coherence or isolation.
• A rejected request continued across the handoff. In SGLang’s disaggregated setup, separate workers process the prompt and generate the answer. The rejected request proceeded with a one-token placeholder; the decode worker then read stale cache pages containing earlier requests. The SGLang team says the single-node rejection path returned an error instead.
• The reported incident had a defined scope. Cloudflare says 155 responses were affected between August 27 and September 9, all for prompts longer than 681,280 tokens. It found no evidence of deliberate exploitation and says no other Workers AI model was affected. That finding is not proof that exploitation was impossible or that other operators were unaffected.
• Configuration and engine fixes addressed different layers. Cloudflare took the model offline on September 9, enlarged the pool, patched the engine and tested the changes before restoring service that day. The SGLang team describes upstream fixes to reject requests before handoff and separate hardening for cache lifetime during interrupted transfers.

Why It Matters

Shared inference systems reuse memory for performance. Their isolation depends on the engine correctly tracking which request owns each page and on workers agreeing about errors and transfers. A coherent HTTP 200 response can look healthy while reading stale state, so availability checks alone are not evidence of confidentiality.

The incident’s rejection-path failure is distinct from the related race involving writes still in flight after an abort. Both show why request boundaries need testing at handoff and cleanup, but the Cloudflare incident should not be treated as evidence that every deployment or model experienced either problem.

Builder Takeaway

For one disaggregated SGLang deployment, compare the running prefill and decode nodes’ KV pool capacity with the configured context limit after a model or memory configuration change. In a safe test environment, verify that requests exceeding capacity fail before handoff and that synthetic markers from one test request do not appear in another. Record the deployed engine version and compare it with the upstream fixes; Cloudflare recommends v0.5.22 or later for operators, while SGLang describes backend-specific fixes and follow-up work. A released version is not evidence that it has been deployed or that every related issue is resolved.