Rethinking Web Cache Design for the AI Era

Abstract

Web caches have long been effective at reducing latency and back- end load by storing popular content close to users, exploiting the temporal and spatial locality of human-driven access patterns. How- ever, the rise of AI-generated traffic is challenging this assumption. AI agents such as search crawlers and data scrapers issue large vol- umes of diverse, low-referrer requests with minimal reuse, which degrade cache effectiveness, interfere with human-relevant content, and increase pressure on backend systems. In this paper, we argue that caching infrastructure must evolve to address this shift. We analyze emerging AI traffic patterns and study their impact on caching performance using a CDN prototype based on Wikime- dia’s architecture. Our results show that even modest amounts of AI traffic lead to significant cache inefficiency. We propose a new cache design paradigm in which caches act as lightweight filters to preserve locality for human traffic and apply differentiated policies or tiers to handle AI traffic more efficiently.