Speculative Runahead Retrieval for Storage-Intensive Agentic RAG

초록

Retrieval-Augmented Generation (RAG) integrates large language models (LLMs) with external knowledge bases to produce reliable responses without increasing model capacity. Modern RAG systems are rapidly scaling beyond DRAM to rely on SSD for massive knowledge bases. In addition, they operate in an agentic manner, where LLMs autonomously generate queries through iterative reasoning and retrieval. However, they introduce substantial latency as the GPU stalls after each reasoning phase to wait for intermediate retrieval phases. We propose a novel system to reduce the latency of the agentic iteration in storage-intensive agentic RAG. We introduce a runahead model that speculatively initiates the retrieval phase while the primary model is executing its reasoning phase. The runahead model operates as an intelligent prefetcher, proactively staging relevant embeddings from the SSD to DRAM. When the primary model issues the actual queries, the required embeddings are mostly accessible in DRAM, effectively masking I/O latency overhead.

키워드