ZeroRAG: Letting Queries Find Knowledge

The common approach to RAG is to first gather documents into a vector database, then retrieve context from it in response to a query. This flow is straightforward, but it has an organizational premise: the data owner is willing to entrust the material to the same infrastructure. In cross-organizational collaboration, this is often the hardest step to coordinate. Hospitals, research institutions, and companies may be willing to answer a question while continuing to keep custody of their own documents. ZeroRAG studies exactly this constraint: if documents remain on their respective nodes, with no global index, how can a query still find relevant knowledge? The paper's approach is to have each node use a small amount of local information to recommend the next destination, while leaving the experience accumulated during a query in the network. Retrieval therefore does two things at once: it finds the documents needed for the current query and improves the paths used to find documents in the future. 1. Custody 1.1 Two layers of retrieval Let's start with the smallest RAG pipeline. After documents are chunked, an embedding model encodes them, and the query is mapped into the same vector space. The retriever finds similar document chunks and passes them, along with the question, to the generative model. There are two separable responsibilities here: deciding what is relevant and deciding where to retrieve it from. A centralized vector database combines these responsibilities, so we often treat them as a single step. Figure 1: Centralized RAG places document indexing, relevance retrieval, and context acquisition in the same pipeline. Once documents are held by different institutions, the retriever must first find the appropriate owners and then request content from them. Each node can still run a local vector database; what changes is the layer between those local databases. A layer of node discovery and query routing is needed to connect the distributed document stores. ZeroRAG primarily redesigns this layer; the generative model still receives the retrieval results and answers the question. This separation also preserves the owner's decision-making authority. A node can choose which vectors to publish for routing and, after receiving a request, enforce access controls related to identity, purpose, or content. The actual document transfer and authorization mechanisms fall outside ZeroRAG's routing protocol. What we discuss next is “how to find nodes that may have something useful,” while the choice of which content to return is jointly determined by the owner and the application layer. 1.2 The cost of coordination The most direct approach is to broadcast the query to every node. As long as the documents are accessible, this provides excellent coverage, but every retrieval requires participation by the entire network. As the number of nodes grows, this quickly becomes expensive. Another approach is a distributed index. For example, vectors could be mapped to buckets using locality-sensitive hashing, with a distributed hash table locating the bucket owner. This decentralizes storage and control, but it still relies on a shared address space and index structure across the network. That structure must be maintained when nodes join or leave and documents are updated. ZeroRAG seeks to narrow the scope of coordination further, allowing only the nodes currently encountered to exchange the necessary information. The design question therefore becomes tighter: if a node knows only a small number of neighbors, how can it decide whom to ask next? The answer has to begin with the local state that each node can retain. 2. Semantic routing 2.1 Neighbor advertisements When nodes connect, they exchange advertisements used for routing. Here, an advertisement is a set of document embeddings representing the content clues a node is willing to expose externally. The original documents remain under their respective nodes' custody, while the advertisement describes “what might be available here.” Why retain a set of vectors rather than compress an entire node into one average vector? An institution may hold documents on several topics at once. Averaging several directions can dilute a highly relevant small topic. ZeroRAG takes the highest similarity across multiple clues from the same node, allowing one clue sufficiently close to the query to bring that node onto the candidate list. Each node's persistent state can be divided into four parts: Local documents D: content retrievable from the local vector store. Neighbor list N: capacity-bounded outbound connections, maintained in first-in, first-out order. Neighbor advertisements A: document vectors and node identities published by neighbors. Query memory M: recently received query vectors and their originating nodes, which will later supplement routing clues. D describes what can be answered locally, while A helps determine what neighbors may provide. These two components alone are enough to begin a progressively expanding search. M also allows the search to take advantage of paths already taken by other nodes. 2.2 Candidate frontier During a query, the initiating node maintains a candidate frontier: a set of nodes ranked by expected relevance. It takes the node currently considered most worth visiting, sends it the query vector, and, upon receiving a reply, merges the peer's local results into a temporary top-k while returning newly recommended nodes to the frontier. Visited nodes are recorded to avoid duplicate queries. This is a best-first search. The direction of each request is determined by neighbor recommendations, but the entire search remains organized by the initiating node: it contacts different nodes in sequence, gathers results and recommendations, and decides where to go next. The multi-step path in the diagram should therefore be understood as the process of discovering additional candidate nodes. Figure 2: The initiating node expands its search through recommendations, adds neighbors after the query ends, and uses received similar queries as clues for subsequent recommendations. In the left side of the paper's original figure, a queries different candidate nodes in sequence and ultimately finds i, which provides the best results. The right side previews a different path: b remembers a question that a previously asked and later recommends h, which asked a similar question, to a. The two sides correspond to “moving toward someone who may hold the document” and “moving toward someone who may know the path.” The search ends when the frontier is exhausted or the contact budget is used up, after which the highest-ranked documents are requested from the corresponding owners. The main experiments set the contact budget to 50, limiting the number of nodes a single retrieval can visit. This budget provides a clear upper bound on communication and makes retrieval best-effort: a limited number of local explorations in exchange for the best possible results. 3. Query experience 3.1 Who asked what If nodes are recommended based only on document advertisements, the network mainly remembers the distribution of content. Yet a successful retrieval also produces another kind of information: the querier may have found relevant nodes, established connections, and perhaps obtained documents after receiving permission. Suppose a asks b about a medical question. b can store a record stating that a asked this question and what its query vector was. Later, when h asks b a similar question, b has a reason to recommend a. a's value comes from the knowledge and connections it has just developed, not only from the document advertisements it originally published. The query cache stores query vectors and querier identities. It allows later queries to find nodes with similar needs and then continue the search through those nodes' documents or connections. Incorporating a history of needs expands the basis for judging “who might be able to help.” The scoring itself remains simple: for each candidate node, the system compares its associated document advertisements and historical queries, taking the highest cosine similarity among them. A node receiving a query returns its top-k document clues along with the highest-scoring node recommendations. It also writes the current query and initiator into a capacity-bounded FIFO query memory. This mechanism relies on an exploitable association: similar needs often lead to similar material. When needs recur or topics overlap, query memory can reuse prior exploration. For a highly distinctive new question, the network must still rely on document advertisements and further search. 3.2 Neighbor reconnection After finding a useful node, the discovery must also be preserved. At the end of every query, ZeroRAG attempts to connect to the node that provided the highest-scoring documents. If no document results are obtained, it can instead select the node with the highest recommendation score. If the neighbor list is full, the oldest connection is removed; each query can add at most one connection. Thus, two kinds of local state accumulate experience in different ways. Query memory lets a queried node know “who has recently cared about this area,” while neighbor reconnection lets the querier retain “who just helped me.” The former changes the basis for recommendations; the latter changes whom the node can directly contact. Later queries may therefore enter the appropriate semantic neighborhood sooner. This also explains why FIFO capacity is part of the design. If a node has more areas of interest than it can retain connections for, neighbors associated with new interests displace those associated with old interests. The effective organization the network forms depends on both capacity and the distribution of demand. Continuously changing interests may keep the connections in flux rather than allowing them to settle into a fixed topology. At this point, ZeroRAG's core is complete: document advertisements provide content clues, query memory provides demand clues, best-first search combines them, and neighbor reconnection turns successful exploration into connections that can be used later. Next, we examine the workloads on which these local actions produce gains. 4. Retrieval performance 4.1 Real-world demand The paper uses three workloads to evaluate single-target retrieval, retrieval of multiple relevant documents, and generation after retrieval. First, let's look at the first two. AOL-IA comes from query and click logs. The authors selected the 10,000 users with the most queries, mapped each user to a node, and initialized local storage with the documents that user had clicked. Query events were replayed at random, and the task was to find a designated target document on other nodes; recall@1 here can be understood as the success rate for recovering the target document. Figure 3: In the AOL-IA simulation, ZeroRAG's target-document recovery rate in the final batch is higher than that of two decentralized baselines. In the final batch of queries, ZeroRAG's recall@1 was 0.91, compared with 0.65 for Semantica and 0.80 for DHT-LSH with 50 hash tables. Semantica describes a node with a single average vector, whereas ZeroRAG retains multiple independent clues, providing finer-grained routing information for nodes with multiple topics. The communication-overhead comparison must also be considered in the context of the configuration. The paper estimates DHT-LSH at 10,000 nodes and 50 hash tables as requiring approximately 50 × log₂(10,000), or 664 request–response round trips, to complete the corresponding bucket lookups. ZeroRAG's budget is at most 50 node-query round trips. This shows that multi-bucket lookup is costly in the baseline configuration. The comparison covers routing-query overhead; document transfer and background maintenance are counted separately. Amazon Electronics tests whether multiple relevant results can be found together. The authors construct queries from product-category paths and assign demands and local documents based on reviewer–product relationships. This workload therefore contains many repeated, overlapping category queries, making it well suited to observing whether similar demands can be reused. Figure 4: When grouped by the number of relevant products available, ZeroRAG recovers a high proportion of relevant results in all three groups. For queries with 1–10, 11–50, and 51–100 relevant documents available, the authors return at most 10, 50, and 100 results, respectively. ZeroRAG's recovery rate across the three groups is 94.8%–98.9%, compared with 90.3%–91.7% for DHT-LSH with 50 hash tables. The gain here concerns coverage of multiple results rather than the hit rate for a single answer; repeated category demands also provide a strong signal for query memory. 4.2 Generation quality After documents are found, we still need to determine whether they help a generative model answer the question. The paper uses 1,000 biomedical questions and answers from PubMedQA as its base and adds five model-generated paraphrases to each question. Topic preferences are used to simulate the interests and document distributions of different nodes. The default setting contains 50,000 simulated nodes, 30 queries per node, and a total of 1.5 million query events. The end-to-end experiments retrieve five documents at a time and pass them to Phi-3.5-mini-instruct, then check whether the answers match the dataset's yes, no, or maybe labels. A centralized HNSW index with a global document index serves as the reference for this setting. Figure 5: In the PubMedQA simulation, ZeroRAG's retrieval and answer performance gradually approaches the centralized reference, while the number of requests needed to find the first relevant document also decreases. At the end of the experiment, ZeroRAG's retrieval recall was 93.0%, compared with 97.3% for HNSW, while answer accuracy was 77% and 79%, respectively. Retrieval still trails by about 4.3 percentage points, while answer accuracy trails by 2 percentage points. These results support a specific conclusion: under this corpus, distribution, and generative-model configuration, the context recovered through local routing is sufficient to support classification answers close to those of the centralized reference. The right side of the original figure also shows that, as query experience accumulates, fewer requests are needed to find the first relevant document. Ultimately, 99% of queries find their best supporting document within no more than 13 requests. This measures “when the first useful result appears”; complete top-5 retrieval may continue exploring. The centralized reference's recall also does not reach 100%. Whether the target document ranks in the top five is still affected by the embedding model and question paraphrasing. ZeroRAG improves node discovery and local search paths, but semantic representation quality still jointly determines the final retrieval ceiling; the answer metric here is limited to matching the three labels. 4.3 Which part works The preceding results arise from the combined effects of document advertisements, query memory, and topology adjustment. The ablation study further answers one question: where does most of the effect come from? Figure 6: Removing query memory lowers retrieval performance, and the Amazon multi-relevant-result task is especially sensitive to this information. On AOL-IA, removing query memory reduced average recall@1 from 0.89 to 0.85; the decline was more pronounced on Amazon's multi-relevant-result task. Disabling topology adjustment alone left results close to those of the complete system, while removing both components produced worse results. This comparison suggests that the key point to understand is how demand history provides additional routing clues, with neighbor reconnection working alongside it to improve subsequent exploration. The cache-capacity experiment for PubMedQA provides a more direct engineering reference. The following table reports the final 10% of queries and gives the mean and standard deviation across the authors' five runs. Cache entries Target recovery rate Requests to find the target 0 77.860% ± 0.177% 16.796 ± 0.067 10 85.834% ± 0.120% 12.757 ± 0.041 100 97.612% ± 0.037% 5.900 ± 0.017 1,000 99.652% ± 0.007% 3.448 ± 0.006 10,000 99.652% ± 0.007% 3.448 ± 0.006 Increasing the cache from 0 to 100 entries markedly improves both recovery rate and request count; further increases produce progressively diminishing returns. This capacity experiment measures target-document lookup, so its measurement criteria differ from the earlier top-5 retrieval and generation evaluations. It is best understood as an indication of the role of routing state. 5. Changes in the network 5.1 Failure recovery A network whose paths are formed through history loses part of that history when many nodes go offline. At the midpoint of the PubMedQA simulation, the paper simultaneously removes 50% of the nodes and then gradually replaces them with new nodes during the second half of the experiment. Figure 7: After half the nodes leave simultaneously, replacement nodes gradually rebuild routing state through subsequent queries, and target recovery recovers accordingly. Recall drops immediately after the failure. As replacement nodes receive queries, accumulate query memory, and form connections, recall@1 over the final 10% of queries recovers to 95.5%. The average number of requests needed to find a relevant document rises from 8.9 before the failure to 15.1 afterward, showing that usable paths have been restored but the neighborhoods around the new nodes are not yet as efficient as those around the original nodes. Recovery depends on subsequent queries supplying new local information and on the availability of remote material to replacement nodes. This experiment observes how routing state is rebuilt. In an actual deployment, if a document is held only by one node and that node remains permanently offline, content availability still requires an arrangement on the owner's side. 5.2 New topics Beyond node failure, there is another common but less severe change: new content appears, but existing paths do not yet point to it. The authors first remove a topic from the PubMedQA workload and reintroduce it when the experiment reaches 60%, while changing the number of nodes that initially hold documents on that topic. From then on, all nodes continue querying the topic. Figure 8: The speed at which a new topic becomes reachable depends on the number of initial holders; a single holder requires a longer discovery and propagation process. When there are 10 or more initial holders, relevant documents are found quickly. With only one holder, recovery is markedly slower. This experiment also allows documents from successful retrievals to enter other nodes' local stores, so the curves reflect the combined effects of routing discovery and authorized content propagation. For engineering design, these results identify a specific source of cold-start difficulty: sparse content must first be touched by an exploration, after which subsequent demands can use the newly formed clues. More initial entry points increase the chance of discovery. If the application prohibits subsequent document replication, the propagation of a new topic would need to be measured again under that constraint. 6. Engineering implications 6.1 Scale and state ZeroRAG's scalability comes first from bounded state. Each node retains a limited number of neighbors, advertisements, and query-memory entries, and each query limits the number of nodes contacted. If the embedding dimension is d, each neighbor publishes at most a vectors, the neighbor capacity is δn, and the query-memory capacity is δm, then the main term for persistent routing state is d × (δn × a + δm), plus node identities and similar information. The size of the local document store is calculated separately. Once these parameters are fixed, a node need not maintain a larger global directory as the size of the entire network grows. The communication budget also does not automatically grow with the total number of nodes, but achievable recall still depends on content distribution, demand overlap, and network organization and must be tested experimentally. The paper extends the PubMedQA simulation to 100,000 nodes. The complete scale-experiment results are reproduced below, still covering the final 10% of queries, with means and standard deviations over five runs. Number of nodes Target recovery rate Requests to find the target 1,000 99.365% ± 0.272% 3.209 ± 0.111 5,000 98.276% ± 0.066% 4.330 ± 0.059 10,000 98.378% ± 0.051% 4.656 ± 0.015 50,000 99.652% ± 0.007% 3.448 ± 0.006 100,000 99.913% ± 0.005% 5.724 ± 0.011 This statistic includes only queries for which the target document was advertised by at least one remote node. It therefore answers the question of whether an existing remote target can be found. The scale experiment uses a limited PubMedQA corpus and a synthetic node distribution. The significance of 100,000 nodes is that the routing mechanism remains effective for a larger set of nodes; document diversity and deployment complexity require broader evaluation. 6.2 Deployment interfaces These results come from a system simulation. The authors use a global directed graph to represent node connections and procedure calls to simulate message exchange. Document embeddings are stored in unified memory, with each node's local store referencing those records. Protocol decisions use local state, while the experimental execution environment centrally manages simulation state. Connecting this to a real system requires filling in the network, storage, and permission interfaces. First is the semantic interface. Nodes must use the same embedding model to compare queries and advertisements directly. Cross-organizational systems must agree on model versions, vector formats, and migration procedures after a model upgrade. The lack of a global index therefore still requires a shared representational foundation. Second is the access-and-trust interface. Document custody remains on the owner's side, but query vectors, advertisements, and querier identities are still exchanged during routing. Deployment must determine which clues may be exposed, which queries may be logged, how long records are retained, and how nodes are authenticated. The paper uses a permissioned network and leaves Byzantine behavior, malicious recommendations, and related issues outside the scope of the research; sensitive scenarios require dedicated security and privacy design for these interfaces. Finally, there are performance and history-storage interfaces. The paper simplifies the process by which a node obtains a document and its provenance record through a HISTORY() call, but a real system must connect this to a storage layer with authorization and provenance verification. The paper also reports centralized and ZeroRAG processing times of 24 ms/query and 379 ms/query, respectively, with ZeroRAG's figure excluding network latency; the original description of the timing methodology is also brief. These figures are better treated as a reminder to pay attention to the computational cost of multiple local queries. Online latency must also include connections, timeouts, document transfer, and permission checks. Parallel requests can reduce waiting rounds, but under the same contact budget, a greater fan-out compresses the depth that can be explored. With a total budget of 30, the paper increases requests per round from 3 to 10, reducing the average number of discovery rounds while recall@1 falls from approximately 99.37% to 97.46%. If the recommendation chain requires several steps to reach the correct neighborhood, budget should be reserved for vertical exploration. Concurrency should therefore be chosen together with network latency, request cost, and routing depth. 7. Conclusion Returning to the opening question: when documents are held by different owners, how can distributed knowledge be connected into a retrievable system? ZeroRAG separates local content retrieval from cross-node discovery, uses document advertisements to describe supply, query memory to describe demand, and links the two through budgeted search and neighbor reconnection. A query leaves behind more than results: it also leaves node clues and connections that later queries can use. The most important design insight from this paper is that a history of demands can also become part of the retrieval infrastructure. A node's value as a source of help may come from what it holds, but it may also come from what it has recently sought and whom it has consequently come closer to. The experimental gains are especially pronounced on workloads with repeated, overlapping demands, offering a route that can continue to be validated for cross-organizational knowledge collaboration. To put this approach into practice, it must still be returned to concrete constraints: whether content may be stored elsewhere, whether query histories may be shared, how node identities can be trusted, and how deep exploration can be retained under a real network budget. Routing coordinates only the local interactions currently needed, while documents and permissions remain managed by their respective owners. The interface between the two will determine how far this design can go. The original figures in the article come from Petru Neague and colleagues' ZeroRAG paper and are used under CC BY 4.0. The original figure content is preserved; the SVG diagrams have only been fully rasterized and proportionally formatted. References ZeroRAG: Adaptive Semantic Peer Routing for Decentralized Retrieval-Augmented Generation: https://arxiv.org/abs/2611.07928

ZeroRAG RAG 语义路由去中心化检索