Agent-written RAG summaries can drop access-control metadata
A developer using Chroma and LlamaIndex reports that background agents can save derived summaries without upstream allowed_groups or classification tags, making private source data retrievable through shared memory. The author’s offline sentinel-eval harness found no leaks in direct RAG queries, while two of three agent-memory synthesis tasks exposed six of 18 restricted canaries; the results are self-reported.
The report highlights a concrete access-control gap in multi-tenant agent memory, where metadata filters can work on direct queries but fail after an agent writes a derived summary.
Everyone relies on metadata filters like allowed_groups to keep multi tenant RAG locked down and under check. That works well when users query the vector store directly, which means unauthorized chunks will get filtered out with 0% leaks.
But, this falls apart in the write path once you have background agents.
If an agent has elevated read access to summarize docs across multiple folders or teams, it pulls those docs, mashes them into a summary chunk, and commits that summary back to a shared collection. On write, the upstream document access control lists just vanish. In a regular Chroma and LlamaIndex setup, the derived chunk gets saved with zero allowed_groups or classification tags attached. Anyone who runs a query to the shared memory is now gonna get the private source data that they shouldn't be allowed to read directly.
I put together an offline reproduction harness (sentinel-eval, Apache-2.0) to test this:
- Direct RAG: 0% leaks (18 public chunks retrieved, 0 restricted chunks exposed).
- Agent memory: 2 of 3 synthesis tasks leaked sensitive data (both cross boundary workloads failed, exposing 6/18 restricted canaries, the 1 public control passed).