OpenWiki Evidence Graph Improves Agent Memory Consistency
LangChain tested OpenWiki as a persistent memory layer for agents, finding in a vendor experiment that it was roughly five times more accurate than search alone on questions requiring information across multiple sources. Adding an evidence graph over wiki citations improved accuracy by a further 27%, doubled citation consistency and reduced cost per correct answer by 58%, though retrieval remained non-deterministic. For simple lookups, search without OpenWiki was equally accurate and faster, while reported answer costs excluded compilation and maintenance expenses.
The experiment suggests that compiled memory and explicit evidence paths can help agents perform repeated cross-source reasoning, but its benefits depend on reuse and have not been independently confirmed.
Key Takeaways
- A compiled wiki gives agents domain context that retrieval alone must reconstruct. On questions spanning multiple sources, an agent using OpenWiki as a memory layer was 5x more accurate than one using grep tools alone.
- Adding an evidence graph helps agents follow the wiki back to supporting evidence. We turned citations and document references into explicit graph edges agents could follow. Compared with the wiki alone, this doubled how consistently the agent cited the same evidence, improved accuracy by 27%, and cut cost per correct answer by 58%.
- Save compiled memory for repeated cross-source reasoning. The wiki captures domain knowledge and complex relationships. Adding an evidence graph improves accuracy and retrieval consistency. For simple lookups, however, the agent without memory was just as accurate and faster.
Agents working across an organization’s policies, manuals, wikis, tickets, and code must navigate knowledge scattered across systems and teams. Finding a relevant passage is only the start. They also need to determine which rules apply, what has changed, and how one source affects another.
A persistent memory layer preserves knowledge and relationships for agents to reuse across tasks. Without it, agents often have to reconstruct that understanding on every question, making their work expensive and error-prone. This is especially challenging in banking and insurance, where policies and guidelines frequently amend or supersede one another.