LSREP Tests Whether Conversational Memory Maintains Evolving State
Independent undergraduate researcher Deepnar introduced LSREP, a protocol for evaluating how persistent memory accumulates, changes and retrieves information during an evolving conversation. It combines chronological replay, lifecycle schedules, repeated probes, evolving reference answers and mechanism-fidelity checks, with ICE v2 used as a local-first architectural case study. Across 1,985 turns, 219 probes, 1,211 probe-checkpoint observations and 52 checkpoints, ICE v2 did not outperform pure vector-RAG in the matched LongMemEval evaluation.
LSREP offers a reproducible way to distinguish endpoint recall from reliable maintenance of conversational state, although its private single-user history and ongoing independent-preprint status limit the strength of the results.
| I'm an undergraduate researcher working independently on conversational memory systems. I developed LSREP to evaluate how persistent memory accumulates, changes and retrieves information over the course of an evolving conversation. Instead of relying only on isolated endpoint questions, LSREP combines chronological replay, lifecycle schedules, repeated probes, evolving reference answers and mechanism-fidelity checks. The accompanying architectural case study is ICE v2, a local-first conversational memory system. The evaluation covers 1,985 turns, 219 distinct probes and 1,211 probe-checkpoint observations across 52 checkpoints, using a private single-user history. An important part of the findings is that ICE v2 did not outperform pure vector-RAG on the matched LongMemEval evaluation. The ordinary-density replay results also did not establish a quality advantage. The paper discusses the limitations and mechanism-fidelity findings rather than treating the architecture as validated in its entirety. I'm particularly interested in feedback on the evaluation methodology: what else should a longitudinal memory benchmark test to distinguish successful recall from reliable maintenance of evolving conversational state? Paper: https://arxiv.org/abs/2609.16730 Code and evaluation artifacts: https://github.com/Deepnar/ice Frozen software artifact: https://doi.org/10.5281/zenodo.21759702 This is an arXiv preprint and an ongoing independent research project. (I couldn't find anything about the tag in the title in the rules, so i assumed [R] for research, if its wrong please inform) submitted by /u/Deepnar_S [link] [comments] |