LSREP评测对话记忆的动态状态维护
AI 总结
独立本科生研究者Deepnar发布了LSREP协议，用于评测持久记忆如何在持续变化的对话中积累、更新并检索信息。该协议结合按时间回放、生命周期调度、重复探针、动态参考答案和机制保真检查，并以本地优先系统ICE v2作为架构案例。在包含1,985轮对话、219个探针、1,211次探针检查点观测和52个检查点的评测中，ICE v2未在匹配的LongMemEval评测中胜过纯向量RAG。
深度见解
LSREP提供了一种可复现的方法，用于区分终点回忆与可靠维护对话状态，但其私有单用户历史和仍在进行的独立预印本状态限制了结论可信度。
| I'm an undergraduate researcher working independently on conversational memory systems. I developed LSREP to evaluate how persistent memory accumulates, changes and retrieves information over the course of an evolving conversation. Instead of relying only on isolated endpoint questions, LSREP combines chronological replay, lifecycle schedules, repeated probes, evolving reference answers and mechanism-fidelity checks. The accompanying architectural case study is ICE v2, a local-first conversational memory system. The evaluation covers 1,985 turns, 219 distinct probes and 1,211 probe-checkpoint observations across 52 checkpoints, using a private single-user history. An important part of the findings is that ICE v2 did not outperform pure vector-RAG on the matched LongMemEval evaluation. The ordinary-density replay results also did not establish a quality advantage. The paper discusses the limitations and mechanism-fidelity findings rather than treating the architecture as validated in its entirety. I'm particularly interested in feedback on the evaluation methodology: what else should a longitudinal memory benchmark test to distinguish successful recall from reliable maintenance of evolving conversational state? Paper: https://arxiv.org/abs/2609.16730 Code and evaluation artifacts: https://github.com/Deepnar/ice Frozen software artifact: https://doi.org/10.5281/zenodo.21759702 This is an arXiv preprint and an ongoing independent research project. (I couldn't find anything about the tag in the title in the rules, so i assumed [R] for research, if its wrong please inform) submitted by /u/Deepnar_S [link] [comments] |