An 11-Case Golden Set for Measuring RAG Retrieval Quality
A smart document assistant developer built an offline RAG evaluation using 11 fixed questions, scoring whether an essential answer phrase appears within the top retrieved chunks and its reciprocal rank. The default parent-child chunking, hybrid retrieval, and reranker top-5 configuration achieved recall@1 of 0.455, recall@3 of 0.818, recall@5 of 1.000, and MRR of 0.652; flat chunking produced lower results, while reranker top-2 reduced recall@3 and MRR. Disabling keyword retrieval produced the same aggregate metrics as the default configuration.
The report provides a reproducible method for isolating the effects of chunking, retrieval paths, and reranking, but the developer explicitly warns that 11 questions support only limited conclusions and recommend 30 to 100 production cases as an experience-based estimate rather than a measured result.
11 道题给 RAG 装了个仪表盘，读数推翻了我三个想当然
智能文档问答这个助手我改了很久，RAG 效果好坏一直靠“感觉答得还行”判断。上周加了一套离线评测，一轮跑完，三个我想当然的地方一个没剩下。这套评测只有 11 道固定题，打一次拿 recall@K 和 MRR；同一份语料、同一套题，每次只改一个配置开关。
只做一件事：每题存一句标准答案里必不可少的话，把问题丢给检索链路，看这句话在不在返回的前 K 条里、排第几。生成的答案它不碰——材料没到手，prompt 再漂亮也补不回来。
题集就是 src/main/resources/rag-golden-set.json，一条用例三个字段，判指标只看中间那个。文件开头第一条长这样，referenceAnswer 那句完整有 48 字、这里截了尾，往下还有 10 条同结构的：
{
"cases": [
{
"question": "七天无理由退货需要满足什么条件？",
"expectedContextContains": "不影响二次销售",
"referenceAnswer": "商品需保持完好、不影响二次销售，…"
}
]
}