NUDT releases CMNIE, a joint-annotated Chinese military-news IE benchmark
According to a Juejin AI report citing the National University of Defense Technology, CMNIE contains 13,000 instances of Chinese military news jointly annotated under a single schema covering 7 event types, 10 argument roles, 7 entity types and 8 relations, with batches accepted only after four quality metrics all reach 95%. The benchmark compares supervised information-extraction models, zero-shot LLMs and fine-tuned LLMs on the same test set: zero-shot LLMs generally locate the relevant semantic units but lose substantial accuracy on exact span boundaries, while schema-conditioned fine-tuned generation performs best on relation extraction and explicit span models stay competitive on precise argument extraction. The report also cautions that the 13,000 instances are moderate in size, sufficient for fine-tuning but not for training from scratch, and that the military domain limits how far the schema and conclusions generalize.
The report argues that LLM extraction pipelines need an alignment or boundary-correction layer, and that the choice of loose versus exact match evaluation determines whether zero-shot LLM extraction appears usable or fails outright.
💡 一句话总结:国防科大发布中文军事新闻信息抽取基准 CMNIE(1.3 万实例,事件/论元/实体/关系联合标注)。最重要的发现不是「谁分高」,而是零样本 LLM 的失败模式:语义单元找得到,span 边界对不齐。用 LLM 做结构化抽取的团队,这个结论直接影响你的后处理设计。
导语:让 AI 读新闻写数据库,卡在哪一步?
情报分析的经典需求:把新闻流变成结构化数据库——谁、在哪、什么时候、对谁、做了什么、关系如何。这意味着事件抽取、论元识别、实体识别、关系抽取四件事联合做对,而不是各做各的。
通用领域的信息抽取数据集不少,但垂直领域(尤其是军事新闻:装备名、部队编制、行动术语高度特殊)一直缺统一 schema 的联合标注资源。于是「领域适配」变成每个团队的私活:自己标、自己评、不可比。