苹果研究:LLM 翻译质量标注仍不可靠
苹果机器学习研究团队在 EMNLP 发表论文,评估大语言模型用于机器翻译质量标注的可靠性,将其与人工标注者的一致性同 MQM(多维质量指标)和 ESA(错误片段标注)两套方案下的标注结果进行对比。研究使用包含 70 个语言对的长上下文测试集以及公开的 WMT23 和 WMT25 数据,覆盖多种语言对与领域。结果显示,LLM 在某些评测任务上与人工标注者的一致性高于人工标注者之间的一致性,但整体一致性随标注方案、语言对和领域大幅波动,在多数场景下仍不可靠;人类在细粒度 MQM 标注和低资源语言对上尤其吃力,LLM 则难以处理细微错误、错误语言变体和错误片段标注。作者提出,人机协同标注流程有望改善这些可靠性问题。
这说明把翻译质量评估大规模交给模型仍需谨慎,短期内人机协同标注比完全替代人工评审更现实。
research area Speech and Natural Language Processingconference EMNLP
content type paperpublished October 2026
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
AuthorsHala Almaghout, Christian Federmann, Qin Gao
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.