创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

苹果研究:LLM 翻译质量标注仍不可靠

AI 总结

苹果机器学习研究团队在 EMNLP 发表论文,评估大语言模型用于机器翻译质量标注的可靠性,将其与人工标注者的一致性同 MQM(多维质量指标)和 ESA(错误片段标注)两套方案下的标注结果进行对比。研究使用包含 70 个语言对的长上下文测试集以及公开的 WMT23 和 WMT25 数据,覆盖多种语言对与领域。结果显示,LLM 在某些评测任务上与人工标注者的一致性高于人工标注者之间的一致性,但整体一致性随标注方案、语言对和领域大幅波动,在多数场景下仍不可靠;人类在细粒度 MQM 标注和低资源语言对上尤其吃力,LLM 则难以处理细微错误、错误语言变体和错误片段标注。作者提出,人机协同标注流程有望改善这些可靠性问题。

深度见解

这说明把翻译质量评估大规模交给模型仍需谨慎,短期内人机协同标注比完全替代人工评审更现实。

research area Speech and Natural Language Processingconference EMNLP

content type paperpublished October 2026

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

AuthorsHala Almaghout, Christian Federmann, Qin Gao

View publication

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

#Apple#MQM#Error Span Annotation#WMT23#WMT25

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。