Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Apple study: LLMs remain unreliable for translation quality annotation

AI summary

Apple Machine Learning Research published an EMNLP paper evaluating how well large language models can annotate machine translation quality, comparing their agreement with human annotators under two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The study uses a long-context test set of 70 language pairs plus the public WMT23 and WMT25 data across many language pairs and domains. It found that LLM agreement with humans exceeds human-human agreement for some tasks, but varies substantially by annotation scheme, language pair and domain and remains unreliable in most settings; humans particularly struggle with fine-grained MQM annotation and low-resource pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. The authors propose human-LLM collaborative annotation pipelines to address these reliability gaps.

In-depth insight

It signals that delegating translation quality evaluation to models at scale still calls for caution, with human-LLM collaborative annotation the more realistic near-term path.

research area Speech and Natural Language Processingconference EMNLP

content type paperpublished October 2026

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

AuthorsHala Almaghout, Christian Federmann, Qin Gao

View publication

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

#Apple#MQM#Error Span Annotation#WMT23#WMT25

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.