WMT23
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Apple study: LLMs remain unreliable for translation quality annotation
Apple Machine Learning Research published an EMNLP paper evaluating how well large language models can annotate machine translation quality, comparing their agreement with human annotators under two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The study uses a long-context test set of 70 language pairs plus the public WMT23 and WMT25 data across many language pairs and domains. It found that LLM agreement with humans exceeds human-human agreement for some tasks, but varies substantially by annotation scheme, language pair and domain and remains unreliable in most settings; humans particularly struggle with fine-grained MQM annotation and low-resource pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. The authors propose human-LLM collaborative annotation pipelines to address these reliability gaps.
Apple Machine Learning Research · 🔥 0
Experience and discussion from the community
Share my WMT23 experienceAsk about WMT23
Nobody has shared their experience with WMT23 yet.