Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Error Span Annotation

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. Apple study: LLMs remain unreliable for translation quality annotation

    Apple Machine Learning Research published an EMNLP paper evaluating how well large language models can annotate machine translation quality, comparing their agreement with human annotators under two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The study uses a long-context test set of 70 language pairs plus the public WMT23 and WMT25 data across many language pairs and domains. It found that LLM agreement with humans exceeds human-human agreement for some tasks, but varies substantially by annotation scheme, language pair and domain and remains unreliable in most settings; humans particularly struggle with fine-grained MQM annotation and low-resource pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. The authors propose human-LLM collaborative annotation pipelines to address these reliability gaps.

    Apple Machine Learning Research · 🔥 0

Experience and discussion from the community

Share my Error Span Annotation experienceAsk about Error Span Annotation

Nobody has shared their experience with Error Span Annotation yet.