Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

MMLU

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 4

  1. LLM Judges Show Weak Link to General Ability in Writing Evaluation

    A study accepted at the NeurIPS 2026 Can We Trust AI Evaluation? Workshop compares LLM judges with human preferences on three long-form writing tasks. Across four text generators, judge quality varied by task and criteria, correlated weakly with MMLU performance at Pearson’s ρ ≤ 0.2, and improved with paired comparisons; the strongest LLMs did not consistently reproduce human judgments.

    Apple Machine Learning Research · 🔥 0
  2. Bits or permutations? Allocating a fixed inference budget

    An arXiv paper compares lower-bit weights with averaging answer-option orders under a fixed inference budget on Qwen3-0.6B and Qwen3-1.7B. Using simulated round-to-nearest weight quantization and 300 items each from ARC-Easy, ARC-Challenge, MMLU and HellaSwag, the authors report that 6-bit weights scored 1.8 and 0.9 points above fp16 on the two models, although they do not explain the difference. At 4 bits, answer-slot skew increased, and averaging four cyclic orders recovered 12.6 points for the 0.6B model. The cost-matched analysis favors 6-bit weights plus up to four single-pass orderings when weights use at least 6 bits. All results are simulated and limited to two small models and the simplest quantizer.

    Hugging Face · Papers · 🔥 0
  3. Study links retrieval context rot to competition, not length alone

    A paper submitted to arXiv by Yinuo Liu reports that neutral filler preserved 0.97–1.00 retrieval accuracy through 32K tokens on Qwen2.5-1.5B, while competing statements at the same positions reduced accuracy to 0.11–0.21. Across 126,473 trials, reduced attention on the evidence explained up to 58% of loss variance, compared with at most 8% for log length, although the author says this relationship is predictive rather than causal.

    Hugging Face · Papers · 🔥 0
  4. MoveBench Benchmarks LLMs for Sports and Health Coaching

    MoveBench is a rubric-based benchmark for evaluating large language models in sports and health coaching, combining multiple-choice questions with open-ended answers grounded in real-world scenarios. The paper evaluated 15 LLMs on 809 questions and 5,012 rubric-level criteria, with the authors reporting substantial variation in safety awareness, program design, and personalization. It was submitted to arXiv on October 19, 2026 and revised on October 26.

    Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my MMLU experienceAsk about MMLU

The Past and Present of Gödel Machines: The Research Path Toward RSI

A coding agent can fix a program, yet the next time it encounters an error, it may still follow the same process of debugging, editing, and testing. The task outcome may have changed, but the method that produced it may not have. When the bottleneck lies precisely in that method, who is responsible for improving the program responsible for improvement? Gödel Machine (GM) [1] framed this as a proo…

RSIGödelMachine自我改进开放式进化Agent