Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Bits or permutations? Allocating a fixed inference budget

AI summary

An arXiv paper compares lower-bit weights with averaging answer-option orders under a fixed inference budget on Qwen3-0.6B and Qwen3-1.7B. Using simulated round-to-nearest weight quantization and 300 items each from ARC-Easy, ARC-Challenge, MMLU and HellaSwag, the authors report that 6-bit weights scored 1.8 and 0.9 points above fp16 on the two models, although they do not explain the difference. At 4 bits, answer-slot skew increased, and averaging four cyclic orders recovered 12.6 points for the 0.6B model. The cost-matched analysis favors 6-bit weights plus up to four single-pass orderings when weights use at least 6 bits. All results are simulated and limited to two small models and the simplest quantizer.

In-depth insight

The study provides a concrete budget-allocation rule for small local models: use 6-bit weights and spend remaining budget on as many as four answer-order passes. The authors explicitly limit the evidence to simulated quantization on two small models.

Computer Science > Artificial Intelligence

arXiv:2611.14413 (cs)

[Submitted on 10 Nov 2026]

Title:Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models

Authors:Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri

View a PDF of the paper titled Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models, by Mohammed Nayeem Khan and 2 other authors

#Qwen3-0.6B#Qwen3-1.7B#ARC-Easy#ARC-Challenge#MMLU#HellaSwag

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.