Bits or permutations? Allocating a fixed inference budget
An arXiv paper compares lower-bit weights with averaging answer-option orders under a fixed inference budget on Qwen3-0.6B and Qwen3-1.7B. Using simulated round-to-nearest weight quantization and 300 items each from ARC-Easy, ARC-Challenge, MMLU and HellaSwag, the authors report that 6-bit weights scored 1.8 and 0.9 points above fp16 on the two models, although they do not explain the difference. At 4 bits, answer-slot skew increased, and averaging four cyclic orders recovered 12.6 points for the 0.6B model. The cost-matched analysis favors 6-bit weights plus up to four single-pass orderings when weights use at least 6 bits. All results are simulated and limited to two small models and the simplest quantizer.
The study provides a concrete budget-allocation rule for small local models: use 6-bit weights and spend remaining budget on as many as four answer-order passes. The authors explicitly limit the evidence to simulated quantization on two small models.
Computer Science > Artificial Intelligence
arXiv:2611.14413 (cs)
[Submitted on 10 Nov 2026]
Title:Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models
Authors:Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
View a PDF of the paper titled Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models, by Mohammed Nayeem Khan and 2 other authors
#Qwen3-0.6B#Qwen3-1.7B#ARC-Easy#ARC-Challenge#MMLU#HellaSwag