BOS token改变6个模型回答
一项社区实验发现,在 Ollama 0.40.14 中使用 Bielik-Minitron 7B v3.0 Instruct 时,加入一个序列起始 token 会改变15个回答中的6个。全部135次试验中,不含 BOS 时通过11项任务,含 BOS 时通过12项,但作者明确表示,这一狭窄测试不能证明 BOS 通常能提升模型质量。Ollama 的该模型聊天模板是否遗漏此 token 并构成缺陷,目前仍不确定。
这项 r/LocalLLaMA 报告表明,看起来相同的提示词在不同推理引擎中可能产生不同结果,基准测试需控制分词、聊天模板、缓存状态和计算细节。
I'm back with another experiment.
After the last heated discussion about KV cache, temperature 0, and reproducibility — where I got quite a beating for both my post and my writing style (partly deserved, to be fair ;) ) — I decided to investigate how models behave in Ollama and llama.cpp.
And, as usually happens with these experiments, I found something I wasn't even looking for.
The Bielik-Minitron 7B chat template stored in the GGUF includes the beginning-of-sequence token <s>, but the template Ollama uses for this model doesn't add it.
I thought to myself: come on, man, it's just one token.
But I decided to check whether its sudden appearance actually changes anything.