BOS Token Altered 6 of 15 Bielik-Minitron Responses
A community experiment using Bielik-Minitron 7B v3.0 Instruct found that adding one beginning-of-sequence token changed 6 of 15 responses under Ollama 0.40.14. Across 135 trials, the model passed 11 of 15 tasks without BOS and 12 of 15 with BOS, but the author said the narrow test does not show that BOS generally improves quality. The author also said it remains unclear whether Ollama’s omission of the token from this model’s chat template should be considered a bug.
The r/LocalLLaMA report shows that identical-looking prompts can produce different model behavior across inference engines, cautioning benchmarkers to control tokenization, chat templates, cache state and computation details.
I'm back with another experiment.
After the last heated discussion about KV cache, temperature 0, and reproducibility — where I got quite a beating for both my post and my writing style (partly deserved, to be fair ;) ) — I decided to investigate how models behave in Ollama and llama.cpp.
And, as usually happens with these experiments, I found something I wasn't even looking for.
The Bielik-Minitron 7B chat template stored in the GGUF includes the beginning-of-sequence token <s>, but the template Ollama uses for this model doesn't add it.
I thought to myself: come on, man, it's just one token.
But I decided to check whether its sudden appearance actually changes anything.