Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

BOS Token Altered 6 of 15 Bielik-Minitron Responses

AI summary

A community experiment using Bielik-Minitron 7B v3.0 Instruct found that adding one beginning-of-sequence token changed 6 of 15 responses under Ollama 0.40.14. Across 135 trials, the model passed 11 of 15 tasks without BOS and 12 of 15 with BOS, but the author said the narrow test does not show that BOS generally improves quality. The author also said it remains unclear whether Ollama’s omission of the token from this model’s chat template should be considered a bug.

In-depth insight

The r/LocalLLaMA report shows that identical-looking prompts can produce different model behavior across inference engines, cautioning benchmarkers to control tokenization, chat templates, cache state and computation details.

I'm back with another experiment.

After the last heated discussion about KV cache, temperature 0, and reproducibility — where I got quite a beating for both my post and my writing style (partly deserved, to be fair ;) ) — I decided to investigate how models behave in Ollama and llama.cpp.

And, as usually happens with these experiments, I found something I wasn't even looking for.

The Bielik-Minitron 7B chat template stored in the GGUF includes the beginning-of-sequence token <s>, but the template Ollama uses for this model doesn't add it.

I thought to myself: come on, man, it's just one token.

But I decided to check whether its sudden appearance actually changes anything.

#Bielik-Minitron 7B v3.0 Instruct#Ollama#llama.cpp#GGUF

Original · 5 min readRead the original
Group chat

No comments yet. Start the conversation.