Community author tests small local LLMs on Polish, JSON and Python
A Reddit author created “Zacharski tests,” an open-source 15-task suite for evaluating small local models through Ollama on a laptop with 8 GB of VRAM. It covers eight Polish tasks, four JSON tasks and three Python syntax checks, with one answer per task, temperature 0, a 2k context and thinking disabled. The initial gemma2:9b Q4_0 result was 7/7 on JSON and code tasks but 3/8 on Polish tasks.
The compact test gives users a reproducible way to check local models under their own hardware and quantization, though its small sample and single-model result are not a general ranking.
I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).
The tasks:
- Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).
- JSON (4): scored strict and lenient.
- Python (3): syntax check only.
Results so far:
| Model | Quant | JSON + code | Polish |
| gemma2:9b | Q4_0 | 7/7 | 3/8 |
#Zacharski tests#Ollama#gemma2:9b#Qwen3#Llama 3.1#Gemma 3#Bielik 11B#Bielik-Minitron 7B#Open PL LLM Leaderboard#SpeakLeash#NVIDIA GeForce RTX 5050