Mica v0.5: a 4B decision model that runs on one consumer GPU
A developer on r/LocalLLaMA released Mica v0.5, a 4B decision model that runs on a single consumer GPU and follows the earlier Mica v0.1. In self-reported measurements on one RTX 6000 Ada, the developer says it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k tokens, claiming it is up to 3.8x faster than the other 4B judges measured, 12 to 19x faster than Perplexity's 27B decider and 25 to 340x faster than API decision models. The developer reports 84/111 on JevBench Hard, described as the best of any 4B model found with only Microsoft's 9B Decision-1 higher, and a JDI score of 58.8, while acknowledging that top 4B models on JDI are benchmark-focused and that training used data from the areas the benchmarks cover. Mica v0.5 handles yes/no, multiple choice and scores from 2 to 255 levels, accepts inputs up to 32k tokens, and is released under Apache-2.0 with INT8 and GGUF files, running on 8 GB cards and up.
It offers a small open-weight decision model with detailed speed, accuracy and deployment claims for local hardware, though all benchmarks are developer self-reported.
| I'm sharing Mica v0.5, the follow-up to Mica v0.1. It's 4B and runs on one consumer GPU. On speed, it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k on a single RTX 6000 Ada. That's up to 3.8x faster than the other 4B judges I measured, 12 to 19x faster than Perplexity's 27B decider, and 25 to 340x faster than the API decision models. On top of the optimizations described on HF, Mica Fast makes long inputs about 1.4 to 1.5x faster again. On accuracy, it gets 84/111 on JevBench Hard, the best of any 4B model I could find (only Microsoft's 9B Decision-1 is higher). On JDI it scores 58.8, ahead of every 9B and 35B model I tested. One honest note: the top 4B models on JDI are all benchmark-focused, and at least one of them was trained on nothing but benchmark-style data. To compete I did train on data from the same areas the benchmarks cover, but nothing contaminated, and most of the effort went into how it does on real work. That part comes from 200+ real use cases people posted on Reddit, Twitter and elsewhere. A few things I cared about. If you shuffle the answer options it still picks the same answer 97.3% of the time. It can hold back too, so if you set a threshold it tells you when it isn't sure instead of guessing. It handles yes/no, multiple choice, and scores from 2 to 255 levels, with inputs up to 32k tokens. If you have an NVIDIA card I'd go with the INT8 version, since it's the fastest. On the same GPU it runs about 2 to 2.7x faster than the Q8_0 GGUF on llama.cpp. GGUF files (BF16, Q8_0, IQ4_XS) are there for AMD, Apple and CPU. It runs on 8 GB cards and up, and it's Apache-2.0. HF: https://huggingface.co/sky7350/Mica-v0.5-4B GitHub: https://github.com/akivet/Mica Happy to answer questions, and I'd really like to hear where it breaks. submitted by /u/Top-Evidence174 [link] [comments] |