Mica v0.5：4B决策模型发布
AI 总结
开发者 /u/Top-Evidence174 在 r/LocalLLaMA 发布了 Mica v0.5，这是继 Mica v0.1 之后的 4B 决策模型，可在单张消费级 GPU 上运行。开发者自报在单张 RTX 6000 Ada 上，256 tokens 时每次决策耗时 19.8 毫秒，32k 时约 1.2 秒；并称其比所测其他 4B 评判模型快至多 3.8 倍，比 Perplexity 的 27B decider 快 12 到 19 倍，比 API 决策模型快 25 到 340 倍。开发者称该模型在 JevBench Hard 上取得 84/111，是其所找到的 4B 模型中最好的成绩，仅低于微软 9B 的 Decision-1；JDI 得分为 58.8，同时承认 JDI 上顶尖的 4B 模型都偏向基准测试，训练也使用了基准所覆盖领域的数据。Mica v0.5 支持是/否、多选以及 2 至 255 级打分，输入上限 32k tokens，以 Apache-2.0 许可发布，提供 INT8 与 GGUF 文件，可在 8 GB 及以上显存显卡上运行。
深度见解
它提供了一个小型开放权重决策模型，并给出了本地硬件上的速度、准确率与部署细节，但所有基准均为开发者自报。
| I'm sharing Mica v0.5, the follow-up to Mica v0.1. It's 4B and runs on one consumer GPU. On speed, it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k on a single RTX 6000 Ada. That's up to 3.8x faster than the other 4B judges I measured, 12 to 19x faster than Perplexity's 27B decider, and 25 to 340x faster than the API decision models. On top of the optimizations described on HF, Mica Fast makes long inputs about 1.4 to 1.5x faster again. On accuracy, it gets 84/111 on JevBench Hard, the best of any 4B model I could find (only Microsoft's 9B Decision-1 is higher). On JDI it scores 58.8, ahead of every 9B and 35B model I tested. One honest note: the top 4B models on JDI are all benchmark-focused, and at least one of them was trained on nothing but benchmark-style data. To compete I did train on data from the same areas the benchmarks cover, but nothing contaminated, and most of the effort went into how it does on real work. That part comes from 200+ real use cases people posted on Reddit, Twitter and elsewhere. A few things I cared about. If you shuffle the answer options it still picks the same answer 97.3% of the time. It can hold back too, so if you set a threshold it tells you when it isn't sure instead of guessing. It handles yes/no, multiple choice, and scores from 2 to 255 levels, with inputs up to 32k tokens. If you have an NVIDIA card I'd go with the INT8 version, since it's the fastest. On the same GPU it runs about 2 to 2.7x faster than the Q8_0 GGUF on llama.cpp. GGUF files (BF16, Q8_0, IQ4_XS) are there for AMD, Apple and CPU. It runs on 8 GB cards and up, and it's Apache-2.0. HF: https://huggingface.co/sky7350/Mica-v0.5-4B GitHub: https://github.com/akivet/Mica Happy to answer questions, and I'd really like to hear where it breaks. submitted by /u/Top-Evidence174 [link] [comments] |