Gemini Decision 1的800个场景评测
一名作者对谷歌Gemini Decision 1的评测覆盖了24个类别和800个会话。该模型不生成文本,而是返回布尔值、分数或类别;在狭窄检查中表现较强,提示注入检测准确率达到99.7%,判断哪些内容值得记忆的准确率为88%。但它在宽泛、主观的判断上可靠性较低:预测用户反对反应的准确率仅为46%,而在切分代码文件以供检索时,确定性代码表现更好。作者测得单题约需180毫秒,一次处理100个问题约需250毫秒。测试使用了作者自己的会话和一个公开数据集,并非穷尽性评测。
报告结果支持将决策模型用作狭窄分类或评分检查的快速、低成本过滤层,同时将宽泛的主观判断留给完整语言模型,并在确定性代码效果更好时优先使用代码。由于评测仅覆盖一名作者的会话和一个公开数据集,这些结果并非穷尽性基准。
With all the excitement around large language models over the last few years, it's easy to forget that machine learning has been around a lot longer than chat models, and that other models are built for very different jobs.
Today, most AI systems leave a lot of efficiency on the table by calling a full LLM just to get a simple true or false response or a basic classification.
Google recently released Gemini Decision 1, also known as a 'System 1' model. Instead of generating text token by token like a chat model, it reads your input in one pass and returns one of three structured statistical responses:
- True or false
- A score
- A category (classification)
Why decision models are fast and cheap
Because there are no generated text tokens, both the speed and the pricing look very different from a standard LLM.