Taalas将Llama烧进芯片，号称17k tok/s
r/LocalLLaMA 用户 /u/IAM_274 发帖介绍初创公司 Taalas 的聊天机器人 ChatJimmy：据帖主称，它通过把模型“烧”进硅片、而非把模型加载到芯片上的方式，让一个 8B Llama 模型在单颗芯片上跑到每秒 1.7 万 token。帖主给出的参照是同一模型在 RTX 5070 上平均约 80 tok/s，即号称约 213 倍差距，并强调该项目只是实验性概念验证，尚不是实际产品。帖子还提到一项取舍：模型一旦固定在芯片上便无法更换，升级模型意味着更换整块芯片；帖主另附一篇 Medium 文章，讨论制造成本与功耗。上述性能数字均来自帖主和其所引用的项目材料，没有独立测量佐证。
该帖让一种思路受到关注——把模型直接印进芯片，而不是作为软件运行在通用芯片上；帖主自己也说这还不是产品，因此所宣称的速度仍未经核实。
I found this project that I never saw talked about enough which is absolutley baffling to me. The title is not clickbait, and I can't blame you if you thought it is, because I was in your shoes once.
Meet ChatJimmy. It's a chatbot made by Taalas. And it's not your average chatbot. This is the closest LLMs have come to a physical brain than mere software that runs on your computer.
Though not an increment to intelligence (since it's an 8B model) and that's not the point of the project, this single chip makes the model run faster than if it were run on a multi-GPU datacenter. At a blazing 17k tok/s. For reference, the average speed of running the same model on an RTX 5070 is 80 tok/s. This chip is around 213x faster. If someone told me this was possible a year ago, I'd ask for what they're smoking.
The magic is actually pretty simple behind the scenes. The way Taalas was able to achieve such absurd speeds is by burning the model into silicon. Yep, you heard me right. The LLM is no longer loaded on a chip. The LLM is the chip. All computing power is purely gone straight the model's internal matrix math. The concept on it's own sounds cool as fuck, and seeing it work in practice is even cooler.