开发者将 Forth 解释器嵌入 Nemotron 权重
Reddit 作者 /u/Mr_Finious 称,在未改动的 NVIDIA Nemotron 3.5 Lightning 模型旁加入了一个未训练的 27 层 Forth 解释器网络。解释器会在模型生成时逐步执行每个 Forth 词,测试中的 765,734 个机器 token 均与 Python 参考实现一致。作者报告,LoRA 微调后其在 600 道留出测试题上的得分从 79.1% 提升至 92.1%,但该模型处理长整数时仍落后于 Python,且不适用于生产环境。
这项作者自述的实验展示了将可运行机器嵌入模型权重的具体路径,同时暴露出其对微调的依赖,以及每一步机器操作都需要完整前向传播的限制。
I learned RPN on an HP calculator and later spent more time with Forth than was strictly useful. When I saw Qwenputer compile a machine into a language model's weights, I wanted to know two things. Would a stack language fit that idea well? And could a model with only 3B active parameters drive a machine like that?
The setup: NVIDIA Nemotron 3.5 Lightning (30B, 3B active) is left unchanged. Next to it, in the same model, is a second network of 27 layers. A Python script calculated its weights; none of them are trained. It is a Forth interpreter. The model writes a Forth block as ordinary text. The interpreter runs each word while the model is still writing, one step per forward pass. When the block closes, it writes the output as the model's next tokens. No tool call occurs, and generation does not stop.
RPN suits this well. A Forth word ends at a space, so the machine can run it as soon as the next word starts, without parsing the whole program. Operands come before the operator, so when the model writes +, both numbers are already on the stack. That is the order a left-to-right model writes in anyway. The stack itself lives in Mamba-2 state. One entry is one row, 20 digits wide, and popping a number means adding its exact negative.
#NVIDIA#Nemotron 3.5 Lightning#Forth#Qwenputer#Qwen#DeepMind#LoRA#vLLM#NeMo AutoModel