RunAnywhere开源GLM-5.3-Flash推理引擎
RunAnywhere开发者称，公司已开源一套推理引擎，可在一台AWS Trainium2 trn2.48xlarge上运行拥有320B参数和100万上下文长度的GLM-5.3-Flash。项目采用融合式NKI分层巨核，将作者所用布局下的批量8解码步骤从32.2毫秒降至24.2毫秒，并提供详细性能分析、配置文件和复现说明。RunAnywhere还在自行运行的SemiAnalysis AgentX基准中报告最高每分钟28.9次智能体请求，但该帖没有提供AWS原生版本或GPU对照数据。
RunAnywhere开源的引擎与性能分析为在AWS Trainium2上部署大型MoE模型提供了技术参考，不过其自报结果仍缺少AWS原生版本和GPU部署的对照验证。
We wrote an inference engine for AWS Trainium2 that serves GLM-5.3-Flash (320B MoE, 1M-token context) on one trn2.48xlarge, and open-sourced it. Most of the work went into decode, and the part I found most interesting is how little of a decode step was the model's actual math.
In an early build, in one KDA (linear attention) layer at batch 1, the recurrence was 2% of the attention half. The rest was the mHC pre-projection (36%), XLA glue (33%), transposes (10%), the stream mix (10%) and other small ops (8%). A batch-8 decode step took 24.2 ms against a 6.1 ms memory floor, and the profile showed why: long chains of small vector-engine instructions at about 153 ns of fixed cost each, a write back to HBM at every kernel boundary, and 125 cross-chip collectives per step.
So we rewrote each layer half as one NKI kernel (each whole layer, for the sparse-attention layers). The kernels follow three rules:
- Keep the working set on chip. Each physical NeuronCore has 28 MiB of SBUF, enough for a layer half's activations, its recurrent state and the weights in flight, so only the layer's inputs, outputs and streamed weights cross HBM.
- Use both physical cores of a logical core. One program streams 340–400 GB/s of weights, the pair 672–708 GB/s, so the weight stream has to be split between them.
- Move a collective into the kernel when it blocks the next one. Run by XLA, an all-reduce makes the next kernel wait for all of it. Inside the kernel, independent work can overlap it.
#RunAnywhere#GLM-5.3-Flash#AWS Trainium2#trn2.48xlarge#NKI#XLA#SemiAnalysis AgentX#vLLM Neuron#NxD Inference#AMD#Hazy Research#Mirage