Together AI发布智能体驱动推理编译器
Together AI推出由智能体驱动的持续推理优化框架Together Compiler,以可组合内核、CUDA Graphs、PDL和并行智能体工作流取代手写超级内核。公司称,默认Torch Inductor内核结合CUDA Graphs和PDL可恢复超级内核98%的性能,后续配置吞吐量最高达到其基线的1.7倍。Together AI已开放早期体验,并称系统分别以1,111 TPS和1,205 TPS运行MiniMax M3.1-Flash与DeepSeek v4.1-Flash。
Together Compiler以持续优化的方式替代面向单一模型的超级内核,Together AI称其在内部测试中覆盖更多工作负载并实现更高吞吐量。
Introduction
AI inference is becoming increasingly heterogeneous. New model architectures are released every week, inference traffic shifts across different batch sizes and sequence lengths, and hardware platforms continue to proliferate. Achieving peak performance across all these regimes demands optimization frameworks that can keep up.
Last year, we helped release some of the first AI megakernels, where we executed an entire model forward pass within a single, highly fused GPU kernel. They reached record breaking speeds, but it was difficult to put them in production. After months of effort, we had a grand total of one model, one batch size, and one customer.
This year, we examined what we could simplify and discovered something surprising. With just CUDA Graphs and PDL on top of default Torch Inductor kernels, we could recover 98% of megakernel performance without writing a single kernel. And with “multi-kernels” and agent loops, we could further beat our old megakernels by up to 1.7x, achieving over 1100 TPS for Llama 1B on H100.
For the past two years, we've been investigating what a next-generation system for inference optimization should look like. After building a series of GPU programming and optimization systems, we arrived at Together Compiler, our framework for continual AI inference optimization. In this blog, we'll revisit the assumptions behind today's compilers, share the experiments that challenged our thinking, and explain the lessons that shaped Together Compiler. Our journey spans three parts:
#Together AI#Together Compiler#Torch Inductor#CUDA Graphs#NVIDIA#Llama 1B#MiniMax M3.1-Flash#DeepSeek v4.1-Flash