Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Together AI launches agent-driven Together Compiler

AI summary

Together AI introduced Together Compiler, an agent-driven framework for continual AI inference optimization that replaces hand-written megakernels with composable kernels, CUDA Graphs, PDL, and parallel agent workflows. The company said its internal experiments recovered 98% of megakernel performance with default Torch Inductor kernels plus CUDA Graphs and PDL, while later configurations achieved up to 1.7 times the throughput of its megakernel baseline. Together AI is offering early access and says the system produced 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash.

In-depth insight

Together Compiler presents an alternative to narrowly optimized model-specific megakernels, with Together AI reporting broader workload coverage and higher throughput in its internal tests.

Introduction

AI inference is becoming increasingly heterogeneous. New model architectures are released every week, inference traffic shifts across different batch sizes and sequence lengths, and hardware platforms continue to proliferate. Achieving peak performance across all these regimes demands optimization frameworks that can keep up.

Last year, we helped release some of the first AI megakernels, where we executed an entire model forward pass within a single, highly fused GPU kernel. They reached record breaking speeds, but it was difficult to put them in production. After months of effort, we had a grand total of one model, one batch size, and one customer.

This year, we examined what we could simplify and discovered something surprising. With just CUDA Graphs and PDL on top of default Torch Inductor kernels, we could recover 98% of megakernel performance without writing a single kernel. And with “multi-kernels” and agent loops, we could further beat our old megakernels by up to 1.7x, achieving over 1100 TPS for Llama 1B on H100.

For the past two years, we've been investigating what a next-generation system for inference optimization should look like. After building a series of GPU programming and optimization systems, we arrived at Together Compiler, our framework for continual AI inference optimization. In this blog, we'll revisit the assumptions behind today's compilers, share the experiments that challenged our thinking, and explain the lessons that shaped Together Compiler. Our journey spans three parts:

#Together AI#Together Compiler#Torch Inductor#CUDA Graphs#NVIDIA#Llama 1B#MiniMax M3.1-Flash#DeepSeek v4.1-Flash

Original · 29 min readRead the original
Group chat

No comments yet. Start the conversation.