Together AI launches TKCC for continual inference optimization
Together AI introduced Together Compiler (TKCC), a continual optimization system that uses high-throughput agents to generate and test inference-engine optimizations for changing workloads and hardware. The company reports that TKCC delivered up to 2.6x higher throughput across several open-source models in less than 24 hours, using persistent engines and parallel experimentation. These performance and benchmark figures are vendor-provided and were not independently corroborated by the report.
TKCC targets a substantial inference-engine bottleneck by allowing agents to repeatedly test optimizations without restarting engines, potentially improving how quickly inference systems adapt to different models, hardware and traffic patterns.
AI inference workloads are dynamic. There are thousands of regimes spanning new models, hardware platforms, customers, use cases, and traffic shapes. It’s no longer scalable to optimize each of these regimes by hand.
- Architectures are highly variable: it’s been ~11 years since the Transformer. We might have thought there would be more consolidation, but none of the top-10 open-source LLMs on OpenRouter use the same architecture!
- Workloads are highly variable: Together serves many trillions of tokens per day across thousands of customers, each with a different traffic shape (batch size, sequence length, MoE expert imbalance, attention sparsity, session duration, domain).
- Hardware is highly variable: H100s, B200s, B300s, NVL72, VR, Groq, and more each require distinct strategies.
Agents are a great fit for this problem, in theory. They can adapt to new workloads and search over the space of possible optimizations without relying on fixed rules or heuristics that plague human experts and traditional machine learning compilers. However, today’s agentic systems are bottlenecked by the speed of experimentation.
#Together AI#Together Compiler (TKCC)#Kimi K3#GLM 5.3#MiniMax M3.1-Flash#DeepSeek v4.1-Flash#Llama 1B#SemiAnalysis InferenceX#NVIDIA H100#NVIDIA B200#NVIDIA B300#Groq