Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

MaxText v0.2.4 adds long-context training for Cloud TPU

AI summary

Google announced that MaxText v0.2.4 supports long-context training on Cloud TPU through four context-parallel strategies: All-gather, A2A (DeepSpeed Ulysses), P2P (Ring Attention), and A2A + P2P (USP). The framework can extend context to 14 million tokens across TPU 5p and TPU 7, and Google reports that Llama 3.1 8B reached 70% Model FLOPs Utilization on TPU 5p for sequences up to 4 million tokens. The release is available on PyPI, Docker images, and GitHub main.

In-depth insight

The release gives AI developers an available open-source JAX framework for training models with million-token sequences on Cloud TPU, addressing the memory and communication bottlenecks that emerge during long-context training.

Announcing Long-Context Training on TPUs with MaxText Context Parallelism - Google Developers Blog

Announcing Long-Context Training on TPUs with MaxText Context Parallelism

NOV. 9, 2026

Michael Tran Software Engineering Intern Google Cloud

Nuojin Cheng Software Engineer Google Cloud

Deepak Patil Group Product Manager

#MaxText#Cloud TPU#TPU 5p#TPU 7#Llama 3.1 8B#Google Cloud

Original · 10 min readRead the original
Group chat

No comments yet. Start the conversation.