PyTorch
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 9
- PyTorch's Zephon brings elastically deterministic data loading to TorchTitan
A PyTorch blog post introduces Zephon, a data loader for TorchTitan that aims to produce identical global batches even when the number of GPUs changes. In experiments training a 1B-parameter model from scratch on 20B tokens across 8 to 64 GPUs, a standard loader let DCLM Core v1 scores vary by up to 0.82 points and FineWeb scores by 0.64 points — roughly four times the thresholds those benchmarks' authors used — while Zephon's spread was 0.05 and 0.011 points respectively. Zephon tokenizes, packs, shuffles and mixes data on the fly inside the training job, using lanes, single-threaded accumulators and periodic flush sentinels to reproduce state, with checkpoints of a few MiB that can be resumed on a different number of data-parallel ranks; the authors say it keeps pace with Google Grain on a vision-language pipeline.
PyTorch Blog · 🔥 6 - ExecuTorch adds Arm Cortex-M CPU backend for PyTorch models
A PyTorch blog post announces a new ExecuTorch Cortex-M backend that runs PyTorch models directly on Arm Cortex-M microcontrollers, using CMSIS-NN-accelerated operators and covering the Scalar, DSP and MVE/Helium capability tiers from one workflow. On a single Cortex-M55 core at 400 MHz on the Alif Ensemble E8, the team measured four of the five MLPerf Tiny models running in 21.5 ms or less and under 90 KiB of RAM, with the anomaly-detection autoencoder at 5.2 KiB; these are the team's own measurements rather than an official MLPerf submission. A µYOLO real-time person detector demo on the Cortex-M85-based Renesas EK-RA8D1 runs at roughly 24 ms per frame and 12.7 FPS, within 640 KiB of flash and 254 KiB of peak RAM.
PyTorch Blog · 🔥 3 - China's AI compute race shifts to systems engineering, Jiliu says
InfoQ China published an interview with Chen Wei, an early member of Jiliu Technology, on how the question for domestic AI compute has shifted from simply getting accelerator cards to running them at scale. Once clusters grow from dozens to thousands of cards, the bottlenecks move to operator coverage, collective communication, scheduling and fault localization, and matching interfaces does not mean production-grade performance. Jiliu says its VenusCL communication component can add roughly 2%–6% more MFU even on thousand-card clusters customers had already tuned, and its VenusNOC Megatrace feature located a single faulty GPU in a roughly 3,800-card training cluster. The piece also covers heterogeneous co-working across chips from multiple vendors and generations, plus the trade-offs of ecosystem standardization, with Jiliu estimating that a 2%–6% gain could correspond to over 100 million yuan of existing compute value in one 10,000-card project.
InfoQ 中文 AI&大模型 · 🔥 3 - Soup v0.76.0 ships layer streaming to fine-tune 8B models on 4 GB laptop GPUs
Soup is a command-line tool that fine-tunes LLMs from a single YAML config; its v0.76.0 release now refuses at load time settings that were previously accepted but never actually applied, and includes breaking changes such as no longer holding out validation rows by default for grpo, ppo and online_dpo. Its opt-in layer streaming feature (stream_layers: true, still BETA) keeps the frozen base weights out of VRAM and feeds them to the GPU one decoder layer at a time: the project records Llama-3.1-8B-Instruct + NF4 on a 4 GB RTX 3050 Laptop at 3.32 GB peak and 119.6 tok/s (v0.72.2, before the correctness repair), while a later post-repair run reads 208.6 tok/s in a 2.40 GB peak — the project notes the two are not directly comparable because clock speeds differ. The release also evaluates validation loss each epoch or every N steps, and reads the streaming disk tier on a background thread via direct I/O with optional striping across multiple NVMe drives.
GitHub Trending · Python · 🔥 0 - Databricks cuts Qwen 3.8 27B fine-tuning time 6x on AI Runtime
In a blog post, Databricks describes how systematic performance work on its AI Runtime cut fine-tuning time for Qwen 3.8 27B on a fixed dataset from 7 hours 44 minutes to 1 hour 17 minutes — about 6x — from a baseline throughput of 9,396 tok/s. The team notes that the initial run already showed 98% average GPU utilization, but that metric only measures whether the GPU is doing anything; the real bottleneck was hundreds of thousands of small ops that the CPU could not enqueue fast enough. Installing fused kernels such as flash-linear-attention and causal-conv1d brought training down to 2 hours 5 minutes (3.7x), and switching to 1D packed tensors with correct position_ids to cut padding waste delivered the final 6x gain, with example code published in an AI Runtime cookbook.
Databricks Blog · 🔥 0 - torch-spyre moves its compiler onto PyTorch Inductor
A PyTorch blog post explains that torch-spyre, the PyTorch integration for IBM's Spyre accelerator, has moved its torch.compile backend from sendnn — a compilation stack built entirely from scratch — to backend=inductor, built by extending PyTorch's own open-source Inductor. Most of Inductor carried over untouched, with the work concentrated in three hardware differences: Spyre moves and computes on 128-byte sticks rather than individual elements; each of its 32 cores has only a 2MB local scratchpad with no hardware-managed caching, so every data movement must be an explicit compiler-emitted instruction; and the cores run as a static dataflow machine, so work division must be decided at compile time. The authors say the resulting compiler is roughly 7x smaller than what it replaced, and report that changing input and output tensor tiling cut batched matmul kernel time by 29–69% across batch sizes and shapes when tensors were already resident in device memory. The Spyre passes currently hook Inductor by monkey-patching GraphLowering.codegen(), and the team is working with upstream PyTorch to formalize those extension points.
PyTorch Blog · 🔥 0 - Axera launches AX8860 AI chip, claims 25% faster LLM inference than DGX Spark
Axera unveiled its AX8860 series AI chip at the 2026 China International Industry Fair, positioning it as a compute base for running large models at the edge and on-device. The chip pairs a 16-core Arm Cortex-A55 CPU with a self-developed NPU rated at 256 TOPS (native FP4, no sparsity), supports up to 128GB of LPDDR5X memory with 273 GB/s of bandwidth over a 256-bit bus, is built on a 6nm process and offers PCIe 5.0 connectivity. Form factors include a 40W module, a single-slot PCIe card and a single-slot four-chip design that Axera claims reaches 1024 TOPS at roughly 80% scaling efficiency, plus planned Mini PCs and Pocket PCs; software support covers vLLM, SGLang and PyTorch with an OpenAI-compatible API, alongside the newly announced Yuanxi A256S module. Axera and Sipeed claim the chip is 25% faster on prefill and 30% faster on decode than NVIDIA's DGX Spark on Qwen 27B, at close to half the price (DGX Spark lists at $4,999 for 64GB and $6,950 for 128GB) — claims that have not been independently verified.
Wccftech · 🔥 0 - Hugging Face Transformers tops GitHub Trending
Hugging Face's Transformers library appeared on GitHub Trending's daily list. The project positions itself as the model-definition framework for state-of-the-art text, vision, audio, video and multimodal models, for both inference and training, and its documentation says there are over 1M Transformers model checkpoints on the Hugging Face Hub. It says the library interoperates with training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP and PyTorch-Lightning, and with inference engines and modeling libraries including vLLM, SGLang, TGI, llama.cpp and mlx, and requires Python 3.10+ and PyTorch 2.6+.
GitHub Trending（每日） · 🔥 0 - torch-spyre brings IBM Spyre in as a native PyTorch device
IBM's torch-spyre team published a PyTorch blog post describing how it made Spyre, IBM's dataflow AI accelerator for inference, a native PyTorch device. The integration uses PrivateUse1 registration to give Spyre a device identity named "spyre", then maps PyTorch's allocator, stream, event and Inductor compiler abstractions onto Spyre runtime constraints such as ordered queues of typed operations, separate compute and data-movement pipelines, and ahead-of-time compiled programs with fixed layouts. IBM says this gives one path for eager and compiled execution, keeps tensors resident on device="spyre", and cuts launch overhead; no performance or benchmark figures are given, and the source text is truncated near the end.
PyTorch Blog · 🔥 0
Experience and discussion from the community
Share my PyTorch experienceAsk about PyTorch
Nobody has shared their experience with PyTorch yet.