Qwen2.5-1.5B

Headlines · 2

1. 23h
   Tsinghua open-sources Puro-2B: 2B pretraining on RTX 5090s for about $4,400

   A Tsinghua University team led by professors Chen Wenguang and Lyu Kaifeng released Puro-2B, a fully open pretraining recipe that publishes the training data, framework, configuration and model weights. The roughly 2B-parameter model was trained from scratch on a cluster of consumer RTX 5090 GPUs using up to 1.4 trillion tokens; on a 15-task evaluation, a version with about $4,400 of pretraining compute beat Qwen2-1.5B on average, while the $6,900 best version came close to Qwen2.5-1.5B on average. The team also fit a Puro Cost Scaling Law to estimate achievable quality at a given budget, combining hardware choice, FP8 training, the MuonH optimizer, Curriculum Model Averaging and data-slice evaluation.

   36氪 人工智能 · 🔥 9
2. 2d
   Soup v0.76.0 ships layer streaming to fine-tune 8B models on 4 GB laptop GPUs

   Soup is a command-line tool that fine-tunes LLMs from a single YAML config; its v0.76.0 release now refuses at load time settings that were previously accepted but never actually applied, and includes breaking changes such as no longer holding out validation rows by default for grpo, ppo and online_dpo. Its opt-in layer streaming feature (stream_layers: true, still BETA) keeps the frozen base weights out of VRAM and feeds them to the GPU one decoder layer at a time: the project records Llama-3.1-8B-Instruct + NF4 on a 4 GB RTX 3050 Laptop at 3.32 GB peak and 119.6 tok/s (v0.72.2, before the correctness repair), while a later post-repair run reads 208.6 tok/s in a 2.40 GB peak — the project notes the two are not directly comparable because clock speeds differ. The release also evaluates validation loss each epoch or every N steps, and reads the streaming disk tier on a background thread via direct I/O with optional striping across multiple NVMe drives.

   GitHub Trending · Python · 🔥 6

Experience and discussion from the community

Share my Qwen2.5-1.5B experienceAsk about Qwen2.5-1.5B