Soup

Headlines · 1

1. 2d
   Soup v0.76.0 ships layer streaming to fine-tune 8B models on 4 GB laptop GPUs

   Soup is a command-line tool that fine-tunes LLMs from a single YAML config; its v0.76.0 release now refuses at load time settings that were previously accepted but never actually applied, and includes breaking changes such as no longer holding out validation rows by default for grpo, ppo and online_dpo. Its opt-in layer streaming feature (stream_layers: true, still BETA) keeps the frozen base weights out of VRAM and feeds them to the GPU one decoder layer at a time: the project records Llama-3.1-8B-Instruct + NF4 on a 4 GB RTX 3050 Laptop at 3.32 GB peak and 119.6 tok/s (v0.72.2, before the correctness repair), while a later post-repair run reads 208.6 tok/s in a 2.40 GB peak — the project notes the two are not directly comparable because clock speeds differ. The release also evaluates validation loss each epoch or every N steps, and reads the streaming disk tier on a background thread via direct I/O with optional striping across multiple NVMe drives.

   GitHub Trending · Python · 🔥 6

Experience and discussion from the community

Share my Soup experienceAsk about Soup