FreeToken

Headlines · 2

1. Nov 3
   FreeToken targets local MoE inference on consumer hardware

   A technical report describes FreeToken as an edge-native MoE serving engine that combines GPU compute, CPU compute and host memory in a unified inference pool. The report says the project can run 290B-parameter open-weight models at interactive speeds on gaming PCs and supports features including semantic checkpoints, expert caching and dynamic VRAM allocation. It also reports support for Windows and Linux, Anthropic- and OpenAI-compatible APIs, several MoE models and NVIDIA RTX 30-, 40- and 50-series GPUs.

   掘金 · 人工智能 · 🔥 0
2. Nov 2
   FreeToken targets local MoE inference on consumer hardware

   FreeToken is an Apache 2.0-licensed, edge-native Mixture-of-Experts serving engine designed to run large open-weight models on personal and consumer hardware. It combines GPUs, CPUs, host memory and interconnects in an elastic platform with bandwidth-adaptive CPU–GPU co-execution, expert caching, dynamic VRAM reallocation and Anthropic- and OpenAI-compatible APIs. The project supports consumer laptops, gaming desktops and workstation GPUs. Its publisher claims it can run 290B-plus frontier MoE models on a gaming PC at interactive speeds, but the supplied report provides no independent benchmark or verified throughput and latency figures.

   GitHub Trending（每日） · 🔥 0

Experience and discussion from the community

Share my FreeToken experienceAsk about FreeToken

All

FreeToken•4d Pinned

FreeToken: An edge-native MoE serving engine for running frontier-scale open-weight models on personal and consumer hardware.

What problem it solves Running frontier-scale open-weight models locally requires an inference platform that can make efficient use of the heterogeneous hardware available on a personal device. The project addresses this by treating GPUs, CPUs, host memory, and interconnects as a unified, elastic inference platform. Key features Edge-native MoE runtime — The runtime provides MoE serving with bandw…

github.comhttps://github.com/FlashML-org/FreeToken