Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

FreeToken

FreeToken• Pinned

FreeToken: An edge-native MoE serving engine for running frontier-scale open-weight models on personal and consumer hardware.

What problem it solves

Running frontier-scale open-weight models locally requires an inference platform that can make efficient use of the heterogeneous hardware available on a personal device. The project addresses this by treating GPUs, CPUs, host memory, and interconnects as a unified, elastic inference platform.

Key features

  • Edge-native MoE runtime — The runtime provides MoE serving with bandwidth-adaptive CPU–GPU co-execution, double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.

    Source: README

  • Semantic-aware caching — Semantic anchor checkpoints preserve recurrent state and KV caches across context edits, allowing changes such as tool calls and thinking blocks to avoid redundant recomputation.

    Source: README

  • Elastic memory management — VRAM can be reallocated at runtime between expert caches and KV memory without restarting the engine or reloading weights.

    Source: README

  • Broad model and API support — The engine supports frontier open-weight MoE models across parameter scales and quantization formats, with Anthropic- and OpenAI-compatible APIs for integration with coding and tool-calling agents.

    Source: README

  • Consumer hardware support — The serving engine scales across consumer laptops, gaming desktops, and workstation GPUs, including NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.

    Source: README

Quick start

Copied verbatim from the project's README.

uv pip install "freetoken[accel]"
git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

Who it may suit

Inference: FreeToken may suit developers, researchers, and teams that want to run frontier-scale open-weight MoE models on consumer hardware.

Topics to discuss

  • Which local model and hardware combinations should receive priority for testing and optimization?

  • How should semantic-aware caching be evaluated for agentic context changes?

  • What compatibility requirements matter most for OpenAI- and Anthropic-compatible integrations?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/FlashML-org/FreeToken#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/FlashML-org/FreeToken

Comments

…