Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

TensorFold

TensorFold: A native language-model inference engine for serving checkpoints on Apple Silicon and NVIDIA GPUs.

What problem it solves

Serving language models requires reading checkpoints, tokenizing requests, and running accelerated kernels while managing a local model server. TensorFold addresses these requirements with a native binary that can run on Apple Silicon and NVIDIA GPUs without requiring Python or MLX for serving.

Key features

  • Native Metal and CUDA inference — The Zig-based engine executes model inference through native Metal or CUDA kernels. It supports serving language-model checkpoints on Apple Silicon and NVIDIA GPUs.

    Source: README

  • OpenAI-compatible local serving — A local server exposes chat completions, completions, Responses, and Anthropic Messages endpoints. The served model can also be listed through the models endpoint.

    Source: README

  • Flexible model acquisition — Users can pull a pinned checkpoint, serve a complete local checkpoint directory, or use an existing Hugging Face cache. The documented model catalog includes multiple architectures, quantizations, and hardware configurations.

    Source: README

  • Local web and desktop interfaces — TensorFold Web provides a local interface for selecting, running, and chatting with models. The desktop workspace bundles the native engine with model downloads, recommendations, generation metrics, coding workers, and document editing.

    Source: README

  • Experimental weight learning — Living Weights can write accepted facts into a running model's checkpoint files and provides a web view of the learned facts and affected layers. This feature is explicitly experimental and supports rollback.

    Source: README

Quick start

Copied verbatim from the project's README.

brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'

Who it may suit

This project may suit developers and teams that want to run local language models on Apple Silicon or NVIDIA GPUs. It may also suit contributors interested in native inference engines, model-serving APIs, or experimental model adaptation.

Topics to discuss

  • Which model and quantization combinations should receive additional native serving recipes?

  • How should the project balance Metal, CUDA, and future hardware support in its release qualification process?

  • What safeguards and evaluation methods should guide the development of Living Weights?


Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/ashhart/TensorFold#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/ashhart/TensorFold

Comments

…