TensorFold: A native language-model inference engine for serving checkpoints on Apple Silicon and NVIDIA GPUs.
What problem it solves
Serving language models requires reading checkpoints, tokenizing requests, and running accelerated kernels while managing a local model server. TensorFold addresses these requirements with a native binary that can run on Apple Silicon and NVIDIA GPUs without requiring Python or MLX for serving.
Key features
Native Metal and CUDA inference — The Zig-based engine executes model inference through native Metal or CUDA kernels. It supports serving language-model checkpoints on Apple Silicon and NVIDIA GPUs.
Source: README
OpenAI-compatible local serving — A local server exposes chat completions, completions, Responses, and Anthropic Messages endpoints. The served model can also be listed through the models endpoint.
Source: README
Flexible model acquisition — Users can pull a pinned checkpoint, serve a complete local checkpoint directory, or use an existing Hugging Face cache. The documented model catalog includes multiple architectures, quantizations, and hardware configurations.
Source: README
Local web and desktop interfaces — TensorFold Web provides a local interface for selecting, running, and chatting with models. The desktop workspace bundles the native engine with model downloads, recommendations, generation metrics, coding workers, and document editing.
Source: README
Experimental weight learning — Living Weights can write accepted facts into a running model's checkpoint files and provides a web view of the learned facts and affected layers. This feature is explicitly experimental and supports rollback.
Source: README
Quick start
Copied verbatim from the project's README.
brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinkingcurl -fsS http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'Who it may suit
This project may suit developers and teams that want to run local language models on Apple Silicon or NVIDIA GPUs. It may also suit contributors interested in native inference engines, model-serving APIs, or experimental model adaptation.
Topics to discuss
Which model and quantization combinations should receive additional native serving recipes?
How should the project balance Metal, CUDA, and future hardware support in its release qualification process?
What safeguards and evaluation methods should guide the development of Living Weights?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/ashhart/TensorFold#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…