TensorFold: A native language-model inference engine for serving checkpoints on Apple Silicon and NVIDIA GPUs.
What problem it solves Serving language models requires reading checkpoints, tokenizing requests, and running accelerated kernels while managing a local model server. TensorFold addresses these requirements with a native binary that can run on Apple Silicon and NVIDIA GPUs without requiring Python or MLX for serving. Key features Native Metal and CUDA inference — The Zig-based engine executes mode…
github.comhttps://github.com/ashhart/TensorFold