Developer launches Tinkerbox to run decision models on Apple's Neural Engine via Shortcuts
A developer posting on r/LocalLLaMA as /u/WeGoToMars7 released Tinkerbox, an app that runs decision models on Apple's Neural Engine (ANE) and exposes Shortcuts actions for automating system events such as incoming email and iMessages, folder and file changes, time of day and alarms. The author says LiquidAI d1-3B runs on the ANE with roughly 40 ms latency for warm short-context decisions under 256 tokens, measured on an M5 Pro, and says the model sits on the Pareto frontier of Decision Index 0.3.2; both are self-reported or cited by the developer, who notes he has no other Macs to test on. The d1-3B implementation uses a tree structure to evaluate multiple decisions in a single pass, source code and AOT-compiled models are published on GitHub and Hugging Face, and the author says he is working on integrating vLLM d3 models based on Qwen3.5 with 0.8B, 2B and 4B variants planned.
The developer argues decision models suit the ANE because they skip token-by-token autoregressive decoding, which he says the engine handles poorly due to relatively low memory bandwidth and static buffer limits, offering a route to low-latency decision models on that hardware.
| LiquidAI d1-3B is currently on the Pareto frontier of the Decision Index 0.3.2 https://huggingface.co/spaces/multimodalart/jev-decision-index, and you can run it on the ANE with around 40 ms latency on warm short-context (<256 tokens) decisions, as measured on M5 Pro. With Shortcuts integration, you can run efficient decisions on events such as incoming email and iMessages, folder and file modifications, time of day and alarms, and other triggers without writing a single line of code. Decision models are a good fit for ANE because they don't use token-by-token autoregressive decoding, which it struggles with due to relatively low memory bandwidth and static buffer limitations. Tinkerbox's implementation of d1-3B uses a tree structure to evaluate multiple decisions in a single pass, further reducing latency. Source code: https://github.com/WeGoToMars/tinkerbox AOT-compiled models: https://huggingface.co/wegotomars/d1-3B-CoreAI-ANE Download the app: https://github.com/WeGoToMars/tinkerbox/releases I don't have access to any other Macs, so please report your results and any issues you encounter. I'm also taking model and feature suggestions! Embeddings, TTS, STT, and image segmentation models are on the roadmap. This isn't a one-off vibecoded repo, I want to continue supporting it and make it a real framework for ANE-optimized inference. I'm currently working on integrating vLLM d3 models (based on Qwen3.5), which would also allow you to use most other Qwen3.5 fine-tunes with minimal modification. 0.8B (lite), 2B (nano), and 4B (mini) should be available soon! submitted by /u/WeGoToMars7 [link] [comments] |
#Tinkerbox#Apple Neural Engine#LiquidAI#LiquidAI d1-3B#Apple Shortcuts#Qwen3.5#Apple