mlx-decision Adds LoRA Image Fine-Tuning for Doom on Mac
AI summary
Developer armbues reports adding image-based LoRA fine-tuning to the open-source mlx-decision tool, enabling its 2B decision model to process Doom frames on Apple Silicon. The author says training on about 3,000 ViZDoom frames took 43 minutes on an M5 Pro, while held-out-frame metrics improved and the model averaged 23.2 kills across 20 episodes. The MIT-licensed project targets macOS 14 or later, Python 3.12, and Apple Silicon.
In-depth insight
The release demonstrates a concrete open-source workflow for training image-based decision models locally on Macs, although its Doom results come from a limited arena setting and are self-reported.
| You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon). How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions: - Is a monster right at the crosshair, so that a shot now would hit it? (yes/no) - Where is the nearest monster: left, centre, right, none? - Is a monster dangerously close? (yes/no) A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision. The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames: - "a shot would hit": precision 0.79 → 0.92 - where the nearest monster is: 92.7% → 96.8% - dangerously close: 93.9% → 98.3% In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level. Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision. Repo: https://github.com/armbues/mlx-decision I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next. submitted by /u/armbues [link] [comments] |