Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

mlx-decision Adds LoRA Image Fine-Tuning for Doom on Mac

AI summary

Developer armbues reports adding image-based LoRA fine-tuning to the open-source mlx-decision tool, enabling its 2B decision model to process Doom frames on Apple Silicon. The author says training on about 3,000 ViZDoom frames took 43 minutes on an M5 Pro, while held-out-frame metrics improved and the model averaged 23.2 kills across 20 episodes. The MIT-licensed project targets macOS 14 or later, Python 3.12, and Apple Silicon.

In-depth insight

The release demonstrates a concrete open-source workflow for training image-based decision models locally on Macs, although its Doom results come from a limited arena setting and are self-reported.

You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon). How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions: - Is a monster right at the crosshair, so that a shot now would hit it? (yes/no) - Where is the nearest monster: left, centre, right, none? - Is a monster dangerously close? (yes/no) A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision. The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames: - "a shot would hit": precision 0.79 → 0.92 - where the nearest monster is: 92.7% → 96.8% - dangerously close: 93.9% → 98.3% In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level. Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision. Repo: https://github.com/armbues/mlx-decision I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next. submitted by /u/armbues [link] [comments]

#mlx-decision#d1-2B#MLX#ViZDoom#Doom#Freedoom#Apple Silicon

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.