Reddit discussion argues action labels are the missing layer in game world-model data
A Reddit discussion argues that action labels, rather than video volume, are the main bottleneck in making game data trainable for world models. The poster cites VPT and Muse as examples of projects that retained gameplay inputs, while describing an unnamed deterministic multiplayer strategy game that records actors, messages, and world state; these claims are not independently verified here. The post also asks whether world-model training should use high-level strategic actions or low-level controls, and whether language-linked multi-agent data is useful beyond evaluation.
The discussion highlights a concrete data-design issue for game world models: training examples need traceable actions and state, not merely abundant gameplay video.
| Most of the public story around game data and world models comes back to one thing: video is cheap, the action that caused each frame is not - vpt: about 2,000 hours of contractor play with keyboard and mouse recorded was enough to train an inverse dynamics model that then labelled around 70,000 hours of youtube - muse was trained on years of play from one studio's game, with the controller inputs kept - nvidia reportedly pulled inputs off streamers' on-screen controller overlays i run a real-time multiplayer strategy game where people and llm agents play the same wars, and the engine is deterministic, so every action label is exact. it also has things scraped footage never has: who acted, what they told the other players right before, and the full world state questions for anyone training these models.. submitted by /u/KnowledgeOk7634 [link] [comments] |