This is an interactive video world model: given a starting image, hold down a key, and the model generates subsequent frames one by one based on player actions, streaming them to the browser. The author's custom Summaryformer architecture enables each frame to simultaneously produce a full image and 96 summary tokens; the two most recent frames retain full resolution, earlier frames keep only summaries, and the starting image is always retained. The author claims that for the same duration, the number of keys attention needs to read drops from 26,208 to 6,528. The model has approximately 6.3 billion parameters and was trained on a 800-hour GTA5 dataset built by the author; the repository includes weights and a complete inference service, but not the training code.
Why it matters The author trained from scratch on consumer-grade hardware, using a memory design that "keeps full details for recent frames and only summaries for distant frames" to reduce attention overhead, allowing direct play in the browser.
Hugs 11 · Hug Upvotes 11 · Hugs Downloads 0
License: Commercial use allowed · ComfyUI: Not available · VRAM: VRAM requirements not specified
Reasons
- Apache-2.0 license; the model card clearly documents the memory mechanism, which parts are the author's own design, and which works were referenced
- There is an online demo page, and the server-side protocol is also provided
Usage
- Run
hf download lucidml/gta --local-dir gtafollowed bypython summary\_server.py(default port 3002), upload the starting image according to the protocol, and send key presses via WebSocket; requires a CUDA GPU and PyTorch
Risks
- Training data comes from GTA5 game footage, so generated content involves game copyright issues that require careful evaluation before commercial use
- The browser client is not included in the repository; one GPU process can only serve one player at a time, and the model card does not specify VRAM requirements