Shot Reader 8B: Analyze a shot to reverse-engineer camera position and scene

A shot analysis model fine-tuned by Voyager using LoRA on Qwen3-VL-8B-Instruct: Input a continuous shot, uniformly sample 16 frames, and output a JSON shot list including camera height, angle, pitch, field of view, camera movement, shot size, the size, pose, and time-varying ground coordinates of each person or object in the frame, action events, and set pieces such as walls and boxes. Coordinates are relative to the origin directly below the camera in the first frame, with units in meters; 3D tools or coding agents can use this to reconstruct the scene in Blender. Training data consists of synthetic shots rendered from procedurally generated scenes (some re-rendered to photorealism using video models), plus public domain films from Wikimedia Commons and the Prelinger Archives.

Original title: voyager-so/shot-reader-8b

A shot analysis model fine-tuned by Voyager using LoRA on Qwen3-VL-8B-Instruct: Input a continuous shot, uniformly sample 16 frames, and output a JSON shot list including camera height, angle, pitch, field of view, camera movement, shot size, the size, pose, and time-varying ground coordinates of each person or object in the frame, action events, and set pieces such as walls and boxes. Coordinates are relative to the origin directly below the camera in the first frame, with units in meters; 3D tools or coding agents can use this to reconstruct the scene in Blender. Training data consists of synthetic shots rendered from procedurally generated scenes (some re-rendered to photorealism using video models), plus public domain films from Wikimedia Commons and the Prelinger Archives.

Why it matters When agents build 3D pre-visualizations based on reference shots, camera position and movement are the most common sources of error; this model first reads the shot into a structured description and can run on a single consumer-grade GPU

Upvotes 29 · Likes 31 · Downloads 61

Reasons

  • The authors tested "correct classification of both camera height tier and camera movement" on 300 held-out shots: 69%, Opus 5.5 scored 44%, and Gemini 4 Argon 46%; however, on the scene reconstruction score for real-world footage in VibePrevisBench (35.6), it slightly underperformed these two models
  • The authors report that on H100 with FP8, a single shot takes 5.9 seconds and costs approximately 0.002 USD; on an RTX 5080 with 16GB using 4-bit quantization, it takes about 25 seconds per shot (including loading time)
  • Stated use cases: creating editable pre-visualizations from reference footage, converting camera positions and blocking into explicit inputs for controllable AI video, and tagging asset libraries by camera movement and shot size

Usage

  • Run pip install torch "transformers>=4.57" pillow huggingface_hub and ensure ffmpeg is installed; download shot_reader.py and schema.md from the repository
  • Run python shot_reader.py clip.mp4 --subject "P01: person (description)" --subject "O01: dog"; for limited VRAM, add --load-in-4bit (requires bitsandbytes); for speed, use --backend vllm
  • Only one continuous shot can be input at a time; for long videos, split them at edit points first; see schema.md for field descriptions, and the standalone LoRA adapter is in the lora/ directory

Risks

  • The authors state that action events and blocking paths are the weakest fields on real-world footage, with estimated positions from single-camera setups typically having errors of several meters
  • Comparative results are self-reported by the authors (using the same prompts and format for all models) and have not yet been reproduced by third parties; labels for real-world footage were automatically annotated by frontier models. This entry has not been run

Open

Hugging Face Trending

Explore the latest AI Radar updates