Original title: apple/DepthPro2
Apple Machine Learning Research team's second-generation Depth Pro, accompanied by a technical report on arXiv 2611.26603. It accepts a single photo, an entire video, or a live feed as input, and outputs per-pixel depth in meters at the original resolution, focusing on sharp object edges and stable inter-frame consistency. The architecture is a multi-view Transformer: the first third of the blocks perform intra-frame attention only, after which intra-frame and cross-frame attention alternate, followed by a DPT decoder that outputs inverse depth for each frame. It comes in four sizes: S (24M), B (97M), L (334M), and H+ (872M), along with a shared focal length estimator that automatically estimates focal length when it is missing from photos or videos.
Why it matters Video relighting, compositing, and 3D reconstruction all require stable per-frame depth; this single network handles both single images and videos, with four sizes ranging from laptop to high-end GPU configurations
Rising score 115 · Likes 126 · Downloads 0
Reasons
- Apple officially compared single-frame and multi-frame methods on the Sintel and Spring datasets, claiming superior performance in both edge sharpness (Boundary F1) and inter-frame jitter (Jitter) compared to previous methods
- For video, it processes 16 frames per pass by default, with adjacent segments overlapping by 4 frames; real-time mode considers only the current frame and the previous 15 frames, suitable for camera input
- It can export 16-bit inverse depth PNGs for direct use in Nuke and Blender compositing, or save as npz files with focal length for point cloud conversion; a Core ML version is also provided for Apple devices
Usage
- Code: https://github.com/apple-aiml-research/ml-depth-pro-2;权重:apple/DepthPro2-S / -B / -L / -H and apple/DepthPro2-focal
- Command line: After installing uv, run uv run depth-pro-2 predict -i photo_or_video -o out/ within the repository; it downloads H+ (3.5 GB) by default, use -m depthpro2-s for laptops; supports CUDA, Apple Silicon GPU, and CPU
- Parameters: Use --mode unbuffered for real-time processing, --window / --overlap to adjust segment length; --save-png16 saves 16-bit depth maps, --save-npz saves metric depth and focal length
Risks
- The weights are licensed under the Apple Machine Learning Research Model License, permitting use only for non-commercial research and academic development
- The repository does not include training or evaluation code; depth at infinity (e.g., sky) is infinite or several kilometers, requiring truncation before point cloud generation; incorrect focal length estimation causes proportional depth shifts. This entry has not been tested