A 8B unified discrete diffusion language model that handles image understanding, visual question answering, object grounding, text-to-image generation, and image editing within a single model. Two key innovations: first, cross-paradigm online policy distillation, which distills the understanding capabilities of autoregressive vision-language models and the generation capabilities of continuous flow models into a diffusion language model; second, "progressive self-correction" diffusion—during training, each token is gradually corrupted over multiple steps rather than being immediately replaced with a mask token, allowing inference to start with tentative predictions and refine them step by step.
Likes 7 · Code Stars 4
Reasons
- The authors claim state-of-the-art performance among unified diffusion language models on tasks including question answering, multimodal reasoning, grounding, text-to-image generation, and editing.
- Weights are available on Hugging Face at nvidia/NLD-Uni and can be loaded directly using transformers, with the text_to_image, image_edit, and generate methods handling generation, editing, and understanding respectively.
- The accompanying repository includes a Gradio demo, SGLang inference (understanding path only), and minimal training scripts for SFT and online distillation.
Usage
- transformers: AutoModelForCausalLM.from_pretrained('nvidia/NLD-Uni', trust_remote_code=True, torch_dtype=torch.bfloat16), then call model.text_to_image(...) and model.image_edit(image, ...).
- Command line: In github.com/NVlabs/Nemotron-Labs-Diffusion-Uni, run python infer.py --task text-to-image / image-edit / understand / grounding; run python gradio_demo.py --model nvidia/NLD-Uni to start a local demo.
- Environment: CUDA 12.6, flash-attn 2.8.0.post2; the model card states validation on H100 80 GB.
Risks
- The license is the NVIDIA Open Source License Agreement, not Apache or MIT; review the terms before commercial use.
- The model card provides no benchmark scores, only listing evaluation tasks and three functional smoke tests; the "state-of-the-art" claim in the abstract lacks verifiable numbers.
- Requires trust_remote_code; the model card claims implementation of 1024×1024 generation configurations but does not list validated resolutions; SGLang supports only understanding, while generation uses native PyTorch.