A recipe for converting "autoregressive → masked diffusion language models" by training only LoRA: all base model parameters are frozen, the adapter accounts for 4.1–5.8% of the final parameters, and the converted model can start from a fully masked sequence to perform parallel prediction and iterative denoising, utilizing bidirectional context in controlled attention tests. The authors successfully converted four models—Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B—without architecture-specific tuning, claiming Ministral is the first publicly reported conversion of a Mistral-family model.
Likes 8 · Code Stars 2
Reasons
- Low cost: The authors state that each conversion takes 25,000 iterations and at most 102.4M tokens, completing in 3.1–4.0 hours on a single Google Colab GPU; the parameter count remains unchanged after merging the adapter back into the base model.
- The authors claim the strongest BYOD-Gemma averages 50.6% across nine tasks, compared to 51.9% for LLaDA 8B Instruct under the same evaluation framework, while retaining most of the parent model's knowledge.
- Directly observable: All four conversions have Hugging Face Spaces online demos, and LoRA adapters for Gemma / Llama, etc., are already on Hugging Face (e.g., Ruurd/BYOD-Gemma-2-9B). Training and evaluation code is MIT licensed, and the Colab notebook allows viewing loss curves and the denoising process while training.
Usage
- Fastest: Open the Colab notebook notebooks/train_diffusion.ipynb in the README, select a model to train, and open the interactive inference page at step 6.
- Local: Run pip install -e ., cp .env.example .env and fill in a Hugging Face read-only token, then run python train.py --model llama --mode quick (quick mode uses a small 1B-class model and rank-128 LoRA, which does not reproduce paper scores; paper mode uses rank-1024 LoRA, with the original run requiring 96 GB of VRAM).
- Inference: Use load_hub_adapter_session("Ruurd/BYOD-Gemma-2-9B", …) from lad-generic along with denoise(…); standard causal generate() cannot be used. python merge_adapter.py can merge the adapter into the base model.
Risks
- The authors explicitly state that open-ended generation quality at low step counts is the main weakness: generating 128 tokens with 64 steps results in a perplexity of 8.3–11.1, compared to 4.1 for LLaDA and 2.2–3.0 for the original autoregressive model.
- Parent models like Llama and Gemma are gated repositories requiring acceptance of terms first; adapters follow the base model's license (e.g., the Gemma license).
- Code benchmarks in the evaluation execute generated Python; the README recommends using an ephemeral environment; scores are as reported by the authors.