Original title: Diffuspace/Clover-35B-A3B-Preview
DiffuSpace's first model: a text-only diffusion language model derived from Qwen3.6-35B-A3B, retaining its hybrid Gated DeltaNet / full-attention MoE backbone (40 layers, selecting 8 out of 256 routed experts, with 1 shared experts per layer), totaling 35B parameters and activating approximately 3B per token, with a context length of 128K. Generation proceeds in blocks: each block starts fully masked, positions with confidence above the threshold (default 0.95) are filled in parallel, and if none meet the threshold, the most confident position is filled; once a block is complete, it becomes the prefix for the next block (default block length 32). To support this, learnable GDN parameters were added to allow bidirectional interaction between tokens within a block while maintaining causality for the prefix. The authors claim that at the time of release, it was the best open-source diffusion language model on the benchmarks reported in their blog. The repository was created on 10/24, and it was not included in previous issues.
Rising score 87 · Likes 247 · Downloads 3572
Reasons
- A large-scale version of a diffusion language model that can run directly: it determines multiple tokens at once; within a block, it can also use only a set of shared experts (accumulating routing scores up to 90%), reducing the number of distinct experts invoked within the same block.
- Multiple deployment options: standalone inference with Transformers; the officially recommended SGLang implementation supports block diffusion denoising and self-speculative decoding, while llama.cpp supports self-speculative decoding (both submitted as upstream PRs).
- Apache-2.0 license, retaining upstream Qwen copyright notices and including file-by-file modification notes; face-hug likes 247 (representing attention only).
Usage
- pip install "transformers==5.19.0" accelerate safetensors (PyTorch 2.10+), AutoModelForCausalLM.from_pretrained('Diffuspace/Clover-35B-A3B-Preview', trust_remote_code=True, dtype=torch.bfloat16, device_map='cuda:0'), set enable_thinking=True when applying apply_chat_template, generate(max_new_tokens=24576, block_size=32, threshold=0.95, do_sample=False). BF16 weights are approximately 69 GB, requiring a GPU with sufficient memory.
- Repetition prevention and termination control are included in the default configuration (presence_penalty 0.9, repetition_penalty 1.05, gradually favoring ending after responses exceed 8192 tokens, and gradually favoring conclusion after thinking exceeds 4096 tokens), so they generally do not need to be changed; use the official SGLang implementation for production services and evaluations.
Risks
- Preview version: neither the model card nor the README lists scores, and benchmark results are only shown in images on the official blog; the authors also note that very long generations sometimes repeat.
- Code for SGLang and llama.cpp backends, as well as SFT code, is marked as "coming soon" in the repository; outputs from the Transformers version and SGLang may differ due to numerical discrepancies.
- Requests exceeding 128K are directly rejected (including space required to complete the final full block).