Gemma 4 E2B / E4B assigns a separate "Per-Layer Embedding" (PLE) to each word at every layer, accounting for 35.2% parameters in E4B and 5.64 GB under BF16. OPTIC stores only a single set of quantized codes shared across all layers for each word, then uses a small affine decoder at each layer to reconstruct them. The codebook and decoders are fitted based on the error at the PLE output projection, rather than on the stored vectors themselves.
Likes 14 · Code Stars 5
Reasons
- The authors claim that under approximately 100x compression, the BF16 E4B retains 97.8% performance across 7 benchmarks, whereas the strongest of six conventional methods, Product Quantization, retains only 66.3%; similar compression retains 96.9% on the official W4A16 (4-bit weights) version.
- More extreme compression also has numbers: 594x retains 90.5%, 1002x retains 84.1%, and at 1002x, the PLE for E4B is reduced to just 5.6 MB (as reported by the authors).
- It saves VRAM with almost no slowdown: The README states that on a single RTX 4090 with a fixed 1 GiB KV cache, BF16 weight VRAM is reduced by 29.7% and W4A16 by 43.1%, with latency increasing by at most 0.40% / 0.69%; vLLM reports model VRAM dropping from 15.3 GiB to approximately 10 GiB.
- Directly relevant to edge deployment: The E-series of Gemma 4 is inherently designed for low-VRAM devices, and the embedding table is the most space-consuming part that is hardest to compress using conventional quantization.
Usage
- Code: github.com/VILA-Lab/optic (Apache-2.0); Compressed artifacts: Hugging Face zcahjl3/gemma-4-E4B-it-OPTIC-100x (apache-2.0, single compressed_ple.npz), which automatically downloads along with the specified version of the original Gemma 4 E4B model on first run.
- Requires Python 3.12 and a CUDA GPU. The README states that 24 GiB of VRAM is sufficient for short generations with BF16 E4B: run pip install -r requirements-inference.txt && pip install -e . then python examples/generate.py --prompt "…"; add --image for visual question answering.
- vLLM (documentation requires 0.19.1): Point PLE_OPTIC_ARTIFACT to the downloaded npz, and serve as usual with vllm serve google/gemma-4-E4B-it; the plugin installed by the repository replaces the dense PLE table during loading; already loaded models can also be integrated using install_compact_ple(model, artifact).
- Reproducing the paper: Run ple-eval evaluate --gpus 0 --run-id optic-e4b to run 7 benchmarks; see docs/fitting.md to fit your own compressed version.
Risks
- Only targets models with Per-Layer Embeddings like Gemma 4 E2B / E4B; other weights are not compressed; only the 100x artifact for E4B BF16 has been released, while E2B and higher compression ratios require self-fitting.
- Under vLLM, compressed PLE only supports tensor parallelism 1; VRAM and speed figures were measured only on a single RTX 4090 and are all author-reported.
Editor's Note: Note that "100x" refers to the embedding table itself, not the entire model; the overall VRAM savings are approximately 30%.