- DiffusionGemma: an experimental open model, delivers up to 4x faster inference on dedicated GPUs, facilitating speed-critical, interactive local workflows.
- This model moves beyond the sequential token-by-token processing typical of autoregressive Large Language Models (LLMs) through its 26B Mixture of Experts (MoE) architecture.
- DiffusionGemma generates entire blocks of text simultaneously, achieving up to 4x faster text generation on GPUs.
- The model integrates a novel diffusion head aimed at maximizing generation speed, shifting the decode bottleneck from memory-bandwidth to compute.
- Blazing fast inference rates are achieved, with 1000+ tokens per second on an NVIDIA H100 and 700+ tokens per second on NVIDIA GeForce RTX 5090.
- Operating as a 26B total Mixture of Experts (MoE) model, it activates only 3.8B parameters during inference, fitting within 18GB VRAM on high-end consumer GPUs when quantized.
- By generating 256 tokens in parallel with each forward pass, every token can attend to all others more effectively.
- DiffusionGemma employs bi-directional attention to facilitate this process, although the overall output quality may be lower compared to standard models.
- Performance on specific tasks may be improved through fine-tuning of DiffusionGemma.
DiffusionGemma has introduced a groundbreaking experimental model that transforms text generation, achieving speed improvements of up to 4x faster on dedicated GPUs. The model utilizes a 26B Mixture of Experts (MoE) framework, diverging from conventional autoregressive approaches by generating entire blocks of text simultaneously rather than one token at a time.12
This innovation enables 1000+ tokens per second delivery on a single NVIDIA H100 and over 700+ tokens per second on an NVIDIA GeForce RTX 5090. Notably, the model operates within 18GB VRAM limits, activating only 3.8B parameters during inference, making it accessible for high-end consumer GPUs when quantized.5
Despite its remarkable speed, DiffusionGemma's output quality is lower compared to traditional models like Gemma 4 due to its priority on fast processing and parallel text generation. However, it offers potential for improved performance through dedicated fine-tuning for specific tasks. This new model opens pathways for exploring speed-critical, interactive local workflows, marking a significant development in the capabilities of generative text technologies.10
“The newly introduced DiffusionGemma model achieves up to 4x faster text generation on dedicated GPUs. It operates as a 26B Mixture of Experts model, generating entire blocks of text simultaneously.”
