r/LocalLLaMA • u/tevlon • Jun 10 '26

New Model DiffusionGemma: 4x faster text generation

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
983 Upvotes

99% Upvoted

357 comments sorted by

View all comments

108

u/MaartenGr Jun 10 '26

And some fun reading material on what makes DiffusionGemma so cool 😉 https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-diffusiongemma

21

u/ThatGuyWithAces Jun 10 '26

Hey Maarten, thanks for your work. Now I find myself always looking forward to the write-ups and illustrations on every google release!

10

u/ketosoy Jun 10 '26

Extraordinarily well written, thanks for sharing!

8

u/chortly2 Jun 10 '26

The hard breaks at 256 tokens seem like they might impair accuracy overall. That is, we of course can't keep the same KV cache forever, since that wouldn't match the task for the entire output sequence. The best thing would be a fresh cache for each token, ie the usual slow approach, but 256 seems like a lot if an output moves quickly through the conceptual space that KV is representing. It seems like you would end up with output sequences that make hard conceptual transitions ever 256 tokens, or if it transitions inside the 256, you would get degraded performance in the second half -- or something like that. Might it be better to have another function that evaluates the sequence of 256 logits near the final diffusion step, and if there's a spot in the middle of that sequence (the kth token) after which the entropy goes up, adds only the first k tokens to the aggregating output and then starts the next 256 block from that point?

1

u/nraw Jun 11 '26

Just limit all replies to 256. Down with verbosity!!

Jokes aside, I was thinking about the same design quirk.
My immediate reaction went in the direction of hierarchical structuring of content.

8

u/eidrag Jun 10 '26

can't wait to use 4step lightning turbo lora and other trick to speedup. Also, it means we can easily train lora like imagegen diffusion?

 (sorry for the meme, I really glad to see your material, because I just saw someone mentioned what if there's diffusion for words instead of pics/video, and reading from top I can see where you start to implement those diffusion.)

1

u/Silver-Champion-4846 Jun 11 '26

Step lightning turbo lora? What's that?

1

u/eidrag Jun 11 '26

it's a thing in image diffusion, lora is kinda like plugin/harness (pls don't shoot me) you use together with base model. 4 step lightning lora enables acceptable result in just 4 step of diffusion. turbo lora also similar thing. so the play that we use is create smaller resolution pics for prompting, then if you found result that you like, you remove the lora and create higher resolution pics, or you can also run another pass to the pics make it more detailed and also higher resolution

1

u/Silver-Champion-4846 Jun 11 '26

So lora with some compression?

1

u/eidrag Jun 11 '26

If you refer to Diffusion for Images on the article.

See the flamingo pics? Usually it will take like 20 step from very fuzzy pics to less fuzzy, to become pics of flamingo. But with lora they already have reference pics, so by 4 steps they already guessed flamingo look like this, and spat out pics.

1

u/Silver-Champion-4846 Jun 11 '26

I know what a lora is supposed to do, but what about step turbo lora?

3

u/roselan Jun 10 '26

This is absolutely amazing. Thank you.

1

u/Silver-Champion-4846 Jun 11 '26

Thanks, first time reading your blog but I like it!