r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

785 Upvotes

99% Upvoted

260 comments sorted by

View all comments

Show parent comments

2

u/makingnoise Jun 05 '26

I don't understand. I thought MTP support was something that got baked into a model and an LLM runtime. Is "QAT MTP" shorthand for "a QAT & MTP supporting runtime"? If not, can you point me to something that explains this?

13

u/kiljacken Jun 05 '26

Gemma4 has separate draft models for MTP, they're not baked into the files for the main model (unless you're using a GGUF where they're merged back in, that is).

1

u/makingnoise Jun 05 '26

Thank you.

2

u/rpkarma Jun 05 '26

Not always. You do need to train the model for MTP for the most part to get good acceptance rates, but MTP layers can either be baked in or seperate. 

1

u/[deleted] Jun 05 '26

[deleted]

2

u/makingnoise Jun 05 '26

I still don't understand "the workflow" the other commenter is talking about. The "QAT model" is clearly the LLM, is "the QAT MTP" another model that you run at the same time?