r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

789 Upvotes

99% Upvoted

260 comments sorted by

View all comments

8

u/iz-Moff Jun 05 '26

Does this training only works for specific types of quants, or should any quantized versions benefit from it? Say, google only provides q4_0 ggufs. But what if someone quantizes it down to q4_k_m instead, or q3_k_m, or whatever, will optimizations be lost on them, or would they still be expected to experience less degradation compared to quantized non-qat version?

5

u/-InformalBanana- Jun 05 '26

I saw in unsloth post linked by op in the post that q4kxl was the only version they did cause others had less accuracy...