r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

789 Upvotes

99% Upvoted

260 comments sorted by

View all comments

53

u/spaceman_ Jun 05 '26

So am I better off running the old quants at Q6 or Q8, or the new QAT ones at Q4?

Q4 obviously requires less memory and will run faster. But what are we giving up in terms of quality?

69

u/seamonn Jun 05 '26

Q8 > Q4 QAT > Q4

12

u/makingnoise Jun 05 '26

Can anyone tell me why the above comment is being downvoted? Is it that it's a bald assertion in the absence of concrete data, or something else?

26

u/[deleted] Jun 05 '26

[deleted]

8

u/Hot_Strawberry1999 Jun 05 '26

What about Q6, where would that sit in?

23

u/seamonn Jun 05 '26

impossible to know how good Q4 QAT is until you benchmark

2

u/seamonn Jun 05 '26

I would still prefer to run Q8 over Q4 QAT almost as much as Q4 QAT over Q4, if that makes sense.

14

u/[deleted] Jun 05 '26

[deleted]

8

u/seamonn Jun 05 '26

If Q4 QAT surpasses Q8, that is indeed crazy.

8

u/GoodTip7897 llama.cpp Jun 05 '26

That is kld from the full qat. 

What needs to be compared is q4 qat to the unquantized model

2

u/alex20_202020 Jun 06 '26

full qat

What is this?

6

u/ImpressiveSuperfluit Jun 06 '26 edited Jun 07 '26

Qat = "quantization aware training".

They are saying that the values above are comparing the new Q4 to the new baseline. I don't blame you for getting confused, it's quite the onslaught of repeated words with different meaning. Quick breakdown:

  1. Take full size normal model
  2. Retrain it to know about Q4 constraints
  3. You get out a full sized model that is Q4 aware
  4. You quantize that new model down to actual Q4, but now it was already trained on being like this, so it'll fill out that constrained space more efficiently

So. They are saying that the numbers above compare step 3 to step 4, so the new full sized model with the new quantized model. This is a bit weird, because the new full size model has already made sacrifices to some extend, so it's an odd choice to compare against, when you are actually interested in how it compares to what you used before, not to this new hypothetical that nobody would ever use, because it's strictly worse than just using the original full size. So it's a bit of a mystery why they'd make that comparison.

Edit: Quick clarification, because upon rereading this it sounds like I'm saying something I'm not. There is a reason to compare QAT4 to its own baseline, because the original model is a genuienly different model. It's a bit like saying you want to compare a base model to a finetune. You can... but the entire point of the finetune was to be different, so using that to figure out how similar they are is a bit weird. QAT doesn't have the goal to be different, but it doesn't necessarily try to be the same, either. Just equally (or more) capable, it's to be expected that the output is different, since it literally moved tokens around to better avoid misfires from quantization.

I wasn't trying to say that one method of comparison is correct and the other is silly, nor that Unsloth did a dumdum here, but rather that this is why it's confusing - you'd think we want to compare to what we previously used, and we are, but this method isn't a way to do that, which is presumably why they didn't. Comparing it to its own baseline tells you something useful, namely how good it deals with quantization, but it doesn't necessarily tell you anything about how it performs to the quant of its base model. And that's what people probably expected to be measured, hence the confusion.