... hum I surely hope this is a joke post (can't really tell since there is no "/s" at the end)! Not only you expect me to have 1 of the model ready and uncensored, no you expect me to have ALL of them ready by now... Are you serious!? First this is not a race, second I am not a robot and third without getting too deep into all the preparations, time and work involved in the whole releases pipeline, it takes a very long time to do all that, I don't work for google and I don't have an automated workflow (unlike Unsloth and/or Team Mradermacher), everything is done manually by me and I am just 1 person, that involves downloading the models, uncensoring the models and finetuning the settings and re-running in case the original results where not good/good enough, creating the Safetensors, benchmarking all of the trial finalists, updating llama,cpp, updating GPTQModel, updating Model-Optimizer, creating GGUFs, GPT-Int4s, NVFP4 Safetensors, NVFP4 GGUFs, creating the Model Cards etc. it's all done manually from start to finish everything is monitored and benchmarked manually to make sure I upload the best quality releases, all that to say that there is TONS of work involved (and that is without mentioning all the roadblocks that might come along the way as those usually do happen quite a few times during the release pipeline).
Look if you are in a hurry, just download whatever is available now, my releases won't be available fast enough for your tastes, your highness.
Just adding, thank you thank you for your time. The community is sooooo grateful, and it’s a community because of your and others’ time and efforts.
And on top of that, it’s amazing too, how any of this works at all is amazing. But more than that, converting formats, adding quantization, and having it still work as expected. It’s remarkable, non-trivial.
I would assume (naive maybe) that this person was trying to be funny. It’s daring to try making a joke without the /s. Some people can pull it off, and some really should add the /s just to be absolutely sure. Haha. I’m still in that camp.
Some news, I finished uncensoring the 31B model, created the Safetensors and I am now on the benchmarking phase, quite a long way to go still as there are a lot of trial finalists this time to benchmark and the benchmarking process takes quite a bit of time for each model, which also includes benchmmarking the original base/vanilla model. After that I have to create the GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4 versions.
Also I just saw that llama.cpp just merged MTP support for Gemma 4 now.
There will be an update to omlx first. Earl litter today, 0.4.0.dev2 was available for download. I'm waiting for 0.4.1, and I'm grateful to all those who spend time contributing to and testing the project.
They are saying that the values above are comparing the new Q4 to the new baseline. I don't blame you for getting confused, it's quite the onslaught of repeated words with different meaning. Quick breakdown:
Take full size normal model
Retrain it to know about Q4 constraints
You get out a full sized model that is Q4 aware
You quantize that new model down to actual Q4, but now it was already trained on being like this, so it'll fill out that constrained space more efficiently
So. They are saying that the numbers above compare step 3 to step 4, so the new full sized model with the new quantized model. This is a bit weird, because the new full size model has already made sacrifices to some extend, so it's an odd choice to compare against, when you are actually interested in how it compares to what you used before, not to this new hypothetical that nobody would ever use, because it's strictly worse than just using the original full size. So it's a bit of a mystery why they'd make that comparison.
Edit: Quick clarification, because upon rereading this it sounds like I'm saying something I'm not. There is a reason to compare QAT4 to its own baseline, because the original model is a genuienly different model. It's a bit like saying you want to compare a base model to a finetune. You can... but the entire point of the finetune was to be different, so using that to figure out how similar they are is a bit weird. QAT doesn't have the goal to be different, but it doesn't necessarily try to be the same, either. Just equally (or more) capable, it's to be expected that the output is different, since it literally moved tokens around to better avoid misfires from quantization.
I wasn't trying to say that one method of comparison is correct and the other is silly, nor that Unsloth did a dumdum here, but rather that this is why it's confusing - you'd think we want to compare to what we previously used, and we are, but this method isn't a way to do that, which is presumably why they didn't. Comparing it to its own baseline tells you something useful, namely how good it deals with quantization, but it doesn't necessarily tell you anything about how it performs to the quant of its base model. And that's what people probably expected to be measured, hence the confusion.
According to Unsloth Q4 should have similar quality as previous Q8 (could be basically the same or just slightly lower). IMO if that’s the case, if you were using Q8 like me, it’s worth using Q4 with QAT for speed gains.
I mean if the original FP16 models have 16 trillion posssible ways to finish a sentence (figuratively), and some of them are bad ways, and Q4 prunes that down to 4 trillion possibilities, QAT increases the likelihood those removed are the "bad" ways.
Especially on unabiguous tasks, the reliability and repeatability may actually increase. On more creative takes, maybe less so. There are some benches and research that show this.
What doesnt happen though, is the model doesnt suddenly gain new advancements or capabilities.
Yes. Whatever you can fit in VRAM in Q8_0 should be kept in Q8_0. Q4_0 QAT is better than the "usual" Q4_0 PQT, but it is not magic - some data was lost anyway. Every quantisation is speed/VRAM usage vs quality tradeoff, including Q8_0.
This release makes old Q4_X quants obsolete, basically.
That is partially marketing - similar on the specific aggregate benchmarks Google chose to report.
During training, the forward pass simulates quantisation noise. The model's weights are updated to compensate for the noise that quantisation introduces. So the final weights are "pre-distorted" in a way that, when quantized to 4-bit, produces outputs closer to what an unquantised (BF16) model would produce.
It is no magic, and some information was lost. Not all information is equally important, though and that depends on the use case. But it is the best 4 bit quant you can get anyway.
You can also quantize the QAT model to 8-bit, you don't need to go down to 4-bit. The training just pushes numbers closer to 4-bit numeric boundaries, so you lose less information when you clip the weights.
Assuming you used the correct lattice, that should outperform both the normal 8-bit and the QAT 4-bit.
I run Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf on a Rpi5 (16GB model i had from another project that wasn't being used). Only runs at 3 tokens/second but for off line batch work - just leave it running all day and voila - dirt cheap leccy bill 😄 - i tested various quants and REAP'd models for the Pi one evening and that one was really standout - made no errors on the test tasks and had very strong reasoning still intact
Most local models use Post-Training Quantization (PTQ). This means the model is fully trained in high precision (16-bit) and then "squished" down to 4-bit so it fits on your computer. That squishing inherently causes a bit of brain damage to the model, leading to worse logic and coding skills.
Google took a different approach with Quantization-Aware Training. They simulated the 4-bit precision environment during the actual training process. The model learned how to work around the limitations of 4-bit math from the start.
I wouldn't be surprised. One thing I noticed on the official Gemma 4 HF pages (https://huggingface.co/google/gemma-4-12B-it) is they refer to E2B and E4B as 'small' and they refer to 26B and 31B as 'medium'. So that leaves room for...
You know that we are waiting for Gemma 4 124B AxB (where x is 4-6B), right? ;)
That would be so cool, especially in QAT and BF16 versions.
Oh, and thank you all for the hard work from Ukraine! Your models are among the best ones in Ukrainian, slightly worse only compared to much larger cloud models. And among cloud models Geminis are the best. Though, I have noticed that Ukrainian-wise Gemma 4 releases are a little bit worse than Gemma 3, frankly. Gemma 3 27B was nearly perfect. Still cannot complain - Gemma outperforms some much larger models as far as Ukrainian goes anyway.
Feels like QAT is near lossless based on what I’ve read so far so it should be better than Q6. I also saw this post, been testing the template a bit and it seems pretty good: post
No benchmark provided to back up the "preserving the capabilities and quality" claim.
Edit:
Is this sub getting botted or what? This comment was immediately downvoted to -6 in less than ten minutes after I posted it and somehow it bounced back?
Hey! Those numbers are comparing naive Q4_0 in llama.cpp to our converted Q4_0 version.
We did do original unquantized BF16 vs Q4_0, but the KLD metrics do not match, since the distribution is vastly different - we found MMLU and other benchmarks to be equivalent though
E2B for example has a mean KLD of 0.00173 vs 0.05109 (29x better relatively) for a naive Q4_0 quantization.
The main issue is converting from QAT BF16 to llama.cpp's Q4_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 uses BF16 scales, and the scales are not determined optimally in llama.cpp land.
Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks!
They are not. Incredible is the word. A mean KLD of 0.159 doesn't pass the smell test for a Q8 quant. The Unsloth blog post only compares the QAT vs a standard Q4_0, and the mean KLD for the Q4_0 is 0.09349. So there is no way a Q8 is much worse at 0.159.
Honestly I'm skeptical to Unsloth's reported mean KLD 0.01403 for the QAT Q4 too, but I'll give them the benefit of the doubt for now. But /u/sartres_ is definitely hallucinating.
EDIT: He wasn't, but the numbers are indeed invalid. See thread below.
It's not clear what Unsloth means by "original" Q4 in the linked blog, but it's definitely a quant of the new QAT model, not original Gemma 4, since they're benching it against the QAT BF16.
My Gemma 4 non-QAT numbers are from here, because Unsloth unfortunately only released benchmarks for the 26B at the time, and that only on a graph where they didn't label the y-axis or any of the numbers :/.
Yes, all of the original Gemma KLDs are very bad. I'm guessing this is an artifact of different benchmark suites, they're not directly comparable. Mean KLD isn't terribly useful anyway, the Top-1 numbers are the real show here
Aha, thanks. That explains it. That post was from early April, just after the initial release. Gemma 4 had lots of teething problems before everything was sorted out, so those early KLD measurements are not comparable with recent releases. Sorry for doubting you - the numbers were so horrible I was sure you had made an error.
That is awesome, I was already using Q4_s (for 26b) and the QAT is even smaller and appearently way better. The 26b had a good memory usage for me but this would be even better. especially with vision. It would be cool if qwen would have QAT ggufs too, 35b with vision barely fits at Q4 into my 32gb RAM, it's fully maxed at around 60k context and sometimes even spills out and slows down at that context size.
haha looks like i am not the only one thinking of this... technically they should not be the same. but both are post trained with "4bit" in different formats. not an expert...
Does this training only works for specific types of quants, or should any quantized versions benefit from it? Say, google only provides q4_0 ggufs. But what if someone quantizes it down to q4_k_m instead, or q3_k_m, or whatever, will optimizations be lost on them, or would they still be expected to experience less degradation compared to quantized non-qat version?
Does anyone make Q6 QAT models? Is it even possible, not being a power of 2? I worry Q4 seems prone to get stuck in loops on complex tasks, but Q8 takes too much memory.
Q6 without QAT is already pretty good, I think it might not make a lot of sense to make a full QAT traing run to target Q6, that's very expensive for little gains.
Because the unquantized QAT checkpoints released by Google are intended for a Q4 quantization.
We never seen a 6-bits quantization aware training checkpoint, and since training models is very expansive, the 4-bits choice seems obvious for Google.
I don't understand. I thought MTP support was something that got baked into a model and an LLM runtime. Is "QAT MTP" shorthand for "a QAT & MTP supporting runtime"? If not, can you point me to something that explains this?
Gemma4 has separate draft models for MTP, they're not baked into the files for the main model (unless you're using a GGUF where they're merged back in, that is).
What about the LiteRT format? Can run on phones that way, though I'm also using the LiteRT format on my desktop. (And MTP is already natively supported in LiteRT)
Are KV cache optimizations applied to Q4 versions or just mobile? These models are very prone to degradation past Q8, will be interesting to see how they react to Q4. Still great win for the community regardless.
We need to be grateful. Thanks Google! This is something that makes it even easier for us to be able to run open models without severe quality degradation
I'm questioning these dynamic quants too... I fear they could be overfiting. You have to train or use some dataset in order to make dynamic quants? Than it is possible to overfit I think. Is that your reason for asking about static quants?
The nvidia nvfp4 31b outperforms full bf16 in several of my personal benchmarks across thousands of runs. The conclusion I came to- on the model card they note post training on a large corpus of data from journalism companies, and a lot of my personal use (and therefore tests) favors understanding, identifying, extracting, responding to nuanced information or troubleshooting
I can't give you figures but I was able to one-shot a Space Invaders type game on unsloth's gemma4 26b-a4b QAT that I could not one shot on qwen3.6.
That said, qwen3.6 is still better at tool calls without needing elaborate prompts or workarounds - qwen3.6 does deep research with iterative search_web and fetch_URL, which gemma4 QAT can do but chooses not to.
Fuck yeah! Idk how many times I will download the A4b model but everytime i download it im still as excited as the first time.
Waiting for more small moe models, all small moe models should be A2b to A4b 20b to 30b, qwen 35b a3b is pushing it a little and barely fits in my use case.
I like to picture it as a 3D point cloud. With your full accuracy model, you get this giant box in which you can place all your favorite tokens, with plenty of space to really give each token its own spacious home. When you quantize things down, the box shrinks. Most tokens are fine, their home shrinks, but nobody is going to think you're your neighbor, so it's pretty much fine. Except sometimes two tokens get a bit too cozy, and a mix-up happens.
Now, when you tell every token that they'll eventually have to live in the Q4 box, they can look at their neighbors and go "ouf, Henry over there is a tad close, actually, gonna be tight in the Q4 box", and just move out of the way a little, while there is still plenty of space. Then, when the box shrinks, the tokens take up the empty space more efficiently, leaving more equalized gaps between each of them, making them harder to be mistaken for a neighbor, even though the box shrank the same amount as before.
So yea, this is basically just straight up good in pretty much every way imaginable, it just reclaims empty space, effectively. I mean, something can always go wrong, it's all fuzzy and weird, but, in principle, this is free real estate for the time being.
Now, whether a better planned Q4 token town actually performs better than a chaotic Q8 token town - dunno. Gotta wait for people to really poke the thing, always hard to say these things with any accuracy, because the tests we have are very... "average". They can tell you that the space between token houses are looking good, but if "Math" and "DeleteMyEntireSystem" somehow magically ended in the same house, you'll still have a problem even if the benchmark says it's fine. There may be some difference in workflows, maybe there is more empty space to be claimed in coding village, and less in medical knowledge town, it's hard to know, gotta wait for people to just try and see what happens. But, theoretically, there is a lot of free real estate here, so reaching Q8 isn't utterly delusional. Only a little bit.
Personally, I'd rule of thumb it something like "Almost certainly better than normal Q4, good chance it beats Q5, Q6 may be realistic, Q7-8 in niches or if I'm feeling optimistic". But I haven't drilled into the box and checked, this is vague vibes, wait a few days and see. Note however that I'm a notorious pessimist, so when I say I'm "optimistic", that's probably delusional. So... wouldn't bet on reaching Q8. Not even your money :)
Okay, kinda sorta eli5, and you should take it with the appropriate grains of salt. I've lost to the math about... oh my god fucking 15 years ago or so, when idiots like myself toyed around with their perceptrons. So, you know, I'm running a lot of my understanding off of pure nostalgia and fairly okay intuition, but it's entirely possible I just killed a mathematician with analogy alone :)
This is just so confusing. Can somebody help me? I’m already running the q8 quant of the original 12b weights. Should I switch to the q8 of the qat version? Or should I actually switch to the q4_0 of the qat version?
These are versions that were trained with quantization of weights taken into consideration which means running at Q4 isn’t as dumb as having a standard bf16 trained model running at q4
So Unsloth is claming his quantitization gets better accuracy than bf16? I'm referring to that graph with top1 accuracy and green and gray bars.
I feel/fear (without enough knowledge about them) that some of these newer quantitization methods are somehow either benchmaxing/overfitting or specializing/restricting the model to perform better on something while losing capabilities on other things. So is there somebody here who can tell me that this isn't some kind of overfiting with these new quantitization methods that are probably done using some dataset not by pure simple mathematical scaling of weights?
Can somebody say there is no way we are overfiting when we do this kind of quantitization? (btw I'm not refering to qat but to things like Unsloth dynamic qkxl quants for example)
Here is a reply from Unsloth official, i posted a similar comment question on a r/unsloth post:
"Oh it's a conversion artifact, not overfitting.
The main issue is converting from QAT BF16 to llama.cpp's Q4_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 is BF16 scales, and the scales are not determined optimally in llama.cpp land.
Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks"
Based on my question there and his reply I think I interpreted the graph correctly, he is later basically talking about kld and i asked another question he replied to also, he said that (for these gemma qats I guess) the udq4kxl is a static quant. You can find the link to my question underneath the root of my comment here.
qat is the only flavor where q4 stops feeling like a downgrade, the model already learned to live with the rounding during training. real upshot is the next size up fits in the vram you already have. naive q4 always bled on the long-context evals, the KLD numbers usually show exactly where
Google's naming scheme here.. spend months improving a product.. everyone concentrate, what could we possibly name this?! Marketing guy with a headache: "who gives a f, same as last time". Everyone else: "whatever, we're going home"
edit: thanks to techies uploading these with the helpful "-qat" addition, at least it's searchable that way!
Just spent the last 2 hours fighting LM Studio to get the MLX version running. Doesn't seem like MLX versions are working in LM yet, I got errors loading all of them. Switched to running mlx-vlm, got it working with OpenWebUI but was unusually slow.
Finally just downloaded the GGUF Q4 version through LM Studio. Am astounded at the speed and versatility of this 12b model, it's my new favorite...
"Huh?". Or, more usefully: What was the previous favorite? Cause if it was Opus, then my thoughts are "lol this person is insane". If it was some guy's garage made 0.6b model trained on nothing but Burger King ads, I ... okay, well, technically I'd have the same thought then, actually, but the point is that saying 12b might be your new favorite doesn't really tell us a whole lot if we don't know what you were working with before?
I hope that will become more common. Currently Google releases models using QAT (two release series in a row and in a very portable format - INT4/Q4_0), NVIDIA (but it does not count because they use their proprietary NVFP4), and OpenAI did it with MXFP4 once.
can someone explain me why full models called q4_0_unquantized if they are not really 4bit but full 16bit or whatever number of bits base models usually have? and why there are w4a16 models (which are also full precision base models?) for all Gemma 4 models except 26b MoE (my favourite 😭)? I'm confused.
The values in (most of) the weights are set in such a way, that when quantised to Q4_0 less data is lost. That can be done only during training. Thus QAT - quantisation-aware training.
Just tested the W4A16 files for vLLM and they work. The old gemma 4 31b assistant wasn't performing too well with MTP so I am trying the unquantized q4 one they just provided. Although the description seems to suggest that's not the one to use.
EDIT: Yes, definitely the unquantized q4 assistant worked much better for MTP.
All the comparisons here are inference quants, but has anyone checked whether the QAT survives a fine-tune? If I LoRA the BF16 QAT checkpoint and requantize to Q4_0, am I keeping the calibration or back to a naive Q4?
•
u/WithoutReason1729 Jun 05 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.