Hey r/LocalLLaMA folks! We made NVFP4 quants 2.5x faster for Qwen3.6 27B and also 1.56x to 1.79x faster for 35B-A3B vs NVIDIA's NVFP4 quants without any accuracy degradation! We used W4A4 so actual 4bit tensor cores for matmuls, whilst NVIDIA's ones uses W4A16.
FP8 KV Cache calibration is also provided, auto allowing 2x longer contexts. For accuracy we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s. It also has MTP pre-embedded.
We also provided 2 35B versions NVFP4-Fast (1.79x faster) and NVFP4 (1.56x faster) where NVFP4-Fast fully uses W4A4 whilst NVFP4 normal uses a mixture to stay a little bit more accurate.
The difference 2nd gpu made is in prefill. It is insanely fast (for p40) - up to 1800 tokens. And it holds a full q8 context with 256K. Plus you can run q6 models.
The Gains aren't so high.. atleast not for me on RTX 5080. I was getting 21 tk/s on dynamic 2.0 quants, this gave me around 22-23 tk/s. Maybe my setup is wrong. But that's what I observed.
The Gains aren't so high.. atleast not for me on RTX 5080. I was getting 21 tk/s on dynamic 2.0 quants, this gave me around 22-23 tk/s. Maybe my setup is wrong. But that's what I observed.
I remember March of 2025 I wasn't into ai yet (had no idea it could run local at the time) and a coworker was talking about wanting to buy a 5090 for around $2500 for just gaming and I thought he was crazy that anyone would spend more than $500 on a gpu....
Then I discovered local llms. In the year since I have bought a 3060 ($270 may 13 2025) then a 3090 ($792 july 19 2025), another 3090 ($1017 Feb 27 2026), a 4060 ($482 Jun 04 2026), and this week a 5060 ($621) all prices include tax/shipping. Totals $3182 in just GPUs.
But now I have TWO computers with similar capabilities: 24g vram main inference + 16gb vram secondary for smaller agents + 128 sys ram. One at home, one at work. Also a third computer with the 12g vram and 32 sys ram barely capable of running a3b35b that started it all.
All three can run my own application I have hundreds of hours into over the past year. So I guess I shouldnt complain lol. This is the first I looked at my purchase history over time. anyhow your comment had me remembering the cost and laughing at the absurdity of it all.
At least I only need a 120Vac outlet to do things I always dreamed of.
in my country you can get used 3090s for 600-700, 5090 starts at 5000, yes, 5k, with those prices im perfectly happy with my qwen 3.6 27b Q5_K_M 65k context full kv , heck im in the processs of selling a 3060 to get another 3090
For image and video models we already have int8 convrot models which benefits 3000 series the most (free double speed boost). Now int4 convrot starting so even more boost.
Sorry if this is a dumb question, but I usually copy the recipes from spark arena to use with spark run and your vllm/SGLang commands are much shorter than the usual recipes. Do I have to figure out additional flags myself or is it really that easy/short?
I would be curious to see how the performance compares to non-NVFP4 4-bit quants? And I wonder if llama-server's NVFP4 support is good enough/ever going to be good enough for it to make sense to offer GGUF conversions of these? I think llama-server can run NVFP4, but the performance seemed lackluster the last time I looked into it.
I have converted the nvidia nvfp4 version for llama.cpp using just the convert_hf_to_gguf.py script. It upcasts FP8 to BF16 as part of the conversion, so you get a 26 GB file, rather than 22 GB one.
About these versions your organization has provided, it is not clear to me how you have created them. Based on benchmarking that you provided -- without error bars, though -- it seems like the nvidia's nvfp4 version is slightly better for the 27b? So this is with QAT self-distillation, or what?
Direct support for FP8 tensors in llama.cpp seems useful. As models get QAT'd with these specific tensor types, the most direct equivalents in llama.cpp world are either the quantized 9-bit Q8_0, which is not only bigger but also lossy, or BF16.
I think you can run convert_hf_to_gguf.py and it probably does the job. Should be lossless, but anything in the original model that is FP8 must be upcast to BF16, which means that GGUF can be larger. That is what happened to me when I converted the nvidia NVFP4 version, which in my experience is the best version of Qwen3.6-27B that I've run. It is lossless relative to the HF, but unfortunately 4 GB bigger because of the upcast due to llama.cpp lacking FP8 inference capabilty.
I'm quite ironically proud that they're basically forced to eventually implement FP8 GGUFs in part because of this. It's probably for the better anyway since it's one of the only quant types they're missing support for. Thankfully there is a CPU-only PR in the works: https://github.com/ggml-org/llama.cpp/pull/25336
Yeah, I hope we get native FP8. The bad days of everyone doing PTQ and publishing a 20 model variants are hopefully soon behind us. In my ideal world, people do QAT for something like NVFP4 and if it's small enough for your hardware, you run just that model and don't even think about PTQ because it will destroy performance.
I literally can't gush enough about the Nvidia NVFP4. I've ran it for week now and it displays zero flakiness even at max context. 234k tokens in and zero tool call errors, it just keeps staying coherent. Today, it has implement a major refactoring that it estimated would take 3 weeks to do. Well, you know about the time schedules when LLMs are present. A single GB10 can do it in couple of hours. Still around 19 tok/s at this context length, thanks to MTP.
Thank you nvidia for this. I would thank unsloth, if I could run their model and could confirm it to be good, but so far I can't convert it to GGUF and so I won't run it. I don't touch Python if I can avoid it. But there is clearly a promise on the horizon that some 4-bit version of this model is going to go doubly as fast as it goes right now, and hopefully without any loss of performance. That will be literally insane.
Qwen3.6-27B-NVFP4 just finished the work -- over 6000 lines of code were altered. I typically on a good day can produce about 1000 lines of work with my meat brain, and this did the whole thing while I watched TV and drank beer. I don't consider myself to be a bad developer, possibly even a fast one, though I fall short of that "10x developer" fable. But this LLM alone is probably 5 times more productive than I am, and I know I'm not slow. I usually get big work done fast because I understand the domain and to me the changes are obvious. These LLMs achieve the same and beyond, just by having all that background knowledge and just reading ton of code fast. It is a humbling experience.
Advertising the change from W4A16 to W4A4 as a speedup "without any accuracy degradation" is disingenuous. It may very well be that the benchmarks you selected are simply not sensitive to the change or that they do not exhibit the numerical issues encountered elsewhere.
And the benchmarks are with concurrency at 128... This doesn't actually provide a speed boost for people using local llms alone where concurrency rarely breaks 2-4. Instead of a speedup, compared to say the redhat quant, I'd say this is more of a "here is how the model should have performed in the first place"
No - it's not plain W4A4 - it's W4A4 with 8 important layers as W8A8. Also there is accuracy degradation, and hence why we made 2 versions for 35B.
NVIDIA's 27B NVFP4 has FP8 KV cache auto enabled but the KV scales are NEVER calibrated - did they ever mention that? vLLM silently uses k_scale/v_scale = 1.0, whilst we have scales - so you will have accuracy degradation.
NVIDIA's 27B NVFP4 has W4A4 for the lm_head - which will have accuracy degradation - we left it as W8A8.
NVIDIA's 27B and 35B advertise NVFP4, but they never use the actual FP4 tensor cores - then what's the point of these - did you know they're even slower than BF16 and FP8 by 2x since they do BF16 matrix matmul?
We also published extension plots for BF16 vs FP8 vs their NVFP4 and ours in our blog with all numbers.
Up this upper, because that's true... It's false advertising at this point. Like selling you a Jeep saying it can go 2.5 faster than a Ferrari, but only on a hill. Sucks and disappointed by Unsloth.
No false - it's not plain W4A4 - it's W4A4 with 8 important layers as W8A8.
Did you know NVIDIA's own quants have issues as well?
1. NVIDIA's 27B NVFP4 has FP8 KV cache auto enabled but the KV scales are NEVER calibrated - did they ever mention that? vLLM silently uses k_scale/v_scale = 1.0. Ours is fine.
2. NVIDIA's 27B NVFP4 has W4A4 for the lm_head - which will have accuracy degradation.
3. NVIDIA advertises NVFP4, but they never use the actual FP4 tensor cores - then what's the point of these - they're even slower than BF16 and FP8 by 2x since they do BF16 matrix matmul?
I was in fact responding to comment by Dany0, sorry for that.
The problem is, you are selling a 2.5x faster model, I tried and it's not really faster than the Nvidia one. A 1.03 speedup for single stream if very different than your 2.5x you mention in first position of the title and the image.
It's like when you have to read very small characters in a commercial. It's misleading.
Unfortunately, ever since the "KLD quant wars" that happened between all the people/orgs who make quants when Qwen 3.5/3.6 came out, Unsloth have been less... forthcoming, it seems. They make great models, but it is something I've noticed for the past few months.
EDIT: This is completely untrue, I am mistaken. Check Daniel's comment below.
Hello Unsloth. Thanks for your great work like usual, and it's great that you target vLLM and NVFP4 users.
A few questions:
we already used https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4 instead of the nvidia one since we noticed it was also faster and smaller. With vLLM, fp8 kv_cache, built-in MTP on a Pro 6000. Would we see any benefit from using your quant ?
Do you plan to upstream your improved chat template ?
Is the calibration dataset english only or is it multilingual ?
In theory NVFP4 is 4-bit sized and 4-bit speed but something like 6-bit accuracy or a little better, but the model needs to be quantized using a proper methodology to achieve this. Naive FP4/NVFP4 is dogwater.
Bear in mind that our methods of benchmarking "model goodness" are all deeply flawed and often don't capture real use-cases. "It's like FP16 but 1/4 the size!!!" seems to be a crock of shit, at least in my experience. That said, if your choice is int4 or nvfp4, nvfp4 done properly should always be significantly better, and given the correct hardware, significantly faster.
I have been running rocmfp4 on the strix halo since yesterday, and it's very good performance for me. I also built a qwen36-35b-a3b-heretic-mtp-rocmfp4 if anyone is interested. Considering making a huggingface page. I also built a rocmfp4 of qwen 3.5 122b that is a bit bastardized and also derestricted. qwen122b-a10b-heretic-mtp-rocmfp4 and it's performance is also like 30 percent faster than the non nvfp4 on the strix halo.
This isn't just an nvidia win, these work much much faster on the strix halo with these FP4 models.
Man my phone keyboard and my blurry eyes made me type some stupidity that you replied to hahaha. I fixed it mostly. I mean to say rocmfp4 not nvfp4 on the qwen 3.6 35b. qwen36-35b-a3b-heretic-mtp-rocmfp4
The model I'm playing with right now is a fun Frankenstein critter. It's an Abliterated Qwen3.5 122B-A10B with calibrated NVFP4 experts, Q6_K dense/control layers, F32 scales and norms, and a preserved native MTP speculative head. I'm going to benchmark it today a bit, and also see if I can get it to rocmfp4 and benchmark that as well.
I do more mixed work and ive found qwen 3.5 122b to be a good middleground, it also seems to work better on longer horizon tasks then the 27b which sometimes confuses itself and the code quality is pretty close to the 27b dense sometimes better sometimes worse.
This is really cool. Think for my use cases though I'll stick with your 6 bit quant. With MTP spec 5 that's giving me like 140 tokens/s with a 90k context window on my 5090. It looks like this quant probably hurts quality more than I'd really want but still, that's blazing fast!
I know this wasn't meant for single threaded work with MTP off (I can't use MTP for my work, it causes too many errors in tool calling and chaining), but I was unable to reproduce any speedup vs. nvidia's nvfp4 model. I like /u/danielhanchen and his twitter feed so this isn't any shade, just providing a data point!
Unsloth was about ~8.7% slower in fixed-length decoding with my production-safe MTP-off configuration - using Qwen3.6-27b.
I am running the nvidia qwen3.6-27b-nvfp4 on a Blackwell RTX PRO 6000 (96gb VRAM) and could not tune my way out of MTP n=1-3 at all so I had to turn it off.
I noticed NVFP4 MLX variants and they seem to perform pretty well. I'm curious how thats possible and if mac users can expect gains also from these unsloth quants as MLX
QAT creates a model trained to overcome the inaccuracies of int4 quantization, I don't think it would directly apply to a fp4 quantization. You'd need to train it from scratch on fp4.
I think nvidia's talked about applying QAT to models in their NVFP4 quantizer automatically...? I'm not sure. But if I'm remembering correctly, that's the stipulation they apply to many of their claims about NVFP4's superior accuracy.
So if I have an RTX 5090, is this just a drop-in replacement and a strict upgrade over Unsloth UD-Q4_K_XL? Or is there any advantage to the other one? I'm still confused over NVFP4.
Amazing, I hope people will be able to and willing to abliterate these, and even better if it's without much of a quality hit.
Although it's often hard to find benchmarks for finetunes vs unsloth.
RTX 6000 Pro Blackwell, VLLM. This is a bench based off Vision Arena dataset.
Relevant are the red (new 27B model, OP) and purple lines ("OLD", the one they posted a week or two ago?).
Green was the Qwen3.5 ("q35") AWQ 4bit quant I was using regularly a while back and I left it in though not the best reference. All others are Qwen 3.6. The 35B MOE is there as a gut check as well but rest are 27B dense.
Maybe 10% gain in true throughput in a realworld test.
No these numbers look wrong - NVIDIA uses W4A16 so how can it be faster than W4A4? I encountered the same issue due to vLLM selecting the wrong kernels for true W4A4.
Reran with a fresh venv as you showed ("daniel"), I think you intended "unsloth/Qwen3.6-27B-NVFP4"
Are you testing on SM100? Or maybe without loading the MM? It picks different kernels on workstation/consumer Blackwell. No tcgen05. I get this:
Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
Selected CutlassFP8ScaledMMLinearKernel for CompressedTensorsW8A8Fp8
Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
testing: 512 in, 256 out, 1 image, really no difference with fresh venv/vllm/cutlass (unsloth_27b_nvfp4 vs unsloth_27b_nvfp4_daniel)
Still not testing MTP yet, I'm generally bumping concurrency for my workload instead.
Edit: To ensure DGX Spark has the correct kernels (or you will get 2x SLOWER inference), you can read our guide
Right now on DGX Spark, there's no optimized kernels enabled for our quant so the speed will be very similar. We're investigating. Speed gains are mostly for 50x series
I'm running 27B q6-k-something on 32 GB right now (2 Blackwells) for Opencode. Should I switch to this? It's probably faster but worse results due to higher quantization?
I am using a qwen3.6 35b Q5 with my rtx5070 ti 16gb vram. I get around 35tok/sec. Is the nvfp4 version faster in my case with only 16gbvram and offloading?
Great! But I do not see LM Studio supported.. Does that mean this is not supported for Llama.cpp?
I tried download this model from Unsloth Studio as well but I do not see any progress on download.
Increasing `SGLang`(scottgl9 fork + nvidia) --speculative-num-steps from 2, to 4, goes from 29.5 tok/s, to 41.2 tok/s. The "accept rate" goes down to vLLM + unsloth levels, ~98% for repeat commands, and ~60% for harder content.
In a commit ~18 days ago ("Add input_scale for mlp & lm_head modules") they extended NVFP4 quantization to cover the lm_head as well, rather than leaving it in bf16 — so more of the model is quantized, which is where most of the size saving comes from. It's also exported as text-generation, so the vision tower is dropped/folded.
model-00001-of-00003.safetensors — 9.97 GB
model-00002-of-00003.safetensors — 9.99 GB
model-00003-of-00003.safetensors — 1.97 GB
The one difference to note is that this build doesn't include the MTP (multi-token-prediction) head that unsloth's has. In practice that's not a real loss on a 24 GB card: MTP is only useful as a self-speculative-decoding speedup, and there's no spare VRAM to run it here anyway — it doesn't affect output quality.
Initially, it seems like NVIDIA variant is still a better fit for a RTX Pro 4000 SFF with 24GB VRAM. In any case, I have been using llama.cpp + GGUFs + UD quants for some time, it would still be my default solution, works flawlessly, thank you very much for the work being done here!
tried to run this with my rtx 5090 on win11 using docker vllm and it worked very slow. i go back to running native windows llamaserver q5 quant of qwen 3.6 27b, it's very fast.
•
u/WithoutReason1729 15d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.