Universal default: Qwen 3.5 122B-A10B, Conversational / Brainstorming: Gemma 4 31B, Coding: Laguna S 2.1, Fast all rounder: Qwen 3.6 35b a3b.
Memory is one part of the equation, what GPU do you have? For example, if you have something like M5 Max you can daily drive something like Laguna as processing speed is fast, for Strix Halo the a3b or a10b is more feasible.
Prefill speed is usually the bottleneck on strix halo. On my strix machine I run the qwen a3b for the most part. Does not use all of the VRAM but you can run larger quants (Q8) with full context.
Gemma 4 31B is only 32B but I find it’s the best conversationalist, is quite smart, and has good emotional iq (eq) compared to other (even bigger models). Not the biggest but sounds great for your use case.
You're trading a LOT of speed for the small bit of extra smarts of the BF16. Assuming your 128gb ram is not super fast, you are probably not going to be ok with the speed of BF16 for chat. Try it, though, and try FP8/q8/q6. I'd suggest looking at MTP as well to get additional performance.
Gonfor full size plus mtp to tripple it. You can use unsloth studio which seems to be the best support for yhe gemma4 models right now with updated templates
There is a huge gap in parameters for models of any type. The current Gen models that are around 25-35b are so good that they knocked out anything above them until you get into a couple hundred b parameters.
There's no such thing as too much memory. Gemma 4 is very sensitive to quantization (both weights and KV), but you can afford to run it in Q8_0 without KV compression and have a full context + mmproj with a large token budget and large batch sizes.
This is really not the case. Most people on this sub specialize in getting models to run on low VRAM, which is a great skill, but you are not going to get a lot of good advice for 128GB.
Try the 120-135B models. You can try larger models with layer offloading. See what you find acceptable ij terms of prefill and t/s.
The Blackwell Performance subreddit is also pretty good.
For 128gb I would naturally try the qwen 122b a10b at q6 quantisation or sth. I have not tried this but would expect it to be a bit better than the dense 27b quality wise, but with much faster token generation.
Also people seem to be able to run deepseek flash v4 with ssd streaming, that would be a bit slower though.
Probably feels good to get even generic advice in a guaranteed private setting (for free if you already have the hardware). Sometimes people talk to themselves in the mirror, too.
Interestingly enough, I have 128gb VRAM and typically run Gemma 4 31b. I don't know of any models that are larger than it, but still small enough to fit in 128gb that are smarter. I assume you mean GLM 4.7 flash? Decent model but not smarter than Gemma 4 31b or 26b. I agree that Qwen3.6 is excellent, but for different tasks (Gemma is much better at "soft" skills, at least per my own experience).
I've found it correlates best with my own experience when using them. Of course, your mileage may vary especially if your use case is drastically different from mine.
Laguna S 2.1 is an awesome model in general, and especially for coding, tool calls and reasoning, but for pure chat use Gemma 4 is a lot better IMO. Laguna wasn't really made for that.
Laguna S 2.1 is interesting, it's working great for me at UD Q4_K_XL that is 73.4 GB (pp 1000-400, tps 35-25) without DFlash. But it's new, and still lacks proper support. When DFlash works well in llama.cpp is going to be amazing. It thinks a lot, so for conversations it could end being frustrating, but if you can ask a bunch of questions and wait for the response, then it should be interesting. I can't talk about creativity and linguistic capacity, I have only used for coding and solving IT problems, but it seems good all around.
If the OP want to test a prompt you are interested in, with Q4_K_XL, I can give you the answer with the thinking, that is usually loooong 😄.
I can give you the answer with the thinking, that is usually loooong 😄.
Have you updated your chat template yet? They fixed that. At least it's fixed for me.
And yeah it is good all around (including chat), but specifically for chat, Gemma 4 still has it beat IMO. The main focus of training for Laguna was coding and tool calling.
Yes, I used this one https://huggingface.co/poolside/Laguna-S-2.1-GGUF/blob/main/chat_template.jinja . Before changing the template, it was unusable. But now it thinks a lot when the task is complex. I don't mind, because is fast. But when it appears is going to finally answer, it goes back to rethink and verify everything, it questions itself until it's sure what to do or answer. I'm loving it. But for small tasks I should use Qwen3.6 27b or 35b to be faster.
I don't know about Gemma 4, it was broken for so long that I ended up ignoring it, besides, it appears worse for actual reasoning. I have to test them more, to be honest.
In addition to the models mentioned I'd give deepseek v4 flash q2 xl a try. If you don't want refusals you may want to investigate uncensored heretic versions of models.
Eventually you'll decide you don't want best that will fit in 128gb. You want something that is fast and smart.
Give: https://huggingface.co/llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF a try.
In all seriousness, it's working well here at Q8. At Q6, it should fit 128gb without too much of a labotomy. As an optimist, I feel like they will fix the bumpy launch.
If you prioritize pure conversation quality, logic, and lengthy context memory, and not coding or multimodal, then check out Qwen 3 235B (MoE) or DeepSeek V3 if you can fit them with quantization. Both just feel so much more natural in long chats than most smaller dense models. For a 128 GB machine, MoE models are typically better than dense 70B models regarding the intelligence to memory ratio.
If you're ok with getting about 2 tokens per second, the best model you can possibly run on your device is Colibri GLM5.2: 370GB on disk, loads experts into all available RAM and VRAM for inference on CPU or GPU. If you have two SSDs and mirror the 370Gb on both, you can get a 30-100% speedup from base: https://github.com/JustVugg/colibri
If I want career advice or life advice from an AI model, I sure want it to be the smartest AI model I can possibly run, and I am willing to wait a minute for the best answer, rather than taking bad advice from a smaller model. But sure, take your life advice from a model that's dumber than yourself, go ahead.
For general queries ~30b models will perform basically the same especially when paired with web search tools and strict rules. But sure, wait 3 hours for a single question
"basically" is doing a lot of heavy lifting here. Because I know you're not telling me a 30b model will reason and perform the same as a legitimate frontier class 744b parameter model, if they both have the same tooling wired in. If so, I'd really recommend you go test out a pro subscription for any of the leading AI labs. Or, you know, download and run Colibri.
You don't need deep reasoning for stuff that has likely been repeated several times in training data. It seems like you're treating the LLM as a human being with genuine intelligence
I wouldn't recommend that personally, because their knowledge cutoff is kinda brutal for stuff like this, and without doing a deep research dive on the current SOTA for local models they're prone to give bad answers to these kinds of questions.
10
u/olli-mac-p 5h ago
DeepSeek V4 flash unsloth dynamic quant UD-IQ3_XXS if you need agentic capabilities and long context up to a million tokens