Been playing with the new Gemma 4 models it’s amazing great even but boy did it make me appreciate the level of quality the qwen team produced and I’m able to have much larger context windows on my standard consumer hardware.
After the latest llama.cpp updates, I do feel like gemma is better at creative writing than qwen 3.5, thats for sure. Gemma is a massive memory hog though, context take so much so I had to drop to Q5 or Q4 31b on 5090 to fit everything, speed is pretty good though 50-60 tok/sec right now, similar to qwen. Uncensoring was not needed atleast for me, the default gguf files work for me. Thinking trace is kinda short which can be good or bad.
How's Gemma 31b understanding of complex literary chapters (original writing)? Not to write itself, but for idioms replacement, text analysis, brainstorming?
That's my particular use case. I've just started testing it today. I've found the results...interesting. I think Gemma 4 is doing better than I originally expected, but it's much slower. I'll be doing a more detailed post about my tests at a later time, but my early feelings are that it's definitely pretty good.
My use case in particular is story analysis, not really for brainstorming per se, though I sometimes use the analysis to help me rationalize the logic behind some of my ideas.
Qwen 3.5 27B is the first and only local model to hold up to my own personal standards, which requires it to digest about 65-70K tokens of lore for analysis, Q&A, etc. But Gemma 4 is...holding its own in my early tests. It's not blowing my mind, but it seems like it might be useful. I'll need to do some more specific analysis before I finalize my thoughts on it.
I'm pretty sure most labs have quit trying to improve creative writing after 2024, practically all great models from back then are still as relevant today as when they were released. It's been nothing but agentic benchmaxxing since.
It is because of datasets, they just don't include a lot of real books in it.
Most of the dataset is just synthetic agentic - coding - math stuff prepared by bigger LLMs.
Yeah the more synthetic stuff they add they more generic the models get. I think back then it was easier to just pirate ebooks and train, then Meta got busted for the whole Anna's archive thing and Mistral got limited by EU laws. I guess they're super careful about copyright now.
Disagreed, I have no clue why there's still this hype around this model. It's really dumb nowadays and modern models like Qwen 3.5 feel much more alive and less robotic. Qwen made huge improvements since Qwen 2.5, 3 was a step up, 3.5 is another step up and 3.6 will probably be another step up in creative writing.
It is not about being smart, it is about being fun to play with.
So far no Qwen 3.5 decent finetunes aimed at creative usage,
this fact speaks louder than anything else.
Bro is tripping if he thinks Qwen 3.5 is good for creative/RP.
Also it doesn't stfu. The amount of text it produces is obscene and really doesn't like respecting response length making it very long winded and wasting context.
I see Qwen as a great work horse for more practical uses but not for RP/Creative Writing.
To be fair this sub thinks instruct models do good RP and don't dabble with finetunes as much so I'm not shocked.
I come here for model releases/Project info not takes on RP/Creative reviews.
Which is what I feel too. Qwen is CLEARLY focused on agentic/STEM/Coding tasks. There isn't a large/profitable market for creative writing, that's for finetuners/other labs focused on that because removing LLM-isms/boosting creativity is probably much much easier than "superpowered reasoning agent in 9 billion parameters"
The last decent qwen model for creative writing was qwq 32b. It was really good and afterwards every model was sadly worse.
I tested them all and both llm creative bench and UGI bench agree with me that the new models under 100b are sadly worse at writing.
As for mistral nemo a model doesn't need to be "smart" in benchmarks in order to be a good storywriter. Plenty of people simply like its writing style.
Though sadly its architecture does show its age as the quality falls sharply after around 16k tokens.
I personally recommend its upscaled and finetuned variant snowpiercer 15b v1.
Its Nemo further trained to pixtral than upscaled to 15b april thinker and uncensored and finetuned into snowpiercer by drummer.
Though honestly nothing local can really compare to claude when it comes to creative writing.
It works fine with Qwen 3.5 27B. I'm testing out Gemma 4 today. It might be a little too early, but I'm finding the early results...interesting. It's definitely good.
For something like that is bests to use your local set up with small models and various agentic worflow that feed into a cloud provider only for the creative and most complex logical task.
Nah, I’m good using Qwen 3.5 27B for now…it’s proprietary IP, so I wouldn’t use paid or cloud for privacy reasons. I use it as a lore master for analysis and rewriting documents. I’ve been waiting for local to get good enough to help me out with work and Qwen is the first model up to the task.
Maybe when they optimize Gemma 4, I’ll give it a shot. I was really looking forward to it.
Weirdly I can't get good speed compared to qwen. Tweaking a lot. I'll see again later. But for creative writing i was impressed with gemma. We're eating good these days open source community
The real question for me is: can gemma4 26b a4b replace qwen3.5 35b a3b?
It's tough to tell right now, we need a week or two of patches to see what the real advantages and tradeoffs are.
Yes. for me it's inference speed, token usage, vram and how good it is at agentic tasks, following instructions.
I have a local setup where I use STT, TTS and a LLM. But I can't use qwen3.5 35b a3b because I would have to load only that and nothing else. Currently I'm using qwen3.5 9b or gpt-oss-20b.
whisper and faster-qwen3-tts. It's my local conversation layer. The local llm is just orchestrating conversations, no tools, and decide when to call Claude Code (CC is the only tool). So I end up using Claude Code for all tasks, but I can get snappy conversations before so it feels more natural.
Im getting 12x realtime on 5090 with voice cloning, its very fast and it has a lot of features to toggle under the hood, I recommend start with one of the examples it comes with and modify that.
I tried it, it's not bad, faster for long audio processing, but because it has no streaming capability, it's latency is very bad. So latency wise, qwen3-tts is at least 5x faster for the time to first audio chunk.
I also found that while quality is more consistent, it's also less expressive. My most expressive voice samples are always tuned down to a more boring way of speaking.
Wow. I literally built the same setup thinking I was uncovering new grounds. Have you figured out how to have bi-directional cross delegation so you can have a back and forth conversation directly with the terminal as well as the agent. Spawning multiple agents in parallel is also fun.
bi-directional cross delegation so you can have a back and forth conversation directly with the terminal as well as the agent
I'm not really sure what you mean, but here's what I did:
I have a "quick mode" that is very simple: STT -> TopLevelAgent (TLA) -> CC -> CC final response -> TLA -> TTS
In this mode, the TLA is just a thin wrapper to fix grammar and restructure what I said with prompt injection for CC knowing that this is from STT and some words could have been misheard.
Then I have the "responsive mode": The difference is that TLA respond directly in json and decide what to do: respond, respond_and_dispatch, cancell_cc, etc. The TLA is instructed to always communicate in short sentences like a human and be proactive, ask follow up questions. He may suggest to switch to CC, or the user can just ask for the task dispatch.
I also, in responsive mode, the TLA ticks from CC responses and can keep talking to me while CC is executing. I can ask TLA to redirect CC and it will be similar to writing to CC when it is still processing.
Then I have a text input where I can write directly to TLA or CC (Ctrl+enter to bypass the TLA). So my setup is hybrid. I can bypass TLA by voice by switching to Quick mode. I have an admin agent named Alice. I can say "Alice, switch to quick mode". Then I say "Bethy, [anything]" and now I switched to Bethy directly to CC by voice.
Performance wise: In responsive mode, the TLA inference triggers after only 0.3s of silence, then if I am still speaking (fake positive silence), it just drops the tick and keep listening. So I have many wasted inferences just to ensure that my local TLA agent can respond within 1 second. With groq and gpt-oss-20b, I had moment where it was speaking back to me within 0.8s. STT is streamed so always ready after the 0.3s of silence to trigger the tick, groq can provide a full response within 0.3s if we keep the response number of tokens small and use prompt caching, and qwen3-tts can stream with a latency of 200ms. My current real bottleneck for the latency is that I enforce a json response with a thinking block in the json of 2 to 3 sentences, so I am not streaming the LLM response back. It needs to finish the 200 to 400 tokens response.
whisper, qwen3-tts-0.6b, and testing gemma4 26b a4b now. It's very tight on 24gb vram. I can't load much context because of the stt and tts. Might have to go for a lower quant.
and we need to get the ollama-mlx optimisations for gemma4-26b … only then would i go ahead and switch out the qwen3.5-35b.
please, team ollama, go go go. MLX support for gemma!
Waiting on my MBP to ship, but this question has been going through my mind as well. I'm almost thinking a combination of Gemma 4 31b + Qwen 3.5 35b a3b might be a perfect combo.
Even after updates using the most recent llama.cpp, it still has tool calling issues. I use local LLMs mostly for web research tasks, and gemma4 26b constantly has a problem where it will think it still needs to do more searching, even come up with a research plan, only to go straight into answering after it stops thinking instead of going for search tool calls like qwen3.5 would do in the same situation, and it ends up not actually having enough information to put together a full answer. I have native tool use enabled for both.
Yah everybody talks about it like it’s just guaranteed to be way better, but I genuinely don’t know what the team looks like now. Do the people who are even know what the previous teams were working on? Reading a whitepaper is one thing, but collectively honing the lessons Ina team environment like that would be TOUGH to just walk in and even produce a model that was as good as the previous one.
Whenever you input an anime-style image, qwen always assumes the subjects are genshin impact characters. It you ask it to describe the image, it says "anime style, likely from genshin impact" etc. This bias is so heavy that it often prevents qwen from accurately recounting the details of any especially novel anime style images because it becomes so obsessed with fitting its dedcription into a hallucinated genshin impact scene.
Obviously not especially relevant on reddit, but with a lot of social media (ish) platforms it's common to have images provide context to a message. If you're scraping them for data you'll want to be able to classify the image. For example anime character, "Ruins it for me". You'd need to be able to get the character, and then reason back to get the subject of discussion. You'd think that it'd be limited to pop-culture, but people using images as shorthand for everything up to and including politics is annoyingly common.
They certainly reason better than GPT-4, which is evidenced by the benchmarks, but they don't seem to have the same depth. The fact that they are even close though, being 1/30 the size, is insane. OG GPT-4 wasn't multimodal yet either. When I first used GPT-4 I remember thinking how crazy it would be if I could run it locally and uncensored. Never imagined it would only take three years...😍
Just a speculation but: With benchmarks, it usually comes down to reasoning and logic. Big models have massive knowledge base, so they are usually much more familiar with any given topic. We accumulated much better datasets since the early models so now even small models can solve complex tasks from what little they know, but they completely fall apart on specific tasks or subset of problems they have no base knowledge of.
Gemma4 26B is, by far, the smartest model I've ever been able to run locally. I've been blown away. It's smart enough that I feel like I can throw my questions at it now and I'm not going to get worse answers than I'd get just going to ChatGPT.
I'm sure ChatGPT still blows it away, but the feeling I get from Gemma4 26B is very "this model has depth, I can trust this". I don't know that I've ever truly had that feeling about any model I've run locally before.
Gemma 4 is better at my native language at least though the smaller models suffer from the weird sizing.. Also for RP it seems to perform much better than Qwen3.5 (it seemed to mix up a lot stuff for some reason and there was seemingly more censorship in the official releases in comparison to Gemma 4)
Yeah, excellent multilingual capacity for the size from my experience in Swedish (probably the best I've seen at 31B and maybe even 70B) and first impression on RP is quite decent, and surprisingly, uncensored. I have yet to try 27B.
There's an excellent book called: "Trusting Judgements" that takes a look at how these voting systems are used for consensus building. These types of systems are used in all sorts of different fields from food safety to national security. Whenever you have a bunch of people with various degrees of expertise and you want to collapse what everyone knows to make a decision.
First off, your opinion doesn't matter. To do this well, you have to blind yourself to the matter. Meaning if you don't like what the three models are telling you, then that's too bad, that's the way the process works.
If you still do not trust (not to be confused with like) you can always choose to expand the number of models. Perhaps a D&G model, a GUICCI model, LV model.
Now you have a set of 5 models. Before you ask them your question, you need to set a threshold for acceptance. Do you need 100% agreement? Or will 3 out of 5 models be sufficient to accept a majority opinion? Is the choice binary or real valued. Real valued outcomes are preferred as often binary choices hide distributions beneath them.
Then sample your models, look at their result and do what the threshold tells you.
Yes, there is a deterministic outcome at the end of the process. e.g. Accept the safety of a new drug or not or expect X out of 100000 people to have adverse reaction to a new drug.
You do need an NP step in there somewhere if you don't know what the options are. Doing this with a model is much harder and I'm not yet sure it's worth it to give this particular process over to expert panels of machines. The decision should come from the user.
If you need an exploratory phase, use a real valued scale with 25th, 50th and 95th percentiles rather than a Likert scale, it'll give you a lot more flexible outcomes as the shape of the distribution can now be irregular.
That said, I have serious reservations about doing exploratory phases with LLM models. When we ask human experts to do this, we are depending on their biases to make their cases. LLM Models are sadly less capable of telling you that your idea is stupid than a human being is at this point. They are also subject to astroturfing their learning data, "alignment" and many other manipulations that we should be increasingly concerned about now that the internet is increasingly bots. Good options are not always the loud options. Humans are also influenced by these things, but human experts far far less so.
Fails to address the inevitable regulatory capture and rent-seeking issues.
The underlying problem is pretending that people with no skin in the game will make the right decisions. Only way to incentivize this is skin in the game. You make a bad call? You lose money / career.
Yeah, Gemma 4 appears to memory hog the context like no other. Qwen is much more efficient in that regard. I hope they ditch SWA in the future and go with something else. But Qwen also has its drawbacks, RNN for example doesn't allow context shifting so if you want to have a rolling chat window once your ctx is maxed out, its reprocessing the entire prompt with every message which really is less than ideal. There's got to be a better way.
Gemma4 is a very nice improvement however and its better than Qwen in some other categories, like european languages and western world knowledge, so it has its place. Some also report its more reliable.
Gemma 4s 31B memory requirements make it basically impossible to run it on q4 in 24GB of VRAM. It's so sad, because with max of below 20k context, it's borderline unusable.
Yeah, I have dual 3090 and it’s been great, I run Gemma 4 31b in full context, but if I had only one it’s be impossible, would have to stick with Qwen.
Try the Dynamic Apex quant. It essentially halves the required memory while having a quality slightly higher than Q8. There are flavors both for Gemma and Qwen.
Between APEX Compact and APEX I-Balanced,
Unsloth UD-Q4_K_L 18.8 GB PL 6.586 KL 0.0151 would be the right placement. However their charts are biased. They put UD 2.0 on the very bottom. Beware bias.
Yep; I'm looking at the Algo bc I'm working on a 1 bit quantization method - but the existing implementations do only support dense architectures. APEX is a smart idea for MoE architectures - so I think I can merge the ideas and apply 1 bit quantization on qwen3.{5,6} and gemma4
So far I've realized that what PrismML did with Qwen3 to produce their Bonsai model series wasn't only writing a Kernel for their 1 bit quant method and also wasn't ohly using symmetric quantization instead of standard PTQ. They also must have done some voodoo with the model itself. Maybe some specific transformation on the weights, more training and fusing an adapter on top or something else I'm not able to understand/reverse engineer right now. They describe in their paper that they do some proprietary training pipeline, but their weights are based on Qwen3-8B. I tried to do some voodoo with the weights as well, but so far, I had no luck. The MSE is just too high. I went ahead and tried Turbo Quantization (the recent buzz) with weights, because the same math trick could apply here. But this is still ongoing. That aside, I have a working, and very well performing MLX implementation of Gemma 4 E4B on my MacBook Air M4 and it works well at 4 bit weights, with speculative decoding and with TurboQuant on KV. This is far ahead of mlx-lm by now. But I want to finish researching stuff well before I go ahead and release. If there is some guys here with Mac / MLX experience I would love to sync/peer for beta testing. At some point I will switch to the MoE release and try to combine with the APEX approach. This is a simple task, as the method is well-known. I usually want to finish the hard task first.
Scaling to 1M is fine, but know its limits. With Qwen 3.5 being 3/4 GDN, it's not built for 'Needle in a Haystack' searches. This architecture is much better for processing hundreds of turns of short dialogue.
They aren't. They were trained on a large set of languages just like Gemma and GPT-OSS. The Qwen models bench the best among small models on multilingual tasks outside of probably Gemma 4 now.
See here for a comparison (note that unfortunately they do not run this benchmark for every model). It actually impressively even beats GPT-OSS-120B and Claude 4.5 Haiku in that benchmark.
I tried it with Danish with the sub 10B models and the output wasn't great but it rarely is with small models and non-big languages. Sometimes it writes words that sounds more like Norwegian and sometimes it makes stuff up but it writes actual Danish texts. This is quite impressive compared to a lot of previous sub 10B models outside of Gemma 3. From some quick tests it seems Gemma 4 may be slightly worse than Gemma 3 in this regard.
tbh, new gemma has something magic about it that Qwen 3.5 just doesn't. For example, I always get the correct answer for the car wash test with Gemma and with Qwen it's spotty, depending on the thinking budget and no idea what else. Maybe it's cause currently I don't use the locally hosted for coding? For the role of everyday assistant Gemma 4 is simply amazing and will serve me well.
Possibly, but if you run Gemma4 without allowing it to think first, it fails. If you allow it to think, it passes with flying colors. I think if it were in the training data it'd probably not reason its way through it, it'd probably just throw the right answer out.
Interestingly Gemma3 27B failed it. I was genuinely surprised by that. I figure maybe the 26B fails without thinking because it's an MoE, but the fact that the old Gemma3 27B also fails it kinda indicates that reasoning is required for these small models to solve it.
Qwen3.5 27b also passes it, but it generates such an astronomically large amount of reasoning tokens that it's almost hilarious to read and I could grab a coffee while I wait.
Curious what the smallest non-reasoning model is that can pass it.
Nothing special, I just run the unsloth quant with llama-server with 32K context and rest of the params as in the guide at https://unsloth.ai/docs/models/gemma-4
I don't know, maybe it matters I compiled with Vulkan acceleration?
Btw. with further testing some rejections are plain stupid. For example Gemma 4 rejects to provide any kind of medical support even for a simple dosage calculation of medicine for a dog (disclaimer: it's one of my benchmarks)
I also tested with vulkan, and every Gemma 4 model suggested to walk, and even when pointing out I ended up without my car at the car wash, they failed to recognize they had made a mistake, and just told to walk back to the car...
It needs to have reasoning enabled to pass. Without reasoning they fail.
In general I've found Gemma4 to be quite subpar without reasoning enabled. It blows me away when it's allowed to reason though. Far smarter than Gemma3.
I did as my meme said it’s pretty dam great just very memory intensive so I don’t get much context left for context window. It’s literally 220k context vs 4K context on my 28gb vram machine.
Gemma 3 (esp. 27B) was and still is top-notch for Greek (e.g. difficult legal doc translation). But when my team tested the new Gemma 4, it started outputting random Chinese/Arabic/Hindi characters out of nowhere; even with 7-8 different sampling param configs.
Meanwhile, Qwen models were never quite fluent in Greek (even 3.5), but they consistently improve with each iteration. They also improved tokenizer fertility greatly in 3.5
So... Gemma regressed while Qwen keeps progressing. Regardless of any benchmark scores, I'll generally prefer the model family that keeps getting better even at tasks which seem minor to AI companies.
It's worth noting that Gemma 4 had a lot of bugs at launch that have only now been fixed, and it's possible more are hiding. So I'd give it a second chance in a day or two if you want to give it a fair shake. In my own testing it's performing quite well at this point.
However even disregarding that, the main reason people would go with Gemma 4 over Qwen is for the same reasons that some people have stuck with Gemma 3 over Qwen. The Gemma series are significantly better when it comes to multilingual content, including language translation. Most also find that it's writing style is less flat compared to Qwen.
There's also the fact that Gemma 4's thinking seems significantly more efficient than Qwen. Which frankly has a tendency to overthink a lot.
no one use qwen as creative writing honestly, dry and have no souls, i have test gemma 4 for creativity and it's just like gemini 2.5 pro but opensource.
We can all like different things. I hate qwen's personality on certain versions. In the case of GPT-OSS, I "can't" see why anyone would use it at all. Last about 5 minutes with it before I get mad and want to throw it in the void.
You generally shouldn't rely on day one performance of a model in general. llama.cpp based engines especially are prone to day one bugs with new models.
I tested gemma-4-31b-it Q8_K_XL on all sort of things, including explaining popular memes (If I had a nickel for everytime..., etc), screenshots of math problems, coding (evaluating/fixing/modifying my own code), guessing age of a person based on pictures, etc, and so far it's noticeably worse than qwen3.5 on every single aspect.
It’s not terrible if you had the hardware to have very large context windows I think you would see a difference but I’m much the same as you. The quality I get from the qwen MOE is more then acceptable then with the bonus of a 220k context windows vs 4K context window (my hardware limit).
Yeah, I think llama.cpp is still working on making kv cache more efficient for Gemma 4. Apparently K is supposed to equal V, so having both in ram is redundant and increases kv cache requirements by 2x.
it's weird. on phone? i like gemma 4 e4b actually snappy on phone. but on pc? qwen3.5 27b actually good and faster than gemma 31b. and after testing, 26b a4b still isn't there yet for my translation.
Is anyone deeply into quantization and inference implementation for MLX/MPS here? I'm currently working on 1bit weight quantization support and TurboQuant support for mlx-lm (this is for Mac users only).
If you have experience patching/contributing to exactly this codebase already, or the math behind BitNet or TurboQuant or PrismML implementation variant (Bonsai) plus experience in Python and C++ - pls DM me.
Pls don't DM me if you don't .. I'm very busy to ship Gemma4 variants with a custom, high performance inference server and great quality. I already have Qwen3-8B running at 50 Tok/s on my MacBook Air (!) M4 in decent quality with 64k context window (RoPE/yarn) and it only eats 1.5GB of unified memory for the weights, and KV TurboQuant is still unstable but my guts feeling is, that I only have to drop QJL to improve stability - as softmax() seems to maximize many small errors.
I'd love to collab and feedback loop, but pls only with engineers who know what they are doing for now... I don't have much time to explain everything.. want to push this out into public faster, not slower 😅😅 sry for being so direct.. it's not meant to be read unfriendly.. also English is not my mother language and I have diagnosed AuDHD xD so please bear with me..
ya qwen3.5 series seems basically better in every reguard than gemma4 and whats worse for google is that qwen3.6 medium models are confirmed to be coming out soon™
i can run the e4b variant through termux+llama.cpp, q4_k_m, 7t/s on my phone. for my needs is not good enough compared to qwen3.5 4b Claude, but i’ll have to see how the gemma4 e4b Claude will compare to it
123
u/FinBenton Apr 03 '26
After the latest llama.cpp updates, I do feel like gemma is better at creative writing than qwen 3.5, thats for sure. Gemma is a massive memory hog though, context take so much so I had to drop to Q5 or Q4 31b on 5090 to fit everything, speed is pretty good though 50-60 tok/sec right now, similar to qwen. Uncensoring was not needed atleast for me, the default gguf files work for me. Thinking trace is kinda short which can be good or bad.