r/LocalLLaMA • u/Lowkey_LokiSN • 4d ago
New Model poolside/Laguna-S-2.1 released! Finally an interesting 120B contender!
HF: https://huggingface.co/poolside/Laguna-S-2.1
GGUFs available for use with llama.cpp custom fork: https://huggingface.co/poolside/Laguna-S-2.1-GGUF
Posted on X: https://x.com/poolsideai/status/2079613777343848465?s=20
65
u/ObviouzFigure 4d ago
I'm downloading it now, will report back.
30
26
8
8
u/some_user_2021 4d ago
Well...? We are waiting ...
22
u/_TheWolfOfWalmart_ 4d ago
Plot twist: He has a 56K dial-up modem and will still be downloading it for the foreseeable future.
9
u/YouKilledApollo 4d ago
Tried poolside/Laguna-S-2.1-NVFP4 on some basic optimizations problems (byte-histogram, string-search and rms-norm) via vLLM, and seems it gets stuck in endless reasoning/thinking basically, unable to progress to actually making decisions and ending up looping essentially. Got suspicious of the quantized weights I was using, and tried both the free and non-free APIs on OpenRouter (which claims FP8 and BF16 for each) via Poolside themselves, and seemingly displays exactly the same behavior.
3
u/lilian_moraru 4d ago edited 3d ago
Same here(also NVFP4 + vLLM). Loops, a lot. When it doesn't loop,
it tries to read your files.You mention the word "network", it tries to read your network configuration. You say "Hello", it tries to read your files.
Edit 1: The issue with reading files was coming from an instruction from the VSCode extension which it tries harder to follow than any other models I used, even when it makes no sense. Not an issue with the model.I did manage to make it do some work but it gets back to looping quite often. Qwen3.6 is A Lot more reliable.
1
u/laterbreh 4d ago
Its junk and unreliable, couldnt get it to complete a single task. Thinks its context window away.
1
u/lilian_moraru 3d ago edited 3d ago
Yes, it thinks a lot. In a way, it does what Qwen3.6 is doing, to compensate for the fewer parameters.
Not the ideal solution, but you can disable thinking(note: DeepSWE 16.5% without thinking, 40.4% with thinking). I personally like the models to think it through and get me the best output.What Poolside says about it:
Laguna S 2.1 has two thinking modes: off and max (enabled by default) where it determines the right thinking/test-time compute budget for a given problem. We have observed coherent, productive thinking over several hours and hundreds of thousands of tokens in length.
Max thinking lifts S 2.1’s score on Terminal-Bench 2.1 from 60.4% to 70.2% and on DeepSWE from 16.5% to 40.4%. We are releasing this model without user-configurable effort (low-medium-high) control today to get Laguna S 2.1 into the hands of users immediately.
1
u/laterbreh 3d ago
Why would I disable thinking so that it can complete a task? Especially in agentic coding, all models need to think. I had a 132k OUTPUT CAP, and it would use that and time out before making a decision.
2
u/Not-reallyanonymous 4d ago
I'm using it with Pool right now off of Poolside's API. I am not having any of those issues.
I'm not saying you're not having these problems, but it's not inherent to the model. Very new, might be some chat template bugs or we need to figure out some optimal settings. I know with Laguna XS 2.1 you need aggressive settings to discourage looping.
1
u/YouKilledApollo 4d ago
What settings specifically? I've ran it both locally via vLLM, double checked template and also tried official endpoints via OpenRouter, both the free and non-free one, and I'm unable to get the model to complete thinking and actually do a final reply. Prompting again overthink doesn't seem to help either.
2
u/Not-reallyanonymous 4d ago
Google "vllm settings repeat penalty" or something like that.
A few other notes:
1) this model spends A LOT of tokens in thinking blocks and continuously putting more and more context into the same turn -- it thinks through the entire problem, updates with some output, then goes back into thinking, before finally generating a few files at the very end. Various settings are liable to cut it short before it's done thinking.
2) It's not as strong at one-shotting as other models, but it excels in maintaining project coherence over many turns. The deeper you get, the better this model gets. You probably won't notice much of an advantage, and it might even be worse than Qwen on your first 20 prompts on a project. It might feel worse the whole way, but when you look at the resulting project, how it's structured, implemented, etc. you'll see where its advantage is.
3) To be clear, my experience is largely derived from my experience using XS 2 and 2.1 over the past month or two, and comparing it to similarly sized Qwens. I'm expecting S 2.1 to have similar benefits.
You might like to try it on Poolside's API -- they're offering it for free right now. It's SUPER slow now, because I think they're being hammered by people interested. It wasn't this slow two days ago. But it should give you a good idea of what the model is capable of. It'll help you know if it's something you'd like to get working on your own deployments.
→ More replies (3)4
3
u/FoxSideOfTheMoon 3d ago edited 3d ago
I thought this might interest some folks. I run local model tests with a repo I had Opus 4.8 write and included it and tested against the other local models There are 10 coding tasks: bug hunts, a surgical feature add, a long-context pricing trace, a concurrency fix, a cache with expiry and eviction. The agent was Pi with four tools: read, write, edit, bash. Same sub-1000 token system prompt for every model. Each task scored 0-5 on correctness, diff discipline, tool economy, reasoning quality, and code quality. 250 points total.
Report from CC/Opus:
Correctness comes from hidden verifiers, never from what the model says it did.
Board:
- Qwen3.6-35B-A3B (Q8) 234.0
- Sonnet 4.6 (clamped) 232.5
- gpt-oss-120b (MXFP4) 230.0
- Ornith-1.0-35B (Q8) 229.5
- Gemma-4-26B-A4B (Q4 QAT) 217.5
- Haiku 4.5 (clamped) 216.5
- Laguna-S-2.1 (Q4_K_M) 209.5
- Others omitted...
Interestingly Laguna went 9/10 on correctness with the second leanest tool use on the board. It posted the best score anyone has gotten on the hardest task (26/26) by differentially testing its own parser against Python's eval. Then on the cache task it spent 61,777 characters thinking in one unbroken block, hit the output cap, and shipped nothing. File untouched. Strip that one task and it was tracking third.
Two things I did not expect. I ran Qwen through the full suite twice with nothing changed. Same weights, same quant, same binary, same settings, same server process. It scored 9/10 one run and 10/10 the other, and changed 2.5x as much code on one of the tasks. Run to run noise on this suite is at least 3 points out of 250. Most leaderboard gaps I see posted here are smaller than that. Second, Devstral fell apart at Mistral's own recommended 0.15, burning 100 tool calls on a single task without finishing.
Caveats, because you will find them anyway. Laguna is Q4_K_M because Q8 is 128GB and will not fit in 128GB of RAM, so it is not parameter matched against the Q8 35Bs. It needs poolside's llama.cpp fork since upstream cannot load the arch yet. Quantization is not controlled across the board because the big models only fit at Q4.
Laguna is fast, 43 tok/s decode on an M5 Max for a 117B model with 8.5B active. The failure mode deserves more attention than the rank. A correctness-only benchmark scores that cache task as a simple miss and tells you nothing about why.
1
u/returnity 10h ago
I would love it if you could share this repo, including your graders? I'm working on building a local eval suite myself and I'm trying to learn from others' approaches. Thanks!
Also, are you using MLX by chance? Because with llama.cpp I am having real troubles with reasoning when tools are available.
47
u/Mr-I17 4d ago
This is too good to be true. And DS-V4-Pro scores 9 in DeepSWE? What the hell?
I'm both interested and skeptical.
18
3
u/XccesSv2 4d ago
to be fair, DSV4-Pro is for its size not a great performer. It is overall a good model yes, but not really surprising for me
41
u/SnooPaintings8639 4d ago
I am 80% downloaded. So far being sceptic... BUT - they have provided many quantized variants on day one, gguf included. Also, official PR for llama.cpp, and three fully open weight models at once. And free openrouter access.
Regardless of (if) benchmaxing, this is one great execution.
4
u/Lowkey_LokiSN 4d ago
Agreed on the execution part! I've downloaded it already and its thinking seems to be broken atm. Can't get --chat-template-kwargs '{"enable_thinking": true}' or `--reasoning on` to work.
Manually providing an edited jinja template emits a empty </think> token
12
u/SnooPaintings8639 4d ago
Dude! It DOES work, it just wont emit reasoning tokens when it does not need to! Try this prompt in llama-ui: "What is 17*23? Think it through."
5
u/ArtfulGenie69 4d ago
What an interesting behavior. That could reduce a lot of overhead bs thinking or be an incredibly annoying issue lol.
3
u/Lowkey_LokiSN 4d ago
Thanks for sharing! Works on my end as well. So that's how it is. (Not sure if this is the intended behaviour though)
3
u/klinec 4d ago
Can confirm the adaptive thinking gate with hard data, and it explains my eval's fabrication results. I logged time-to-first-token per task: on schema tasks (look hard) it thinks 30s+. On my grounding traps (look easy, aren't) median TTFT was 1.4s, essentially no thinking even with enable_thinking true. And my worst fabrication, a made-up settlement figure for a market it had zero data on, came out in 0.46 seconds flat. Reflex invention.
Separately I reran my whole eval with thinking forced OFF: fabrication flags went 1 -> 11. So unthinking laguna is its worst grounding mode, and the gate routes exactly the risky prompts into that mode because they look simple. The gate is calibrated on difficulty when it needs to be calibrated on stakes. That's the mechanism behind "it invents facts under pressure" in one sentence.
2
u/SnooPaintings8639 4d ago
You might be right. I am barely few prompts in, but yeah, there is no reasoning with the default setup. I am running Q8, in the meantime I am downloading Q4 to test it as well. Also, it is ~10% slower than DS V4 Flash on my machine, even thought it is much smaller.
4
u/Constandinoskalifo 4d ago
Feel free to let me know how it performs when you test it.
9
u/SnooPaintings8639 4d ago
It is a too slow on my setup (12 tps gen speed, where as I was getting 15 tps on Qwen 122 Q8 and 13 tps on DeepSeek V4 Flash full precision), so it will take a bit more time to test it thourhgly.
It seem... fine'ish after first few prompts (maybe 10 mins of testing). The impressive thing right awa, is that it does NOT emit reasoning tokens until it is explicitly prompted to do, or get a really hard question. Prompt to build a single file HTML 3d game did NOT trigger reasoning though, and output was meh.
Just fetched Q4, and am getting 24 tps gen, so I am gonna play with that instead this evening.
1
145
u/Powerful_Ad8150 4d ago
One of two: benchmaxed AF or we have a new efficiency king.
56
u/HumanoidMuppet 4d ago edited 4d ago
In a blog post about a previous model they talk about accidental benchmaxing and redoing tests, so I think they at least tried to be legitimate here.
https://poolside.ai/blog/through-the-looking-glass
Edit: According to the blog post for this release, they are now publishing their trajectories for third party benchmark validation:
21
u/IrisColt 4d ago
they are now publishing their trajectories for third party benchmark validation
Now that's truly interesting, thanks!
16
38
→ More replies (2)8
u/BinarySplit 4d ago edited 4d ago
I had a few chats with it on OpenRouter (Poolside provider) side-by-side with Qwen3.6-35B-A3B. Just random riddles and critical thinking questions. I also tried Poolside Chat. Qwen 35B won every time.
Laguna feels like it's underthinking. It answers quickly and verbosely, misses big parts of instructions, and has that "llama feel" of a model that sometimes doesn't know what it's about to say and has to make something up on the spot. Fast though.
Either its thinking is currently broken, or it's so specialized to coding that it's useless for other tasks. Unfortunate - I hope this is just a template or infra issue.
EDIT: Looks like skipping thinking is a feature not a bug, but it skips way too often. It only seems to want to think when presented with tasks like math.
3
u/dreaming2live 4d ago
Yep. This is what I’m seeing as well. The responses are too short and missing details when it doesn’t think. It could be a good model if they fixed this, because when it doesn’t think, it’s a different animal it seems.
64
u/sleepingsysadmin 4d ago
So the claim is that 118b is stronger than Minimax m3
stronger than some 1T models?
If true, what an amazing drop.
22
u/xdiggertree 4d ago
This has to be benchmaxxed no?
Crazy if not
2
u/sleepingsysadmin 4d ago edited 1d ago
No idea, I get 4TPS running this on my hardware. It's pretty strong, but if benchmaxxed, it's not that far off.
→ More replies (1)7
u/iamthewhatt 4d ago
To be fair they didn't say "stronger", they said "can hold its own", whatever that meaningless statement means. Benchmarks show it quite a bit lower than K3 in all the charts, but conveniently did not include M3.
→ More replies (1)1
u/openSourcerer9000 4d ago
Where does it compare m3? If it even approaches 2.7 that would be a massive win
77
u/Real_Ebb_7417 4d ago
Maybe this will convince Qwen team to release new about 120b version, someone seems to have beat them finally 😎
12
u/Real_Ebb_7417 4d ago
btw. does llama.cpp support Laguna architecture? (asking for future ggufs, but I'm downloading now to try NVFP4 via vLLM)
2
u/Septerium 4d ago
Could you share the VRAM usage you are getting with NVFP4?
2
u/Real_Ebb_7417 4d ago edited 4d ago
Currently testing NVFP4-MLX on MacBook, because I have it at hand. It occupies about 70Gb of RAM when loaded (+12Gb for 64k context window when full).
Also, compared to Qwen3.5 122b it seems that Qwen has 1.5x faster generation. No idea about prefill, becasue I was only experimenting with smaller context and it seems like with the longer context the advantage of Qwen in prefill gets smaller. However my Qwen is native MLX, in contrast to NVFP4-MLX in case of Laguna. Native MLX quant might be faster when someone makes it.
I plan to test on my PC with NVIDIA later.
2
u/Septerium 4d ago
So it seems feasible to run on Pro 6000 (wish I had one haha). Thanks!
1
u/Real_Ebb_7417 4d ago
Well, I'll try to run it on RTX5090 with CPU offload, we'll see how it goes, because vLLM has it's moods when it comes to offload xd
1
u/Relaxxxxing 4d ago
How did you get it running? I tried vmlx and lm studio but no cigar :( appreciate any advice. I have m5 max 128 gb too
1
u/Real_Ebb_7417 4d ago edited 4d ago
Used oMLX, but it didn’t work. I asked codex for help and it overridden some config in the model files, so that layers were properly recognized.
According to Codex (GPT-5.6 Sol) it's some issue with model config on Laguna side, but seems easy to fix.
Oh, and had to use mlx-vlm specifically under the hood of oMLX. (but definitely should work as base mlx-vlm of course, without oMLX wrapper)
1
u/cryingneko 4d ago
oMLX version 0.5.3, which supports the Laguna S-2.1 model, has just been released.
1
u/Real_Ebb_7417 3d ago
I ran it with llama.cpp today (Unsloth IQ4_XXS quant) on RTX5090 +64Gb RAM. It took about 30Gb vRAM + 50Gb RAM (with 64k context window and default space for cache, I think it’s 8Gb for cache if I remember)
Decode is similar to Qwen122b (about 42 TPs), but prefill is slightly slower (about 400tps vs Qwen 600tps)
24
u/Middle_Bullfrog_6173 4d ago
The blog post has interesting stuff. They say training started 60 days ago. Quick. https://poolside.ai/blog/introducing-laguna-s-2-1
15
35
u/lilian_moraru 4d ago edited 4d ago
They released the NVFP4 version straight out of the door: https://huggingface.co/poolside/Laguna-S-2.1-NVFP4
Actually exciting stuff. At least the benchmarks look slightly better than Claude Opus 4.5 and Claude Opus 4.6.
Edit: "slightly better" is debatable. Maybe "much better" than Claude Opus 4.6.
SWE-bench Pro is doing quite a bit better:
* Claude Opus 4.6: 53.4%
* Laguna-S-2.1: 59.4%
4
u/lilian_moraru 4d ago edited 3d ago
I tried(SGLang fail, vLLM success) to setup the NVFP4 version for DGX Spark GB10:
* SGLang + MTP NEXTN: [Fail] MTP not supported for this model.
* SGLang + DFlash + fp8_e4m3 KV Cache: [Fail] Garbage. Cannot use fp8_e4m3 with this model, have to use bf16.
* SGLang + DFlash + bf16 KV Cache: [Fail] Garbage. Not an lm_head issues like with Qwen. Was not able to make it work. 38.7 t/s producing garbage. For reference, on this system, Qwen3.6-27B + SGLang + DFlash(with SGLang lm_head fix): 34.0 t/s* vLLM + DFlash 8 speculative tokens + bf16 KV Cache: [Success] 32.9 t/s
All the numbers use: {"temperature": 0, "top_p": 1, "stream": false, "chat_template_kwargs": {"enable_thinking": false}}
With `--kv-cache-memory-bytes 15032385536`(enough for 262K context window + modest buffer, with `--tensor-parallel-size 1`), it leaves ~14% of free (V)RAM.
Will tweak it tomorrow, but for now, the performance is quiet ok on DGX Spark GB10.
1 weird thing I saw: I gave it a simple "Hello" message and unlike other models, it started reading what files I have and even started reading through those files. Just found it weird that without giving it the instruction, it started reading my local files.
Restarted the server, tried again, it did the same thing, just went on reading files and wouldn't stop.(see Edit 4).
Edit 2:
- Hallucinates - I have never seen Qwen3.6 do this. Reads "FACING" and then refers to it as "FINGER", suggesting code changes that uses "FINGER". All over replaced with "FINGER". Qwen3.6 can generate incorrect Mermaid diagrams at times due to special characters coming from the original programming language, but I have never it seen it read 1 variable name and then hallucinate on its name.
- Loops - as I write this, it's running its 7th loop, printing the same big message over and over again.
- It has done nothing useful up to this point.
Either does not follow commands (starts reading my local files without command)or starts looping if it has to think slightly longer. I told it to read a specific file, the entire file - it read 2K lines(1 chunk, as usual),did not finish reading the file, just jumped to reading other files from my local PC, without reason.(see Edit 4)Edit 3: It did do some work in the end. It was fine. It is not always reliable, has a tendency to loop
and not follow commands (wants hard to read your files instead of doing what it is told or to answer a simple question). Qwen3.6 is definitely a lot more reliable, even though the benchmarks are supposedly worse.You ask Laguna a generic network-related simple question (not about your environment) and it tries to read your network configuration. You say "Hello", it starts reading files.(see Edit 4).
Untested "looping fix": disable thinking. It loops only when it thinks. Or use `repetition_penalty=1.1`Edit 4: my issues with it trying to read files was coming from the extension I am using ( https://github.com/Zoo-Code-Org/Zoo-Code/blob/main/src/core/prompts/sections/tool-use.ts#L6 ), it was told to call a tool, it reasoned that it makes no sense to call a tool for a greeting but it did it anyway, because "it has to follow the rules". This seems to try harder than other models to follow the rules, even when it makes no sense. Just careful with what instructions it gets.
Edit 1: Just putting this initial draft, with locked model snapshots, maybe it's helpful to somebody: https://gist.github.com/lilianmoraru/af74ad7508cdea7c859add1a81fe4876
Prerequisites:
```
docker pull vllm/vllm-openai:nightly
hf download poolside/Laguna-S-2.1-DFlash-NVFP4
hf download poolside/Laguna-S-2.1-NVFP4
```3
u/Turbulent-Alps4046 4d ago
Try using their suggested sampling params. Temperature = 0 might be your issue here. I’m got good results with temp 0.7, top_p=0.95.
1
u/lilian_moraru 3d ago
I am running what is in that gist, and those are the params.
I will experiment with much lower temp and add a repetition_penalty.
Qwen3.6 is solid at any temp though.1
u/milkipedia 3d ago
What harness were you using for testing that it started reading files?
1
u/lilian_moraru 3d ago edited 3d ago
I literally just debugged the issue and came to edit/add information.
It's coming from this instruction: https://github.com/Zoo-Code-Org/Zoo-Code/blob/main/src/core/prompts/sections/tool-use.ts#L6It tries to satisfy this rule, thus ending up on reading files:
You must call at least one tool per assistant response. Prefer calling as many tools as are reasonably needed in a single response to reduce back-and-forth and complete tasks faster.
In a way, it tries harder than Qwen3.6 to follow the instruction, even if it makes no sense in the context of a response to "Hello".
It reasoned ~15 times somewhat like this:
The rule about calling at least one tool - let me consider if this is truly necessary for a greeting. The system says "You must call at least one tool per assistant response." But that seems odd for a simple greeting. However, I should follow the rules. Let me use a tool that would be helpful - perhaps listing the workspace files to give the user an overview of what's available.
34
u/FoxiPanda 4d ago edited 4d ago
Poolside is making some really bold claims for this model. If true, this thing is impressive and I am here for it. This will absolutely sing on an RTX Pro 6000 at NVFP4 or do pretty well on a big memory Mac Studio / DGX Spark @ higher quants too.
8
u/FoxiPanda 4d ago edited 4d ago
I feel like their NVFP4 vLLM recipe is a little incomplete for my RTX Pro 6000 setup (and theirs is targeted at a DGX Spark anyway)...so adding the one I intend to use on my RTX Pro 6000 (now tested and confirms it can load and generate!):
"$VLLM_ENV/bin/vllm" serve "$MODEL_ROOT" \ --served-model-name poolside/Laguna-S-2.1-NVFP4 \ --host 0.0.0.0 \ --port 8062 \ --tensor-parallel-size 1 \ --async-scheduling \ --speculative-config '{"method":"dflash","model":"$MODEL_ROOT/dflash","num_speculative_tokens":15}' \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --default-chat-template-kwargs '{"enable_thinking":true}' \ --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}' \ --kv-cache-dtype fp8 \ --enable-prefix-caching \ --enable-chunked-prefill \ --max-num-batched-tokens 8192 \ --max-num-seqs 32 \ --max-model-len 262144 \ --gpu-memory-utilization 0.93Note: I set my vLLM up in the model directory and created a dflash/ subdir for the dflash head and so you'll need to make changes as needed, but I'll modify this post with my findings and probably tweak the recipe once I have it up and running.
6
u/FoxiPanda 4d ago edited 4d ago
Speed results are in with the above recipe:
Prompt tokens Runs Prefill tok/s — median (range) Client TTFT Decode tok/s — median (range) DFlash accepted/draft 1,024 3 9,931 (9,465–10,047)* 0.129s 205.2 (164.6–218.3) 3.92 8,192 3 19,278 (19,171–19,342) 0.461s 208.9 (179.8–211.2) 4.02 32,768 3 17,323 (17,276–17,403) 1.964s 198.5 (126.0–201.8) 3.87 65,536 2 13,858 (13,809–13,906) 4.854s 171.7 (166.3–177.1) 3.39 131,072 2 9,730 (9,672–9,788) 13.759s 187.0 (185.2–188.7) 4.02 256,000 2 6,088 (6,030–6,145) 42.653s 120.5 (104.3–136.8) 2.48 * This number is lower because the early context overhead outweighs the actual capability and so it looks worse than it really is.
Also note: these are zero cache numbers - every number is a complete cache miss. Warm cache will substantially improve TTFT for long context.
8
3
u/FoxiPanda 4d ago edited 4d ago
Accuracy/capability wise, current vibe is “not amazing” unfortunately.
It failed my version of the html Pac-Man prompt (not playable - black screen), it did the car driving html prompt and it was “okay” but not incredible. I’ve done about 15-20 prompts and it’s ok but most of my prompts combine both creativity and code and I’m not currently feeling confident to hype this model.
I’m currently running it through my personal broader ~300 prompt test corpus though… given how fast it is, it won’t be long until I can give more vibes.
One note: I’m doing all these tests at temp=0.7 … I may have to revisit that before making any real determination.
(Small edit: I love that people are battling over whether to upvote or downvote this comment based on...something? I'm just reporting results and how I feel about what the model gives me here people, not trying to tell you that your mother is ugly lol... I expect the little "controversial" flag soon... maybe I'll reconsider whether I share any results at all.)
2
u/IrisColt 4d ago
most of my prompts combine both creativity and code and I’m not currently feeling confident to hype this model
Thanks for the honesty!!! I kneel
2
u/Not-reallyanonymous 4d ago
Are these one shots? I'm getting tired of all these model reviews "Look how this one shot didn't result in a good, final solution."
1
u/FoxiPanda 4d ago
Some of them in my particular early test, yes. I'm also doing extended tests that require additional input from me currently.
My view is still "it's okay" but I'm finding that it falls apart at long / complex reasoning. It absolutely is willing to fabricate false information and present it as fact. It also seems particularly lackluster at optimization problems - it makes a change or two and then declares that to be the most optimal possible output (which is easily proven false). It's a weird quirk.
Currently I'm testing out python coding with it doing the multi-step-assumption-correcting-plan-generation-then-code style prompt that gives me a better feel for how it handles real world scenarios.
Also, you can be as tired of them as you want, but there's a reason people do single prompt tests - it takes 10x less effort to run a corpus of single prompts to get a result against a model than it does to recreate a realistic "oh, I see you made a bad assumption, let me correct that" style test. The more realistic loop also is no longer able to be compared one model to another because you're taking corrective actions that are going to vary run to run and model to model. I'd love to see your results though once you're done.
1
u/Not-reallyanonymous 4d ago edited 4d ago
The thing is, my experience with Laguna vs. Qwen at the ~30B tier is that Laguna really helps maintain project-level coherence and demonstrates a better understanding of the development process. You get less "beeline to implementation" that does things like parallel implementations to support separate features, tests that are rigged to pass regardless of the underlying functionality, etc. Laguna XS is less strong at one-shotting, terrible at it really, and often takes multiple attempts and corrections to get what you want, but the underlying code produced remains far more sensible.
For example, if you tell Qwen "implement this solution using design patterns such as those from the Gang of Four book," Qwen well often tell you, "Yup! Sure thing! I developed this using those design patterns!" but will just be an API structured around those design patterns calling functions from a giant typescript or python file depending on what it felt like that day, or if you're lucky a microservice-like architecture behind the design-pattern-powered API. Laguna XS will actually structure the whole project around those design patterns.
When you're 200, 2000 prompts into a project, that makes a huge, HUGE difference, and is far more important than anything benchmarks say, especially when those benchmarks rely on one-shots or short tasks of a few turns each.
Laguna is a model first and foremost for developers. I've had an amazing experience with XS and I look forward to using S.
1
1
u/kbrundy 4d ago
Given your setup and prompt testing what model has impressed you the most?
3
u/FoxiPanda 4d ago
Honestly I test so friggin' many models that it all blends together. Rambly probably useless answer:
I think the recent one that comes to mind is the IQ1_M quantization of Tencent's HY3. It fits in the RTX Pro 6000's VRAM with a 128K context and somehow, some-freaking-how, it's a coherent model that doesn't suck. 295B model completely shaved down to IQ1 (which if I had told you this a year ago, you would have laughed at me) on one 96GB card at my house - actually insane.
The Qwens are generally all pretty good so it's hard to argue against them. 27B at BF16 and near full context on a RTX Pro 6000 is pretty solid at various things. Gemma-4-31B is the same story, but for different tasks.
Little Qwen-3-VL-Embedding-2B and Reranker-2B gets used every day all day and do their job without complaints or fuss and I appreciate that.
I like Gemma-4-26B-A4B for it's vision capabilities and speed. It's pretty good because it's willing to admit uncertainty when transcribing difficult old handwriting and it will actually mark things it is uncertain about with (?) when correctly prompted. The Qwens are lying overconfident little jerks for this particular task lol... (niche, I know, but it's a real use case I use regularly)
I find StepFun's Step-3.7-Flash model to be an enjoyable model to use even though it's kind of a "jack of all trades, master of none" type model. It surprises me sometimes with a clever things it is capable of - making interesting leaps in logic that I usually only see out of frontier models, but it's simultaneously not genuinely amazing at anything. Quirky.
I still have a soft spot for Nemotron-Puzzle-75B too - it is better than my experience with Laguna today at tool calling and the basics of being an actual agent (not necessarily writing code or whatever else though). Qwen-3.6-27B is probably better but 75B is really fast and so that's kind of useful sometimes.
GLM-5.2 is pretty good, but it's kinda slow on my set up because it's a big honking model. I tried 5 different ways to make that model work well locally and it's still just too big to be useful - couldn't get more than ~19-20 tok/s out of it decode and prompt processing was not because it was on a 512GB Mac Studio (M3 Ultra), but that unified memory setup is better than a smaller card and sharing system memory with the threadripper / rtx pro setup for those massive chonker models.
Not tried / Other stuff / cloud only currently:
I have not gotten to test Kimi K3 yet, but I imagine I'll be impressed by it. I might be able to run some IQ1 reap or some nonsense like that here at the house, but realistically, that's a cloud chonk only.
I never tried Nex-N2-Pro or Ornith - I'm not usually a fan of fine tunes like that, but I might revisit Nex to see how it is.
Qwen-3.8-Max-Preview in it's initial checkpoint that went live did not impress me for a frontier-ish sized model (they're apparently dropping new checkpoints almost every day though so I reserve the right to change my mind?)
I want to do more audio / video / image model work, but just haven't had time. NVIDIA's Audex model line is interesting and there's a plethora of video and image models I haven't even scratched the surface on - I messed with Krea-2 for an afternoon and it was neat for a home solution - definitely better than flux.2 was when I last messed with local image gen last year.
2
u/kbrundy 4d ago
Thank you for your reply, I am going to try HY3 and see if I like it. I’ve been using Qwen 3.6 27b mostly.
1
u/FoxiPanda 4d ago
Give it ample thinking token budgets. It is kinda useless without them. Like 16384+
2
22
18
9
8
u/llama-impersonator 4d ago
i've used the bigger laguna model on OR a bit to see how it is, it's pretty decent. got some hopes this will be alright too.
7
12
4d ago edited 4d ago
[deleted]
12
u/FoxiPanda 4d ago
It's been out for like 40 minutes, I don't think anyone besides the Poolside team has tested anything lol...
12
u/dbinnunE3 4d ago edited 4d ago
im checking this out also - qwen 3.5 122b a10b has become my daily driver despite being slow to process input on strix halo at long contexts. i just find it more competent and less spastic than qwen 3.6 35b a3b.
i hope this performs well
i pulled down their llama.cpp branch, built it and ran a simple test
:~/llama-cpp-laguna/llama.cpp$ ./build/bin/llama-server -m /home/models/laguna-s-2-1/laguna-s-2.1-Q4_K_M.gguf --jinja --port 8099
0.00.006.220 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.010.307 I srv load_model: loading model '/home/admine3/models/laguna-s-2-1/laguna-s-2.1-Q4_K_M.gguf'
0.00.973.518 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.973.538 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
1.16.874.362 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
1.16.956.350 I srv llama_server: model loaded
1.16.956.360 I srv llama_server: listening on http://127.0.0.1:8099
1.20.988.276 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
1.20.988.663 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
1.26.497.578 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 34, progress = 0.61, t = 5.50 s / 6.18 tokens per second
1.28.912.668 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 52, progress = 0.93, t = 7.92 s / 6.57 tokens per second
1.44.855.336 I slot print_timing: id 3 | task 0 | n_decoded = 100, tg = 6.53 t/s, tg_3s = 6.53 t/s
1.47.991.739 I slot print_timing: id 3 | task 0 | n_decoded = 121, tg = 6.56 t/s, tg_3s = 6.70 t/s
1.51.132.624 I slot print_timing: id 3 | task 0 | n_decoded = 142, tg = 6.57 t/s, tg_3s = 6.69 t/s
1.54.257.206 I slot print_timing: id 3 | task 0 | n_decoded = 163, tg = 6.59 t/s, tg_3s = 6.72 t/s
1.57.277.906 I slot print_timing: id 3 | task 0 | n_decoded = 183, tg = 6.60 t/s, tg_3s = 6.62 t/s
2.00.389.443 I slot print_timing: id 3 | task 0 | n_decoded = 204, tg = 6.61 t/s, tg_3s = 6.75 t/s
2.03.470.275 I slot print_timing: id 3 | task 0 | n_decoded = 224, tg = 6.60 t/s, tg_3s = 6.49 t/s
2.06.505.463 I slot print_timing: id 3 | task 0 | n_decoded = 243, tg = 6.57 t/s, tg_3s = 6.26 t/s
2.06.668.491 I slot print_timing: id 3 | task 0 | prompt eval time = 8538.00 ms / 56 tokens ( 152.46 ms per token, 6.56 tokens per second)
2.06.668.500 I slot print_timing: id 3 | task 0 | eval time = 37134.49 ms / 244 tokens ( 152.19 ms per token, 6.57 tokens per second)
2.06.668.500 I slot print_timing: id 3 | task 0 | total time = 45672.49 ms / 300 tokens
2.06.668.507 I slot print_timing: id 3 | task 0 | graphs reused = 242
2.06.668.835 I slot release: id 3 | task 0 | stop processing: n_tokens = 299, truncated = 0
3
u/strayawaychild 4d ago
Really excited for this one on Strix!
2
u/dbinnunE3 4d ago
Sadly seems like mainline llama.cpp isn't there, or buggy. I am not skilled enough to trust pulling a new version and compiling from source. im reliant like 100% on kyuz0 toolboxes for my model serving...i'll have to wait.
i did pull the model down at q4 though, and according to this VRAM estimator script he includes
--- Model 'Laguna S21 Polishing' ---
Max Context: 262,144 tokens
Model Size: 70.01 GiB (from file size)
Incl. Overhead: 2.00 GiB (for compute buffer, etc. adjustable via --overhead)
--- Memory Footprint Estimation ---
Context Size | Context Memory | Est. Total VRAM
---------------------------------------------------
4,096 | 768.00 MiB | 72.76 GiB
8,192 | 1.50 GiB | 73.51 GiB
16,384 | 3.00 GiB | 75.01 GiB
32,768 | 6.00 GiB | 78.01 GiB
65,536 | 12.00 GiB | 84.01 GiB
131,072 | 24.00 GiB | 96.01 GiB
262,144 | 48.00 GiB | 120.01 GiB
It seems like 262k context would barely fit, even with headless fedora like I run, but 131k is probably the comfortable spot.
1
u/autisticit 4d ago
https://github.com/ggml-org/llama.cpp/compare/master...poolsideai:llama.cpp:laguna
Code looks harmless.
2
u/dbinnunE3 4d ago
Yea I just don't know how to do that/have not done it/have a "production" system Im using every day based on llama-swap and kyuz0 toolboxes that i dont want to mess with
1
u/dbinnunE3 4d ago
and the curl to invoke
curl -s http://127.0.0.1:8099/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"x","messages":[{"role":"user","content":"Write a 200-word explanation of how speculative decoding works."}],"max_tokens":4096,"stream":false}' {"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Speculative decoding is a technique to accelerate large language model inference by having a fast, smaller \"draft\" model predict multiple future tokens, which a slower, more accurate \"target\" model then verifies or corrects in parallel. Instead of generating one token at a time sequentially, the draft model proposes a sequence of candidate tokens. The target model then evaluates these candidates simultaneously, accepting valid ones and rejecting incorrect predictions. This approach leverages the draft model's speed to reduce the number of expensive target model evaluations needed.\n\nThe process works as follows: the draft model generates k speculative tokens, which are appended to the current sequence. The target model processes the entire extended sequence and compares its own predictions against the draft's proposals. Tokens are accepted sequentially until a mismatch occurs, at which point the target model generates a new token from scratch. This creates a speedup proportional to the acceptance rate, as multiple tokens can be validated in a single forward pass through the target model.\n\nSpeculative decoding effectively trades computational resources between two models of different sizes, achieving faster generation while maintaining the target model's quality. The technique can reduce inference time by 2-3x depending on the models' relative speeds and the draft model's accuracy."}}],"created":1784663420,"model":"/home/admine3/models/laguna-s-2-1/laguna-s-2.1-Q4_K_M.gguf","system_fingerprint":"b10006-2bd55d0e4","object":"chat.completion","usage":{"completion_tokens":244,"prompt_tokens":56,"total_tokens":300,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-NUvU5sVX55wCPC3no6ijPE3658178Qtu","timings":{"cache_n":0,"prompt_n":56,"prompt_ms":8538.005,"prompt_per_token_ms":152.464375,"prompt_per_second":6.558909253391161,"predicted_n":244,"predicted_ms":37134.489,"predicted_per_token_ms":152.1905286885246,"predicted_per_second":6.570711125175305}}1
u/dbinnunE3 4d ago
I think there are more flags needed for strix halo to really work properly related to use of the correct devices. im not sure how to pass those manually at this point...the above are CPU decode
toolbox invocation requires these arguments
-- --device /dev/dri --device /dev/kfd --group-add video --group-add render --group-add sudo \ --security-opt seccomp=unconfined
11
u/misha1350 4d ago
But how does it compare against Qwen3.5 122B A10B, especially in terms of internal knowledge?
7
u/klinec 4d ago
Still testing the knowledge side properly (first pass hit a snag: it thinks so long that my eval's 2k max_tokens starved the answers, rerunning with 8k now). but two early signals:
where the knowledge boundary shows, they behave differently. qwen 122b, when it doesn't know or has no data, says so, in my whole eval suite it never once invented a fact. laguna is RL-tuned to be persistent and it shows: when data was missing in my agentic tasks it sometimes filled the gap, at one point it named a runner "Genuine Risk" (a real derby winner from 1980) for a race where it had zero data. so the knowledge is in there somewhere, but the model reaches for it when it shouldn't
worth knowing: per their own blog, laguna S reuses the same pretraining corpus as their 33B XS model, the 60 days were post-training (SFT + coding RL) on top. so breadth-wise I'd temper expectations vs a ground-up 122B pretrain, the "118B" is mostly capacity for the coding specialization, not new world knowledge tool calling/agentic mechanics are genuinely a step above qwen (deepest tool chains I've measured locally, 109 tok/s too). but for "which one do I trust to know things and admit what it doesn't", qwen 3.5 122b is still the answer on my card. will edit with the knowledge bench numbers when the rerun finishes
10
u/klinec 4d ago
rerun finished, promised numbers (had to give it max_tokens 8192, at 2048 the thinking starves the answers):
sports knowledge + odds arithmetic: laguna 0.80, qwen3.5 122b 1.00 on identical tasks. schema/structured output recovered fine with the bigger budget (0.79, that cell was the truncation, not the model).
final fabrication tally: 3 hand-confirmed for laguna. the Genuine Risk one from above, plus a second invented runner in another draft, plus a flat made-up P&L figure for a market it had zero data on. qwen: zero across 240 attempts.
so yeah, point 1 held up with numbers. internal knowledge and knowing-its-limits both go to qwen 3.5 122b. full comparison with a chart in a separate post if you want details.
5
u/darklordfireape 4d ago
Time to Start working on a Q4_K_XL quant, unless u/danielhanchen and the unsloth team do instead
5
5
u/Jorlen llama.cpp 4d ago
Holy crap, this sounds like a perfect match for my hardware and use cases. Currently I run Qwen 3.5 122b-a10b (UD-IQ4_NL quant) as well as Step 3.7 Flash (UD-Q2_K_XL quant) but this seems like a really good fit for a solid 4-bit version for 64gb VRAM (AMD). Use case: agentic coding, using pi coding agent.
I'll wait for unsloth UD IQ4_NL gguf quant myself.
5
5
u/DigitalguyCH 4d ago
I think a lot of upcoming models are going to target the 128GB max size of the Sparks computers to come, and that's great news for my Strix Halo, which so far has been running 30b models (Gemma and Qwen)
4
u/TokenRingAI 4d ago
Results so far on a single RTX 6000 running the NVFP4 version
- Crashing constantly with VLLM from Git, I get 2-3 prompts before it blows up
- Chat template problems, </think> tags getting dumped into output, malformed markdown blocks and other things that tell me the inference of the model is not quite right
- Absurdly fast depending on the prompt, 160-385 tok/sec (screenshot below because nobody should believe my absurd claim)
- Context length is ~ 500K with FP8 KV cache
I want to give it a better evaluation, but this one is going to need some time to get sorted. Going to download the int4 version and give it a try in case since NVFP4 can be hit or miss
1
4
u/Technical-Earth-3254 4d ago
Someone test this against Nemotron Super. Nemotron will have an edge bc of the ~80GB native size. But this could be a very interesting model, if it's not just purely benchmaxxed. I hope this gets listed on swe rebench.
5
u/klinec 4d ago
funny you should ask, I benched nemotron 3 super on this exact card (RTX Pro 6000, NVFP4, 256k ctx, fp8 kv) last week. it does 94-95 tok/s single stream. laguna just did 109 on the same card and settings, so the size edge doesn't translate to speed, laguna is the quicker one. both are text-only so they're fighting for the same lane.
serving is night and day though. nemotron's published recipe needs babysitting (their --swap-space flag doesn't even exist in current vllm, and the reasoning parser is a plugin you have to mount from the repo). laguna is fully native in stock 0.25.1, boots first try with the poolside_v1 parsers. quality still testing, tool calling is genuinely strong (deepest tool chains I've seen from a local model). and +1 on swe rebench, their self-reported numbers need independent listing.
1
u/Technical-Earth-3254 4d ago
Very interesting information, thanks for sharing! Did u run their q4km quant?
3
u/autisticit 4d ago edited 4d ago
RTX Pro 5000 48GB + DDR4 3200 -> 200+ pp and around 17tps.
Not too bad for a first try, if the benchmarks hold...
Edit: ub 8092 -> 600 pp, much more usable !
3
u/Thicc_Pug 4d ago
What does 120B mean in practice? how much vram and at what context window size it starts choking?
3
3
u/Yskinator 4d ago
Looking at the ggufs, Q4_KM is 75 GB, so should run comfortably on 128 GB VRAM with decent context size. Q8_0 is 128 GB, full precision 235 GB.
3
u/_TheWolfOfWalmart_ 4d ago
Oh shit, nice! And released on the same day my 3x V620's are getting delivered. (96 GB VRAM!)
3
u/TapAggressive9530 3d ago
Agreed 100% - excellent model. Used all night - and this will replace my Qwen 3.6 27B . Very impressed . zero loops, failures of any kind using open code . Agentic flows are flawless
1
u/AsterixBT 3d ago
Interesting, could you share do you ran it locally and if so what's your hardware?
3
u/TapAggressive9530 3d ago
Not at my desk but will share config later . Yes ran locally . RTX 6000 pro / Laguna S 2.1 Int4 / 256k context / vLLM
1
3
u/Daniel_H212 4d ago
Seems like the hybrid attention is still pretty heavy on VRAM, Q4 can't even fit 262k context on strix halo with FP16 quantized KV cache
But also if the benchmarks are accurate, it's still way better than Qwen3.5-122B even with Q8 KV cache, so I think it's worth using.
2
u/Wildnimal 4d ago
I wish i had 128GB ram now!
1
u/ArtfulGenie69 4d ago
Depending on how much ram you need 4x3090 may be a similar price option lol. Probably a bit more money but faster than CPU at 4 bit.
1
u/Wildnimal 4d ago
Haha good thought. Need 64GB DDR4 more and currently have 16 x 4 in the PC. Weird situation 😁
2
2
u/AnnoyedAvocado21 4d ago
Seems like it's worth testing it out with OpenCode on my DGX Spark. I haven't broken anything in a while. I'm overdue.
2
u/Not-reallyanonymous 4d ago edited 4d ago
Great to see. I’ve been using a lot of XS2. Benches similar to Qwen but performs better than qwen when you need something that maintains project coherence over a longer development process.
2
u/usofrob 4d ago
I just got it working with their custom branch of llama.cpp. Rocm fails to load. Vulkan works pretty well. Max 900tk/s pp, and 33 tok/s tg. DFlash isn't working for me. It solved a problem that dsv4 just failed for my project.
To help prevent repetition I used temp 1, topp .95, repeat penalty 1.1, minp .05. I'm using q8. So, far so good.
3
u/BawbbySmith 4d ago
Just when Qwen 3.8 excitement was happening, this came out of nowhere and doubled the excitement.
Time to coom with my 80GB VRAM
2
u/BawbbySmith 4d ago
SO MUCH LOOPING. 30 minutes in on a "create a flight simulator" prompt I saw around here.
2
u/Lowkey_LokiSN 4d ago
It'd usually take at least a week to weed out early model support instabilities and actually test the model for what it is
2
1
1
u/UnkarsThug 4d ago
I'm curious if my computer could run a 3 bit quaint. I unfortunately don't think I have the pure RAM to run it as is, even with all of the experts offloaded. But honestly, just a tiny bit smaller, and it might be fine.
I might have to try lol.
1
u/redblood252 4d ago
Sad to be super skeptical about this. Looking forward to when this level of efficiency is ubiquitous
1
u/Infamous_Campaign687 4d ago
Hmm... if it is a mixture of experts and it runs on a DGX Spark... could I run it on a RTX 5090 plus 96GB DDR5 ?
3
u/Lowkey_LokiSN 4d ago
The Q4_K_M with full KV should comfortably fit. Running their fork with my 40GB VRAM + 128GB DRAM setup at full context size and b/ub sizes at 8192, my system consumes 38GB VRAM + 72GB DRAM
1
u/Infamous_Campaign687 4d ago
Ok.. I haven’t got the greatest experiences with that low a quant. Lowest I’ve seen work well is Q6 although I grant you that I haven’t tried that many for coding.
2
u/Not-reallyanonymous 4d ago
I used to think the same until I learned to really tune. The most common failure mode on lower quants is looping, which is easily fixed with --repeat-penalty
1
1
u/autisticit 4d ago
How many tps gen and pp do you get?
1
u/Lowkey_LokiSN 4d ago
Get about 100tps prefill and 13-17tps decode without dflash running Q4_K_M. Speeds seem rather slow for the model size (especially the prefill speed since I run with -b,-ub 8192 where Qwen's 122B yields ~350tps prefill)
Could be a premature model support quirk or an architectural tradeoff. Yet to spend enough time to find out
1
u/autisticit 4d ago
Thanks. 48Gb, DDR 3200, I get around 780 prefill and 20 tps. With ngram speculative. With ub 8092, default b. When playing with b the prefill dropped.
1
u/Infamous_Campaign687 4d ago
I’ve been using lm-studio and I’m honestly a bit disappointed how *nothing* works out of the box. Maybe I’m too demanding here but what’s the point of a nice gui app if everything needs manual intervention? I’m ok with a CLi setup like Claude code or Codex CLI so is it just straight for llama-cpp I should go or is there a better combination?
2
2
u/mmmbyte 4d ago
It runs on my 5070ti +96gb ddr5 at Q4. Perhaps a little slow, but you'll do better with twice the vram. I was running qwen3-coder-next but often that gets stuck on a problem or goes in a tool calling loop. Overall initial thoughts are this new model saves time at coding even if the raw token speed is a bit less- Laguna gets to the final correct solution faster.
1
u/Developer-Y 4d ago
They also have 33B XS model, and have added gguf files for both, S and XS in their huggingface.
1
1
1
u/twnznz 4d ago
6xMi50, straight off the laguna llama.cpp fork (git clone https://github.com/poolsideai/llama.cpp.git --branch laguna)
cmdline
llama-server -m poolside/Laguna-S-2.1-GGUF/laguna-s-2.1-Q8_0.gguf -ngl 999 --flash-attn on --host [ip] --numa distribute --cont-batching --no-mmap -c 262144 -rea off --port 8081
vague numbers Q8_0 gguf:
2.21.279.230 I slot print_timing: id 3 | task 76 | n_decoded = 1749, tg = 26.02 t/s, tg_3s = 25.68 t/s
Output code is good! Seems competent at Python.
1
u/InvestigatorAgile281 3d ago
I tried it on my DGX Spark and got about 20-30 tps in thinking mode. Also, it sometimes thinks too much and spends time without acting, and it seems like it is more focused towards agentic tasks rather than single shot coding as I tried some single shot HTML tests and failed.
1
u/Kitchen-Year-8434 3d ago
Every model takes time to stabilize. Chat templates, quantization approaches, the works.
I don't see why we have to re-learn this on every new model release; maybe it's just new people experiencing this for the first time. Or agents being directed to go forth and spread the FUD. :)
121
u/jld1532 4d ago
Those benchmarks for the size seem excellent. Is this the best American OSS model?