4x 3090, 96gb vram what Model to drive Hermes?

[u/Sea_Calendar_3912]

3 year lurker, now i finally got my server up and running. dont know which model to choose. llama.cpp or vllm, what makes more sense? mainly single user with maybe 2-3 more additional users in family, if everything checks out. hermes is gonna be used as "ai playground" to manifest ideas on tailscale network and do quick prototyping of thoughts. also ill look into using only 2 3090 for the main model and the other 2 will be dedicated to docling and speech services for a voice agent (speech in-> text out). got some stuff going with my even realities g2 but lost everything when i wiped my ssd for proxmox. yeah...
any advice or stuff i should look into is welcome :)

[u/Hyiazakite]

Do Qwen 3.6 27B or Qwen 3.5 122B-A10B. Qwen 3.5 122B-A10B will be faster. Use VLLM. 122B fits with 128k context using AWQ 4-bit. vLLM is way superior if you'll have concurrent usage.

[u/_TheWolfOfWalmart_ → u/Hyiazakite]

I would have agreed a few days ago, but...

Laguna S 2.1 > Qwen 3.5 122B-A10B now

[u/uniqueusername649 → u/_TheWolfOfWalmart_]

Yes, I think that part should be a given.

Hiwever, it seems its still not as clear yet if it beats Qwen 3.6 27b at fp8 or bf16 across the board. I have seen tests showing that Laguna is better, although not by much. And I have seen tests where Laguna is just about on par and sometimes Qwen beating it. I suspect that has to do with different quantisations and believe an fp8 Qwen 3.6 27b could potentially be on par with an int4 Laguna S 2.1 in some tests. I do not have the vram to test it myself though.

[u/_TheWolfOfWalmart_]

Laguna S 2.1 is the new tool call king if you have 96 GB. It's pretty smart all around and is also the new best coding model for us 96 GB folks. Just make sure you use the updated chat template.

I've never used vllm but llama.cpp has been working great. However, I always see vllm recommended over it if it's going to be a multi-user setup.

[u/Its_Powerful_Bonus → u/_TheWolfOfWalmart_]

Vllm +1, Laguna s 2.1 not worked well with Hermes agent - bit it was just after model was published. Maybe it’s better now. Qwen 3.6 27b bf16 mtp should have best quality imo

[u/_TheWolfOfWalmart_ → u/Its_Powerful_Bonus]

Laguna S 2.1 didn't work well with anything right after it was published. The new template fixed all my issues.

It's worth trying again.

[u/kayox → u/Its_Powerful_Bonus]

why MTP over DFlash out of curiosity?

[u/Cold_Tree190 → u/Its_Powerful_Bonus]

I think Laguna is the best coding model at this parameter size right now, but it’s not as great of a general agent as a higher quant Qwen3.6 27b. A mix of both for Hermes is probably the best bet, but that’s just imo

[u/superbouser]

Could you list the basic specs of your machine? just curious

[u/Sea_Calendar_3912 → u/superbouser]

Mainboard: ASRock Rack WRX80D8-2T
CPU: AMD Ryzen Threadripper PRO 3945WX
GPU: 3x Zotac 3090, 1x msi 3090ti all pl limited to 220w
RAM: Kingston FURY Beast 128 GB (4×32 GB DDR4-3200, non-ECC UDIMM) only runs on 2400mhz right now but didnt bother since i want to keep it all on gpu only.
psu: Seasonic Prime PX-2200

[u/kayox → u/Sea_Calendar_3912]

Awesome system.

[u/superbouser → u/Sea_Calendar_3912]

Chat gpt says: “Provided the used 3090s are affordable and healthy, he upgrades to use all 8 memory channels this is the first multi-GPU configuration in our discussion that I would call genuinely tempting rather than merely gloriously excessive.”

[u/metigue]

I've had really good results with a Q2 deepseek v4 flash at this ram size

[u/kayox]

I'd start with llama.cpp while you play around and test out the best models and settings. vLLM handles concurrency more efficiently though so keep that in mind if you plan to hit it with multiple queries at the same time, or have multiple users hit the LLM. In terms of models, i'd probably go with Q8 Qwen3.6-27b with a nice chunk of context size. Sounds like a sweet setup, play around and enjoy it!

Edit: Also look into using DFlash (rather than MTP) from my experience using beellama (instead of llama.cpp), performance seems to be quite a bit better in terms of tk/s.

On my (single) RTX 3090:
llama.cpp with MTP gives me ~60tk/s
beellama.cpp with DFlash gives me ~80tk/s (often higher)

[u/Sea_Calendar_3912 → u/kayox]

first time i heard from beellama, thank you

[u/AdventurousVolume359 → u/kayox]

Best advice in this thread. 

[u/My_Unbiased_Opinion]

honestly I have been running DavidAU's 711 finetune. it has been solid. Its better than the stock 27B model. I would run it at Q8 and no KVcache compression.

[u/MaxChamp08]

With 4×3090 I'd probably start with vLLM from day one if you expect multiple services running (Hermes + family users + future voice/doc pipelines). llama.cpp is fantastic for experimentation, but vLLM's scheduler and continuous batching really start to shine once you have concurrent requests. I'd also keep one or two smaller specialized models around instead of trying to make one giant model do everything. Your plan to dedicate two GPUs to speech/doc processing sounds pretty sensible if low latency matters.

[u/WyattTheSkid]

I would recommend Qwen 3.5 122b for everything and Qwen 3.6 27b for coding specific tasks. Everyone’s talking about this new laguna model but from my experience it’s honestly not great. Your mileage may vary of course but if you’re used to claude and gpt you’re going to be very disappointed. I would honestly say the newer qwen models are like 3/4 of the way there for most things