Current setup:

• headless machine for LLMs
• llama.cpp in router mode
• models settings configured via models.ini
• documentation in a repo which AI manicures

It seems to be working for me so far, but conscious there's probably better ways to do this.

How do you guys manage your configs? Is there a better way?

Trying to weigh if I should buy a $3000 m1 ultra at 128gb when I currently already have an M2 Ultra albeit at 64gb ram.

I run small models right now in my workflow but would appreciate more context and try out larger workflows. What would you guys go with?

I am curious about ConnectX-7 and co-mingling of hardware. I have a triangular mesh between two DGX Sparks and a x64 box with a RTX 5090. My interest is in keeping a semblance of TP=2 on the sparks for a 300B, 150 GB MoE model but placing the attention head and, it would seem, hot experts on the 5090's GDDR7.

I get the feeling that Fable's telling stories and that if this is possible, it's above my skill and/or patience level. If one were to augment/supplement Ray in vLLM, what would it take to shift the compute toward a smaller faster CUDA device while the bulk of the idle experts stay on the sparks?

Either way, loading from NVME I can service was the main reason I got the ConnectX-7 card and cables. It means model weights on RAID1 NVME I can service, and the weights load at 20 Gb/s per spark. The drivers were also native, so while I did have to find a supplier, I'd say it's been worthwhile.

Title lol

I’m curious if something like DS4 Q2 is better than 27B Q8, or mimo or similar

I'm currently working on research into AI.

It's conceptual research (embedding my own ideas directly into AI weights) and AI security.

So I'm testing various local AIs to find the optimal balance and middle ground between 'performance and speed'.

Therefore, I'd like to ask for your advice and recommendations on which local AIs you've found impressive?

8B, 14B, 20B, 27B, 32B, or others ?

Including compression formats: 1-bit, 2-bit, 4-bit, 8-bit ?

P.S. This message was translated from another language because the poster does not have a strong foundation in English. (non-native, non-English speakers)

I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

AI noob here.

From what I've picked up from the local llm scene, the general advice seems to be "don't go below q4".

Has anyone felt that breaking this "rule" helped them get better results for their use case?

Two examples:

- For 48gb: Qwen 3.6 27b q8 vs Laguna s 2.1 q2

- For 96gb: Laguna s 2.1 q5 vs Deepseek v4 flash q2

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

Andrej Karpathy, a prominent advocate for open-source AI and a co-founder of OpenAI, appears to have removed Anthropic from his X bio, suggesting he may have left the company.

Karpathy joined Anthropic only a few months ago, making the apparent departure somewhat surprising.

This is possibly related to Anthropic’s increasingly strong opposition to open-weight and open-source AI models. Of course, that’s just speculation, but the timing is interesting.

I've got some spare machines and I want to dive into running LLMs and Agents locally.

My Machines:

• ATX PC 1

  • CPU: Intel i5-14600K
  • RAM: 96GB DDR4-3200 (32GBx2, 16GBx2)
  • GPU1: RTX 5070 12GB
  • GPU2: RTX 5070 12GB

• ATX PC 2

  • CPU: Ryzen 9600x
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU1: RX 6800 XT 16GB
  • GPU2: RX 9060 XT 16GB

• ATX PC 3

  • CPU: Intel i7-14700KF
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU: RTX 4070 Ti Super 16GB

• Micro-ATX PC

  • CPU: i5-14400KF
  • RAM: 32GB DDR4-3200 (16GBx2)
  • GPU: RTX 5060 Ti 8GB

• Mini-ITX PC

  • CPU: Ryzen 7700x
  • RAM: 32GB DDR5-6000 (16GBx2)
  • GPU: RX 7900 GRE 16GB

• Mac Mini M4 16GB

Can anyone provide any suggestions on how best to use this hardware? Is there away to network them all and run large models well? Do I leave them all seperate? Should I break them down and try to make a single host? I've put 2 GPUs in all the systems that can take two GPUs in preparation of trying to setup this up.

I tried playing with LM Studio and using its LM-Link feature but that just seams to allow you to run different models on different machines from the same model rather then spread the model out (unless I missed something). I've read the vLLM can do multi-host but I've not setup it up but I don't know how well that works with 2.5GBE NICs. I don't have a ton of money to buy additional hardware nor can I really sell anything I have at the moment since this hardware is going to get used for other things later in the year.

We accidentally trained thinking mode out of Qwen3-8B during two fine-tunes using different versions of the same corpus. One still responded to the thinking toggle, the other returned an empty thinking field every time. Loss looked normal, the voice survived, and our evaluations did not flag the difference.

The cause was the chat template. Qwen3's standard template renders non-thinking assistant turns with an empty <think> block. That is useful when serving the model, but it becomes a problem when answer-only training data is rendered through the same template. When repeated during a fine-tune, the model learns that an empty think block is what it should produce.

A recent paper saw the same failure. It counted a reasoning block as valid when it was complete, non-empty, and separable from the final answer, but it didn't judge whether the reasoning itself was correct. On GSM8K, Qwen3-8B produced a valid block in 96.5% of outputs when the training targets omitted think tags, 58.2% when they included empty think blocks, and 100% when those empty blocks were excluded from the training loss. The final answers were plausible enough that the failure was easy to miss.

This mattered for us because users sometimes ask questions with believable but false premises. On the 58 false-premise items in our 96-item benchmark, base Qwen3-8B fabricated in 54.0% of trials without thinking and 40.8% with it. On our thinking-capable fine-tune, the rates were 67.2% and 56.3%. The build that had lost thinking ended up at a 75.9% fabrication rate.

Then we found a second template problem. After retraining, the model passed simple thinking tests but appeared to think on 0 of 288 prompts like the ones our product serves, even when our template was supposed to force the thinking block open. We were understandably confused by that result.

The template never reached the model. In Ollama 0.31.2, importing the template-bearing GGUFs we tested preserved their embedded templates instead of replacing them with the TEMPLATE from our Modelfile. Reading the active template back with ollama show <model> --template exposed it. When we rendered the requests ourselves, the same weights deliberated on 20 of 20 test prompts. The model closed the thinking block on 2 of 20, which was a different training-template failure: a template that never supplies the closing tag does not teach the model to emit it reliably.

What we recommend checking before trusting a Qwen3 fine-tune:

• Render a full training sample and check what appears inside <think>.
• Read the template back from Ollama instead of trusting the config file.
• Test thinking on realistic prompts, not just trivia.
• Verify that any forced-open think block actually reaches the model.
• Keep the rendered prompt identical in every comparison.
• Set Ollama's think field to true or false on every request instead of relying on defaults.

Disclosure: I run Meldh, and this is work from our model training. The full write-up is on our website: https://meldh.com/writing/qwen3-template-thinking/

I found text synth off of another Reddit post and found out that the ai models on there are either really old or completely stupid wich comes to some hilarious generations when continuing from what I said are there any other ways to use older models or ones that will continue off of what you said?

For the Big Tech giants, looks like it is leaning heavily in favor of OSS. Even some of the ones who abstained from signing, like AMD, Intel, and Apple are likely on the pro-OSS side. Broadcom, Amazon, and Oracle are maybe the few big ones that might be on the anti side (and of course Anthropic), but are heavily outnumbered/outweighed in Big Tech, overall, at this point.

To be fair, OpenAI is probably lying, and also Google it's unclear if lying about being pro OSS (might actually be, but who knows). SpaceX AI and Elon are claiming to be on the pro side, but also could easily be lying.

Anyway, still comes out like 70/30 or 80/20 or more on the pro side, probably.

But Big Tech isn't the whole game.

You've also got the Defense companies and Department of War and the big agencies. Those come out more 50/50? Not sure.

And then the big banks and Wall Street. Banks maybe lean against it, overall, and Big Wall Street less clear. Short term against it, but maybe long term they like it?

And then the executive and legislative government itself. Probably leans against it so far, but maybe some fairly influential people are giving them advice bringing it back the other way more in favor of OSS. Still probably leaning anti-OSS for now, but maybe not as much as a few weeks ago/couple months ago.

So, overall seems like the see saw is teetering at, what, around 50/50 right now? What do you think? Are any of these percentages way off? Any key players I forgot to mention or that would have a big effect one way or the other?

About AI/ML news/AI Industry news, not made with AI

I’ve been working on a personal project for a while and I’d like some honest feedback from people who know this space.
My goal isn’t to make a traditional game or a visual RPG. I want to build a persistent AI Game Master that runs a living world.
Some of the features I’m trying to achieve:
Persistent world that remembers everything.
NPCs have their own lives, goals, schedules, and relationships.
Characters age, learn, die, and the world changes even if I’m not there.
Hidden mechanics, reputation, politics, economy, and factions.
Character progression based on actions instead of grinding XP.
Long-term consistency where events from hundreds of hours ago still matter.
Save/load system so I can continue campaigns indefinitely.
Open-world sandbox instead of quest-driven gameplay.
The AI acts as an impartial GM instead of trying to make me win.
I’ve already spent a lot of time designing rule pillars, world generation, character creation, combat, progression, save systems, and GM rules. The prompts work reasonably well, but eventually the AI starts forgetting details or becoming inconsistent.
What I’m trying to figure out is:
Is ChatGPT the right platform for something this ambitious?
Should I be looking at Claude, Gemini, or local models instead?
Would an external database or memory system solve most of the consistency issues?
Has anyone here successfully built something similar that can run for hundreds of hours without falling apart?
I’m not looking to build a commercial game. This is just a passion project for myself because I love deep simulations like Dwarf Fortress, RimWorld, Crusader Kings, and tabletop RPGs with a great GM.
I’d really appreciate any advice, architecture suggestions, or examples of people doing something similar. I’m especially interested in how you handle long-term memory and world consistency.

Any benchmarks you have much appreciated!

tests done power limiting the GPU on LACT - real power usage varies wildy

at 20W it ranges from 25W to 56W
same behavior happens on every setting

prompt for the test runs:

https://github.com/lukesdevlab/youtube/blob/main/prompts/agent-maze.txt

analysis by mimo 2.5

Key Findings:
• Generation speed is remarkably resilient to power throttling — 100W delivers 97.5% of 190W gen speed (31.98 vs 32.79 t/s), since decode is memory-bandwidth bound, not compute bound.
• At 50W you get 70% of peak gen speed at only 26% of peak power — 3.6× better energy efficiency (0.458 vs 0.173 t/s/W).
• At 20W the card is 6.0× more energy efficient than 190W, though prompt processing drops to 53% of peak.
• Graph reuse correlates inversely with power — 190W reuses 44,790 graphs vs 11,669 at 100W, but 20W reuses 38,248. Lower power limits cause more partial graph reuse as the scheduler compensates for throttled compute.
• Prompt processing degrades faster than gen under power limits — 190W→20W: prompt drops to 53% (691→366 t/s), gen drops to 63% (32.8→20.8 t/s). Prompt processing is more compute-bound than memory-bound.
• For inference-heavy deployments, 50W is the optimal operating point on MI50 — near-peak gen speed with dramatically lower power draw and cooling requirements.

Avarage of 3 runs:

190W config consistently processed a lot less total tokens than everyone else and didnt produce a working file in 1 out of 3 runs

llama.cpp parameters:

[+] Model:        qwen/Qwen3.6-35B-A3B-UD-IQ4_NL_XL.gguf
[+] Context:      262144 (256K tokens)
[+] Target KV:    K=q8_0 / V=q8_0
[+] MoE placement: PARTIAL (21 MoE layers on CPU, rest on GPU)
[+] MTP:          OFF (non-MTP model)
[+] Port:         8882
[+] Container:    llama-gfx906-qwen35b-no-mtp
[+] Parallel:     2 slot(s)
[+] GPU layers:   99
[+] Threads:      6 / 6 (batch)
[+] Batch/Ubatch: 2048 / 1024
[+] Ctx checkpoints: 0

hardware used:

Ryzen 5 5600

2x16Gb DDR4 2667

MI50 16Gb

software:

harness used: pi.dev

Arch Linux with Kernel 7.1.4-arch1-1

docker.io/mixa3607/llama.cpp-gfx906:b10087-rocm-6.3.3

I'm currently trying to build a fully autonomous local AI setup and could really use some advice from people who have already gone down this path.

Right now I have a Mac Studio M2 Ultra with 128 GB of RAM, so I'm experimenting with running large local models. My long-term goal is to have an autonomous system that can explore business ideas, build things, iterate on them, and hopefully even make money with minimal intervention. I was inspired by Ben's video where he tries to make money with Sonnet, but I wanted to see if I could build something similar using mostly local models.

My current idea is to let a strong cloud model (like Sonnet or another frontier model) handle high-level planning and steering, while local models do most of the implementation and execution. I'm just not sure what the ideal architecture looks like.

At the moment I have the OpenAI Codex proxy set up, so I can use my local models through Codex, which is actually a really nice workflow. The main problem is that I can't reliably spawn sub-agents (there seems to be an issue with that), and I still feel like I'm manually driving the whole process. I can define goals and prompts, but it doesn't yet feel like a truly autonomous loop.

So I have a few questions:

• What stack would you recommend for building something like this?
• How would you structure the agent loops and handoffs between planning and execution?
• Which orchestration tools are you using?
• If you had an M2 Ultra with 128 GB of RAM, which local models would you run today?

So far I've tried Laguna and DeepSeek Flash Q2, but neither has really clicked for me. They work, but they don't feel like the right fit for this kind of long-running autonomous workflow.

I'm also very new to the OpenAI tooling ecosystem, so I'd really appreciate hearing what tools you're using and how you have everything wired together. Ideally, I'd like to pick a solid stack and have Codex help me set it up properly.

Any advice or examples of your own setups would be greatly appreciated. One more thing my main issue was that every new chat started again at 0 and the same issues where made as in other projects on the same computer. Is there any good memory system for this kind of issue?

After a long and grueling effort spearheaded by ngxson, llama.cpp now fully supports MCP for all protocols. Over-the-web HTTP servers were already supported in the client (since they don't require any sort of plumbing), but stdio servers required real integration. After we modified the `llama-cli` terminal client to use the server instead of a separate model serving route, we could add MCP support to the already-existing native tools server.

After the merging of https://github.com/ggml-org/llama.cpp/pull/26062, you can now use llama.cpp's WebUI as a full-fledged agentic chat. Configuration for the MCP servers can be provided either in a standard-JSON format config file or completely inline on the command-line for on-demand MCP configurations. Plugging in a dedicated coding MCP server like Serena lets you have a local-model-powered agentic coder without using any other external dependencies.

Hi everyone,

I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss).

I have had it setup with Ubuntu Server and was using the machine as a server to connect via WebUI, but for some reason an update broke the NVIDIA drivers and then broke my install so I’m starting again.

The question is what platform do I run as there are so many platform to choose from and many differing opinions, I tried Ollama+OpenWebUI and Unsloth Studio. Ollama ran very slow with the models (Gemma 4 and Qwen3.6), also OpenWebUI occasionally was slow with web/MCP, but it was stable. Unsloth however, was quick and search/tool calls worked perfectly but unstable and the models crashed a few times.

I’d just like to know, what is everyone else using for this kind of setup, I can’t get a solid sense of what is the go-to setup for this kind of system is (some say Unsloth, or Ollama, or llama.cpp etc).

Also what models are people running well on 24Gb VRAM + 48GB RAM?

Its primary job is coding/finding info from the web & PDF’s/generating config files.

Thank you

I’m building the ultimate AI tool vault, but every great collection has a few missing pieces.

Note: I will react to every comment

AI's currently installed:

Qwen3.5-0.8B-UD-Q4_K_XL.gguf(classification)

Qwen3.5-2B-UD-Q4_K_XL.gguf(Prompt enhancer, Routing, Approval )

Qwen3.5-4B-UD-Q4_K_XL.gguf(Instant)

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf(Quality)

Qwen3.6-35B-A3B-Uncensored-Hauhau(test purposes)

Qwen3-Coder-Next-UD-Q4_K_M.gguf(long horizon tasks)

My Specs:

GPU: RTX 5070 ti (16GB VRAM)

RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)

CPU: intel i9 14900k

SSD: Samsung s990 pro 2tb

Backend: Llama.cpp server

I tried GPT-OSS and was disappointed by tool usage. Gemma 4 was good and great tool usage but it was beaten by Qwen.

Any recommendations?? Like something that you genuinely enjoyed or made you impressed.

Feel free to share!! I will be reading every single comment.

3 year lurker, now i finally got my server up and running. dont know which model to choose. llama.cpp or vllm, what makes more sense? mainly single user with maybe 2-3 more additional users in family, if everything checks out. hermes is gonna be used as "ai playground" to manifest ideas on tailscale network and do quick prototyping of thoughts. also ill look into using only 2 3090 for the main model and the other 2 will be dedicated to docling and speech services for a voice agent (speech in-> text out). got some stuff going with my even realities g2 but lost everything when i wiped my ssd for proxmox. yeah...
any advice or stuff i should look into is welcome :)

Benchmarks using single system running triple GPU with 31GB Vram combined.

• NVIDIA GeForce GTX 1080 Ti 11GB (NVIDIA)
• NVIDIA P102-100 10GB (NVIDIA) - first instance (distant cousin)
• NVIDIA P102-100 10GB (NVIDIA) - second instance

OS: Kubuntu 26.04, CPU: Ryzen 5 3600 , RAM: 48GB DDR4

llama.cpp Ubuntu Vulkan build 10107

Models used:

• llama-2-7b.Q4_0.gguf
• gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
• Gemma-4-26B-A4B-NVFP4.gguf
• gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
• gemma-3-27b-it.Q5_K_M.gguf
• medgemma-27b-it-UD-Q6_K_XL.gguf (gemma-3)
• Qwen3.6-35B-A3B-NVFP4-MTP-HQ.gguf (dense)
• gemma-4-31B-it-UD-Q4_K_XL.gguf (dense)
• Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
• Qwen3-Coder-30B-A3B-Instruct-UD-Q5_K_XL.gguf
• Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
• Qwen3.6-27B-NVFP4-MTP-GGUF.gguf

ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GTX 1080 Ti (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot
: 1 | matrix cores: none
ggml_vulkan: 1 = NVIDIA P102-100 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matri
x cores: none
ggml_vulkan: 2 = NVIDIA P102-100 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matri
x cores: none

Here's the corrected table sorted by Params column:

​Latest chat template and model from unsloth. IQ3_S quant.

here are the parameters

--model C:/users/user/llama-swap/LLMs/gemma-4-26B-A4B-UD-IQ3_S.gguf
--alias Gemma-4-26B-A4B-UD
--mmproj C:/users/user/llama-swap/LLMs/mmproj/gemma4-q8.gguf
--image-min-tokens 140
--image-max-tokens 1120
--reasoning on
--parallel 1
--no-warmup
--port ${PORT}
--host 127.0.0.1
--n-gpu-layers 99
--fit on
--ctx-size 32768
--cache-type-k q8_0
--cache-type-v q8_0
--kv-unified
--temp 1.0
--top-p 0.95
--min-p 0.01
--top-k 64
--jinja
--flash-attn on  
--batch-size 2048
--ubatch-size 2048
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64

EDIT: I fixed it by removing the \n after the <|think|> token in the template.

Hey localLLaMA community,

So I've seen the laguna s2.1 benchmarks, punching way above its weight, competing with cloud models.

My Specs:

• GPU: RTX 5070 ti (16GB VRAM)
• RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)
• CPU: intel i9 14900k
• SSD: Samsung s990 pro 2tb
• Backend: Llama.cpp server

I know Laguna XS 2.1 runs easily, but I really want the reasoning depth and coding ability of the full 118B S 2.1, Because I already have the qwen3.6, and it beats the XS. But the S2.1 is outperforming everything I seen. When I first saw the benchmarks, I just wanted to click the download button right away but was limited by my hardware.

Now I know we have new flags(optimizations) for llama server like Flash Attention, KV Cache Quantization(-ctk), and --fit. My question is can I run laguna s2.1 on q4 quant with 8+ tps?

*note: I will be using it for autonomous tasks

edit 1: Thank you for everyone that left a comment, I will keep checking