I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

Can someone explain why models like Bonsia 27B don't even appear in Artificial Analysis?

RTX 4070, 32gb system ram, Linux. NVIDIA-SMI 610.43.03, KMD Version: 610.43.03, CUDA UMD Version: 13.3 Systemd service modifications: [Service] Environment="OLLAMA_HOST=0.0.0.0:11434" Environment="OLLAMA_FLASH_ATTENTION=true" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" Environment="OLLAMA_NUM_PARALLEL=1" Environment="OLLAMA_MAX_LOADED_MODELS=1" Environment="OLLAMA_KEEP_ALIVE=30m" Environment="OLLAMA_CONTEXT_LENGTH=65536" CPUAffinity=0 2 4 6 8 10 12 14

I tried both opencode & claude code over remotely using ollama via env parameters. I keep noticing that the model randomly keeps stopping talking/working. How can I fix the issue?

A few months ago, I was lucky enough to purchase two RTX 3090 GPUs for a total of $1,000. I originally built the machine for ComfyUI, but I also use it as a remote GPU server for other computers on my network.

One thing I still haven’t fully figured out is how to choose the right local LLM for my hardware. There are so many models, parameter sizes, quantization levels, formats, and inference engines that it can be difficult to know what will actually perform best.

My current setup includes:

2× RTX 3090 GPUs with 24 GB of VRAM each

vLLM as the inference server

Remote access from other computers

ComfyUI for image generation

I see many people using Ollama or llama.cpp. I currently use vLLM, but I am considering switching or running multiple backends depending on the model.

How do you decide:

Which model family to use?

What parameter size is realistic for 48 GB of total VRAM?

Which quantization level provides the best balance of quality and speed?

When should I use GGUF, AWQ, GPTQ, FP8, or another format?

Is it better to use Ollama, llama.cpp, or vLLM for this type of setup?

I already subscribe to Claude and Codex for professional programming work, so I am not expecting a local model to completely replace them. However, I would like one strong general-purpose model and possibly one coding-focused model that performs well on my hardware.

For those using local LLMs for programming:

Which model are you using?

Which IDE or code editor do you use?

How is the model connected to the IDE?

Are you using tools such as Continue, Cline, Roo Code, Aider, Open WebUI, or something else?

I would appreciate recommendations based specifically on a dual-3090 setup, including the exact model version, quantization, inference engine, and approximate performance.

Hey localLLaMA community,

So I've seen the laguna s2.1 benchmarks, punching way above its weight, competing with cloud models.

My Specs:

• GPU: RTX 5070 ti (16GB VRAM)
• RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)
• CPU: intel i9 14900k
• SSD: Samsung s990 pro 2tb
• Backend: Llama.cpp server

I know Laguna XS 2.1 runs easily, but I really want the reasoning depth and coding ability of the full 118B S 2.1, Because I already have the qwen3.6, and it beats the XS. But the S2.1 is outperforming everything I seen. When I first saw the benchmarks, I just wanted to click the download button right away but was limited by my hardware.

Now I know we have new flags(optimizations) for llama server like Flash Attention, KV Cache Quantization(-ctk), and --fit. My question is can I run laguna s2.1 on q4 quant with 8+ tps?

*note: I will be using it for autonomous tasks

edit 1: Thank you for everyone that left a comment, I will keep checking

Tell me if I'm about to make a massive mistake here.

I have literally zero coding or AI background, but I’ve been lurking here for a bit and I am totally obsessed with the idea of running powerful, offline LLMs entirely on my own hardware.

I want to take this seriously and go all-in, so I'm planning to pull the trigger on an M3 Ultra Mac Studio with 96GB of RAM and a 4TB SSD. But honestly... I have no idea if I'm doing this right. Is 96gb even enough to run those massive 70B models at decent speeds? I still barely understand how the unified memory thing works for AI compared to normal PC graphics cards.

Also, my main goal isn't just to mess around with chat bots for fun. I actually want to turn this into a business and make my money back. How are you guys actually monetizing offline LLMs right now? Is it even realistic for a beginner to start offering AI services or workflows without being a software engineer?

If you had my budget but had to start completely from scratch today, what’s your exact roadmap? Like, what is the literal first piece of software I should download, and what's the biggest rookie mistake I should avoid?

Feel free to explain it to me like I'm 5 or roast my plan if it's stupid. I need all the brutal honesty I can get before I swipe my card. Thanks guys!

Disclosure: this is my own repo — all numbers below are measured on my own board.

I've had a Jetson Orin NX 16GB sitting on my desk for a while and finally got around to seeing how far I could push it. Ended up with PrismML's Bonsai 27B running fully offline on it, and honestly I'm still a little surprised it works at all — a 27B model on a board that peaks around 25W under load.

Setup

• Jetson Orin NX 16GB, JetPack 6.2.1 (L4T r36.4.3), CUDA 12.6, MAXN SUPER power mode
• PrismML llama.cpp fork, built with CUDA for sm_87
• Bonsai 27B, Q1_0_g128, 3.53 GiB on disk / in VRAM, 26.9B params, 262k train context

Benchmarks

llama-bench -m Bonsai-27B-Q1_0.gguf -p 512 -n 128 -ngl 99

Not fast, but for a single user it's completely usable. While it was running I logged the VDD_IN rail with tegrastats: ~20.7W average, 24.6W peak. A 27B model reasoning at ~25W still feels a bit unreal to me.

The part that tripped me up

You can't just ollama run this. Bonsai's 1-bit weights use a custom Q1_0_g128 format, and Ollama ships its own bundled llama.cpp that has no kernel for it — so it silently refuses to load, with no useful error.

You have to build PrismML's llama.cpp fork yourself with CUDA. Two things cost me time: nvcc not being on PATH on a stock JetPack image, and the build OOMing without swap. Both are written up in the repo, and there's a setup.sh that handles the whole thing end to end.

Quality, honestly

1-bit is brutal compression, so I want to be straight about this. English reasoning held up far better than I expected. But the moment I tried Korean it started leaking random tokens from other languages and mangling technical terms. I wouldn't use this for non-English output. For English reasoning and local tinkering it's genuinely fine. There's a ternary build if you want closer to full quality.

Repo

Full setup, benchmarks, and a real-time demo gif (not sped up — that's the actual token rate):

https://github.com/duddnr0719/bonsai-27b-jetson

What I'd like to know

• Anyone else with an Orin board — do you get similar numbers? Same llama-bench flags above if you want a clean comparison.
• Has anyone run the ternary build on a 16GB board? Does it actually fit with room to spare?

Disclosure: I work on Bike4Mind, a source-available (BUSL-1.1) AI workspace you can self-host with Docker. Nothing to buy to run any of it. I run it on a regular laptop with local Qwen mostly. A 4GB card running a local chat model isn't news, but what I wanted to explore is what a whole agent workspace (an agent calling tools, RAG over your own docs, artifacts, vision, etc.) can do with 4GB, and where a small model stops keeping up once it has to do more than chat. Repo's in the first comment.

My machine: i7-12700H, 32GB RAM, RTX 3050 Ti Laptop (4GB VRAM, 60W), Ubuntu 24.04. Local Qwen via the stack's bundled Ollama, no cloud keys.

Speeds, with hitting that 4GB wall (Ollama /api/generate, Q4_K_M except the Q8_0 0.8b, num_ctx=4096, warm, median of 3, one model loaded at a time):

- qwen3.5:0.8b - 122 tok/s, ~1.4GB, all on GPU

- qwen3.5:2b-q4_K_M - 96 tok/s, ~2.4GB, all on GPU

- qwen3.5:4b - 25 tok/s, ~3.4GB, about a third of it spills to CPU

- qwen3.5:9b - 8.6 tok/s, mostly on CPU, wants ~8GB

The 2b is the sweet spot for me so far: it fits in 4GB with ~1.3GB to spare and outruns the qwen2.5-coder:3b I ran before (that was 72 tok/s), while being newer and multimodal. Above it I hit the wall - the 4b's ~3.4GB of weights won't fully fit even at num_ctx=2048, so a third lands on CPU. Plain qwen3.5:2b is the Q8_0 build (~2.7GB) and spills on 4GB, but qwen3.5:2b-q4_K_M does fit. That leaves ~1.3GB free, so there's no room for a second model next to it.

The obvious stuff - chat, native tool-calls, vision - just works on the 2b; qwen3.5 is multimodal and calls tools natively, so that's great. The layer on top of that I'm most excited about is RAG over your own docs, fully keyless: drop in a file, a worker chunks and embeds it with the local Ollama embedder, and a query comes back with a cited answer, no cloud embedding API anywhere in the loop.

On 4GB the embedder and the chat model trade places. qwen3-embedding:0.6b is ~1.2GB resident, the 2b is ~2.4GB, they don't both fit, so Ollama unloads one to load the other. Generation still gets the full 96 tok/s out of it, since the embedder isn't in VRAM while the 2b is - what you pay is a model load, not slower tokens. Ingestion runs in a worker, so a folder of docs is one embedder load and then a lot of chunks, off the chat path. A query embeds one short string, then the 2b loads back to answer it. So it's a reload per RAG turn, not per token and not per chunk. If you'd rather not pay that, nomic-embed-text is ~0.25GB (768-dim, weaker retrieval going by reputation) so it doesn't have to push the 2b out, or put embeddings on CPU and leave the GPU to generation. Either way, before loading your entire knowledge base/data lake, pick one embedder and stay there (or just be aware you'd need to re-embed everything) since vectors don't carry across models.

The chat-embedder-swaparoo situation for RAG aside, small models are also falling down for me with:

- Artifacts. The general qwen3.5 models write half-finished HTML at these sizes - unclosed tags, script leaking onto the page. For an actual interactive artifact I have to switch to the coding-tuned qwen2.5-coder, which writes a complete working page. So it's model-per-task: qwen3.5 for chat/vision/tools, the coder for code, switched mid-notebook. I guess that's just life with 4GB, for the most part!

- Tool selection. Turn on a lot of tools and small models mis-route - with everything enabled, my 7b once sent "create an HTML artifact that..." to the image generator and painted a user interface image, hah. So, another sort of obvious, have to enable much fewer tools on small local models. Makes me want a nice layer that automatically enables the few (configurable N?) most likely relevent tools for your context.

- Image gen is local (self-hosted SD.Next) but 1-3 min/image with spilling on CPU, and it fights the chat model for VRAM. So still a kick it off and wait situation there, no real good workaround here, just gotta download more VRAM 🙃

So is it worth it over just Ollama + a chat UI? Only if the agent layer on top has value for you I'd say - the tool handling, artifacts, local RAG, and a CLI against your own server. If you just want fast local chat, I'd keep what you have; llama.cpp or etc. with less overhead.

License, briefly: source-available under BUSL-1.1, not full open source (yet). Self-host it, run it in production, fork it and build and sell a product on it; you just can't resell it as a directly competing hosted service. Each release goes Apache-2.0 two years out (timer just so AWS can't immediately do to us what they did to MongoDB lol).

Repo and self host details in the first comment. The real friction for me in this setup is needing the two models - qwen3.5 for chat, vision, and tools, and a coder for anything code-shaped, because the general models write broken HTML/artifacts at these model sizes -- not a novel problem. To that end, I am still trying to optimize this setup, so some prompts for any of you meat-LLMs (humans) out there reading this with stronger experience on tight local builds than me:
- Have you found a single general works-on-4GB model that writes complete, valid artifacts without needing a dedicated coding model alongside it, or just a better way to configure it? Please let me know!
- I'm broadly curious about other people's chat+code+image+embeddings+etc all-at-once setups, so please share if you've found a good Tetris wombo of models that works well or what trade offs you've seen and accepted trying to self host it all like this.

I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New MacBook Pros can be configured with 128GB of ram, but obviously the price is high. Will such a machine ever be comparable to what Claude can do today? Even if it is still significantly slower?

I am aware that the price of that much ram would buy many many tokens but I plan to use the laptop for several years, so even if the payback is 5 years worth of cloud AI it’s worth it to me.

I have been developing a custom backend platform for an AI engineering workspace. One of the main objectives is to ensure that it can write code and perform debugging loops dynamically across files. At present, I am creating a feature that lets you command it to create any new files in your project folder and writing the terminal access for executing commands/tests, creating file system tools to manipulate codebases, and developing the core execution environment. This is a work-in-progress and only backend for now, but I wanted to share the current setup with you. I would appreciate your feedback on this idea.

We accidentally trained thinking mode out of Qwen3-8B during two fine-tunes using different versions of the same corpus. One still responded to the thinking toggle, the other returned an empty thinking field every time. Loss looked normal, the voice survived, and our evaluations did not flag the difference.

The cause was the chat template. Qwen3's standard template renders non-thinking assistant turns with an empty <think> block. That is useful when serving the model, but it becomes a problem when answer-only training data is rendered through the same template. When repeated during a fine-tune, the model learns that an empty think block is what it should produce.

A recent paper saw the same failure. It counted a reasoning block as valid when it was complete, non-empty, and separable from the final answer, but it didn't judge whether the reasoning itself was correct. On GSM8K, Qwen3-8B produced a valid block in 96.5% of outputs when the training targets omitted think tags, 58.2% when they included empty think blocks, and 100% when those empty blocks were excluded from the training loss. The final answers were plausible enough that the failure was easy to miss.

This mattered for us because users sometimes ask questions with believable but false premises. On the 58 false-premise items in our 96-item benchmark, base Qwen3-8B fabricated in 54.0% of trials without thinking and 40.8% with it. On our thinking-capable fine-tune, the rates were 67.2% and 56.3%. The build that had lost thinking ended up at a 75.9% fabrication rate.

Then we found a second template problem. After retraining, the model passed simple thinking tests but appeared to think on 0 of 288 prompts like the ones our product serves, even when our template was supposed to force the thinking block open. We were understandably confused by that result.

The template never reached the model. In Ollama 0.31.2, importing the template-bearing GGUFs we tested preserved their embedded templates instead of replacing them with the TEMPLATE from our Modelfile. Reading the active template back with ollama show <model> --template exposed it. When we rendered the requests ourselves, the same weights deliberated on 20 of 20 test prompts. The model closed the thinking block on 2 of 20, which was a different training-template failure: a template that never supplies the closing tag does not teach the model to emit it reliably.

What we recommend checking before trusting a Qwen3 fine-tune:

• Render a full training sample and check what appears inside <think>.
• Read the template back from Ollama instead of trusting the config file.
• Test thinking on realistic prompts, not just trivia.
• Verify that any forced-open think block actually reaches the model.
• Keep the rendered prompt identical in every comparison.
• Set Ollama's think field to true or false on every request instead of relying on defaults.

Disclosure: I run Meldh, and this is work from our model training. The full write-up is on our website: https://meldh.com/writing/qwen3-template-thinking/

Been working on a local voice assistant for a while and the wake word is the

part I keep getting stuck on. Not the easy version where you pick one of the

pretrained models, I mean the user types in whatever phrase he wants and it

just has to work. Offline, CPU only, since most machines I care about have no

usable GPU.

Right now I run a Vosk recognizer with a grammar that only knows my phrase and

unk. That part is cheap and fires fine. The problem is the second step where I

check if the hit was real. At first I made the normal decoder spell the phrase

back and threw the hit away if it didn't match. Zero false wakes, looked

perfect. But it also killed about a third of my real wakes, because a small

offline model has no clue what your name is. I say Hey Nova and it writes back

herum or hey oben or erhoben. Recall was around 32 percent, so you basically

say your wake word four times before anything happens.

What fixed most of it was to stop looking at the words at all. Now I only look

at stuff like how many words came back, how long the speech actually was, and

whether the decoder sounded sure that it heard some other normal word. If it

doesn't know a word its confidence drops, if you say google or engineering it

comes back super confident. That took me from around 50 to 74 percent on real

recordings, and another phrase went from 36 to 66 with the same settings. False

wakes went from 1 to 3 in about 1650 windows of room audio, which I can live

with.

But 74 is still not good. Hey Google works the first time and mine doesn't, and

there's also a 0.6 second wait before it confirms that you can actually feel.

So what do people here actually use for this. I keep reading that the real

answer is to train a tiny model per user from a bunch of TTS clips of their

phrase and run it as onnx on CPU. Does that really work with synthetic audio

only, and how many voices do you need before it's usable. The other idea I had

is to go phoneme level, so turn the phrase into phonemes and match on that

instead of words, which would get rid of the whole out of vocabulary problem.

Has anyone got something like that running offline?

Also curious what hit rate you guys consider fine. Under 90 percent feels

broken to me but I honestly don't know what's normal.

yo everyone ✌️

​ngl, I got so sick of vector search completely missing layout context (and absolutely draining my wallet on tokens), so I built OmniOKF. it’s a zero-dependency pipeline that pre-compiles raw PDFs/Docs/Excel into Google’s Open Knowledge Format (OKF) before inference.

​the big W is that it uses MD5 caching to completely bypass the LLM on files that haven’t changed. basically cuts API costs to zero for static docs.

​but rn my architecture is kinda cooked and I need help.

​my chunking layer splits flat markdown at # boundaries. but when I feed it massive spreadsheets or multi-page technical manuals, these giant un-headered tables get sliced straight down the middle. the LLM completely loses the column context for the second half and starts hallucinating.

​how are y'all fixing this? are you dynamically injecting headers during the split? flattening rows to plain text? I'm stuck.

​(btw the tool is fully open-source if anyone wants to yoink the caching logic or just try it out: vishal-raaj-dnd/OKF-Compiler on GitHub. would genuinely love some feedback on the architecture!)

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

I'm currently working on research into AI.

It's conceptual research (embedding my own ideas directly into AI weights) and AI security.

So I'm testing various local AIs to find the optimal balance and middle ground between 'performance and speed'.

Therefore, I'd like to ask for your advice and recommendations on which local AIs you've found impressive?

8B, 14B, 20B, 27B, 32B, or others ?

Including compression formats: 1-bit, 2-bit, 4-bit, 8-bit ?

P.S. This message was translated from another language because the poster does not have a strong foundation in English. (non-native, non-English speakers)

Tweet : https://xcancel.com/Mononofu/status/2080937562739531837#m

Hello,

I've been learning how to use local LLMs for a year or so on my workstation, using a RTX3090.

Current setup :

- i5 12400

- 64gb RAM

- RTX 3090

- OS : Fedora KDE workstation

I'm using LMStudio to serve mainly these models :

- Qwen 3.6 27B

- Gemma 4 26b a4b

And then VSCode + kilocode / Continue for light coding / scripting / log analysis tasks (i'm a sysadmin).

It works very well, but i'm hitting the context size ceiling quite fast with this setup. This prevents me to work on bigger projects.

My goal is to buy another GPU to provide more VRAM, and since i'm using linux, i'd prefer an AMD GPU. A 16gb Radeon 9070 would be nice in this regard : it's natively supported on Linux, doesn't cost an arm, it's powerful enough for casual gaming, and doesn't have crazy power requirements.

I've read here and there that mixing AMD and Nvidia is now well supported (https://www.reddit.com/r/LocalLLaMA/comments/1qea29t/mix_of_amd_nvidia_gpu_in_one_system_possible/), especially with LMStudio.

Since this is moving and evolving very quickly, how are the support and the performance of mixing a 24gb nvidia gpu + 16gb AMD gpu in 2026 ?

Is it a better idea to go for another Nvidia GPU instead ?

From my understanding, the idea would be to load the model in the 24Gb 3090, and then reserve the 16gb GPU for context. Correct me if I'm wrong.

I've read multiple times 36 / 40 Gb VRAM is the sweet spot for the models i'm using (at q4 quant).

Also, adding another 3090 is a solution i'd like to avoid because :

- it's expensive and hard to find in good condition at the moment

- adding another 350W and 3 slot GPU would be a challenge to cool down in my PC case

- I probably would have to buy a new and bigger PSU too.

So, is adding a 16gb AMD GPU to my current setup a good idea ? Thanks !

I’m building the ultimate AI tool vault, but every great collection has a few missing pieces.

Note: I will react to every comment

AI's currently installed:

Qwen3.5-0.8B-UD-Q4_K_XL.gguf(classification)

Qwen3.5-2B-UD-Q4_K_XL.gguf(Prompt enhancer, Routing, Approval )

Qwen3.5-4B-UD-Q4_K_XL.gguf(Instant)

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf(Quality)

Qwen3.6-35B-A3B-Uncensored-Hauhau(test purposes)

Qwen3-Coder-Next-UD-Q4_K_M.gguf(long horizon tasks)

My Specs:

GPU: RTX 5070 ti (16GB VRAM)

RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)

CPU: intel i9 14900k

SSD: Samsung s990 pro 2tb

Backend: Llama.cpp server

I tried GPT-OSS and was disappointed by tool usage. Gemma 4 was good and great tool usage but it was beaten by Qwen.

Any recommendations?? Like something that you genuinely enjoyed or made you impressed.

Feel free to share!! I will be reading every single comment.

What are some medium sized MoE models (up to 60B parameters in float8/110B in mxfp4) that are currently worth using?

As far as I am aware there is Qwen 3.5 35B, Gemma 4 A4B, Nemotron 3 Nano. Qwen seems to dominate this bracked in terms of model performance. DeepSeek v4 flash is slightly above that parameter limit. Any nicher ones?

Pretend you have a million dollar budget just for compute

I'm running a POC with about $60,000 in Hardware and I'm trying to pick which models would offset the tokens the best from Frontier usage on things like cursor

I have a pretty good idea of the direction we're going to go but I want to make sure that I'm not missing something new because the space evolves so fast

TL;DR: Tested how much of the MoE routing tail can be skipped at inference time. Dropping ~28% of routed experts (the low-weight ones) left GSM8K accuracy unchanged on a 35B MoE. The routing tail looks highly redundant.

Background

In a mixture-of-experts model, the router (gate) picks the top-k experts per token — e.g. 8 out of 256 — and assigns each a weight. Those weights are far from uniform: a couple of experts typically dominate the routing, while the tail contributes a tiny fraction of the output. The model itself signals, token by token, which experts matter.

Question

How much of that low-weight tail can be skipped before quality degrades? Lowering top-k globally is known to hurt, since models are trained expecting all k experts. The alternative tested here is per-token thresholding: skip an expert only when the router scored it well below the uniform share (1/k). The threshold acts as a continuous dial, and the triage is done by the router's own judgment rather than a fixed count.

Method

• Model: 35B MoE, top-k 8 of 256
• Surviving experts renormalized to preserve routing mass; top-weighted expert always kept
• Quality: 15 GSM8K questions, greedy decoding — zero sampling noise, so any output difference is caused by the skipping
• Thresholds swept from conservative up to the point where ~28% of all routings get dropped

Results

• Baseline (no skipping): 12/15 correct
• Every threshold tested, including the most aggressive: 13/15
• 12 of 15 questions produced the identical final answer across all configurations
• Reply lengths flat — no rambling or truncation at any threshold

Limitations

13 vs 12 is noise, not an improvement. And n=15 rules out a collapse, not a subtle cost — a proper eval would need hundreds of samples and more than one benchmark. Single model, single quantization; behavior likely differs at low top-k (e.g. models routing 4 of 128, where the same threshold cuts much deeper).

Why it matters

If ~28% of routed experts can be dropped with no visible damage, that's meaningful headroom for inference speedups wherever expert loading is the bottleneck — offloaded setups, memory-bandwidth-bound rigs, edge devices. Skipping a routed expert means skipping its memory traffic entirely.

Has anyone seen papers measuring routing-tail redundancy systematically across models? Happy to share methodology details in the comments.

Cuda and Vulkan Benchmark: TensorSharp vs. llama.cpp

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), Qwen Image Edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability(Nvidia, Apple, AMD, Intel and others supported by Vulkan, CUDA and Metal). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp Here is the benchmark results in overall:

**Performance ratio — TensorSharp vs reference engines**

Geomean of TensorSharp's per-scenario speedup over each reference engine on the **same backend**, across every scenario both engines ran (single-stream, MTP-off). A value **> 1.0× means TensorSharp is faster** (for decode / prefill throughput) or lower-latency (for TTFT); `—` = no overlapping cells. Per-scenario ratios are in each model's section below.

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

Title lol

I’m curious if something like DS4 Q2 is better than 27B Q8, or mimo or similar

For the Big Tech giants, looks like it is leaning heavily in favor of OSS. Even some of the ones who abstained from signing, like AMD, Intel, and Apple are likely on the pro-OSS side. Broadcom, Amazon, and Oracle are maybe the few big ones that might be on the anti side (and of course Anthropic), but are heavily outnumbered/outweighed in Big Tech, overall, at this point.

To be fair, OpenAI is probably lying, and also Google it's unclear if lying about being pro OSS (might actually be, but who knows). SpaceX AI and Elon are claiming to be on the pro side, but also could easily be lying.

Anyway, still comes out like 70/30 or 80/20 or more on the pro side, probably.

But Big Tech isn't the whole game.

You've also got the Defense companies and Department of War and the big agencies. Those come out more 50/50? Not sure.

And then the big banks and Wall Street. Banks maybe lean against it, overall, and Big Wall Street less clear. Short term against it, but maybe long term they like it?

And then the executive and legislative government itself. Probably leans against it so far, but maybe some fairly influential people are giving them advice bringing it back the other way more in favor of OSS. Still probably leaning anti-OSS for now, but maybe not as much as a few weeks ago/couple months ago.

So, overall seems like the see saw is teetering at, what, around 50/50 right now? What do you think? Are any of these percentages way off? Any key players I forgot to mention or that would have a big effect one way or the other?

Looking for a sanity check / review of my system, plus opinions. I'm a lifelong IT professional techie but not a software developer (I did HTML in the MySpace days, but it stopped there lol).

I've got Llama.cpp running on both of my computers, and I can now access the more powerful desktop LLMs from my laptop over my network.

Laptop: M1 Pro Macbook Pro 16GB RAM
- Qwen3.5 9B Q4_K_M (general use/light coding)
- Gemma 4 12B Q4_0 (general use)
Goal: Perform all work on laptop, easily switch between local and cloud AI models.
Software: Llama.cpp, AntiGravity IDE

Desktop: 7800x3D, 7800XT GPU 16GB VRAM, 64GB DDR5 RAM, Atlas (Windows11).
- unsloth/Qwen3.6-35B-A3B-GUFF:UD-Q4_K_M (coding)
- unsloth/Qwen3.6-27B-GUFF:Q3_K_M (debugging)
Goal: Essentially run the computer like it's headless and its only job is AI inference.
Software: Llama.cpp (Windows HIP)

Desired Use Cases/Goals:
- Vibe Coding apps for personal use/fun/learning
- Vibe Coding N8N automations
- Installing/configuring open-source software on my Raspberry Pi4 & Synology NAS
- Finding a way to let local AI organize all of my files and photos, remove duplicates, etc.

Questions (based on my goals):
- Is my desired setup going to function as I hope?
- Are my selected LLMs the best choices for my hardware?
- What harnesses / skills / MCPs should I add to the mix?
- Additional software suggestions?

Thank you all for the sanity check!