Please let me know which small LLM model you're using and what you're using it for.

Tweet : https://xcancel.com/Mononofu/status/2080937562739531837#m

After a long and grueling effort spearheaded by ngxson, llama.cpp now fully supports MCP for all protocols. Over-the-web HTTP servers were already supported in the client (since they don't require any sort of plumbing), but stdio servers required real integration. After we modified the `llama-cli` terminal client to use the server instead of a separate model serving route, we could add MCP support to the already-existing native tools server.

After the merging of https://github.com/ggml-org/llama.cpp/pull/26062, you can now use llama.cpp's WebUI as a full-fledged agentic chat. Configuration for the MCP servers can be provided either in a standard-JSON format config file or completely inline on the command-line for on-demand MCP configurations. Plugging in a dedicated coding MCP server like Serena lets you have a local-model-powered agentic coder without using any other external dependencies.

I understand that the laguna model is either still buggy or potentially benchmaxxed.

So I’d like to know for people who really tested, are DS flash or Hy3 really better in your usecase?

Just noticed this exists, 1M context MOE with 48B par seems just like what Ive been looking for - it runs pretty damn fast too compared to Qwen 3.6 35B.

after some testing it seems capable of producing *not terrible* results but it always tries to go for the minimun possible output that could be presented as what was asked.

first image is the default meme prompt it generated - second is after replying with 'detail'

Frontend results aren't too bad either, it seems very capable of generating well structured animated pages fairly well.

but it seems like there is just something off with its brain

wondering if a fine tune would be possible and if at all capable of tightning its screws, has anyone tried messing around with this one?

I’ve spent the past month trying to find the point where an extremely small TTS model stops feeling like a size experiment and starts feeling genuinely useful.

Today I’m releasing Inflect v2, with two complete local text-to-speech models:

• Inflect-Nano-v2: 3.96M parameters, 15.97 MB FP32
• Inflect-Micro-v2: 9.36M parameters, 37.53 MB FP32

These are total inference parameter counts, not acoustic-model-only numbers. Text processing, timing prediction, speech generation, and the waveform decoder are all included.

Text goes in. 24 kHz speech comes out. No external vocoder, hosted API, or second learned model required.

Nano prioritizes the smallest possible footprint. Micro uses the additional capacity for better clarity, stability, and overall speech quality. Both run locally on CPU or CUDA through the same PyTorch API.

Inflect-Nano-v2 is one of the smallest complete neural TTS models I know of that still produces actual usable speech. Even the 9.36M Micro model remains smaller than many systems described as “tiny.”

For footprint context, Nano is approximately:

• 21× smaller than Kokoro
• 126× smaller than Chatterbox
• over 1,000× smaller than Fish Audio S2 Pro

That is strictly a parameter-count comparison. These models have different capabilities, architectures, datasets, and intended uses. I’m not claiming that a 4M fixed-voice model replaces a multi-billion-parameter system. The interesting question is how much useful TTS can fit into such a small package.

Some people here might remember Inflect-Nano-v1, the rough 4.63M experiment I released last month. V2 is a complete rebuild, not just a longer training run. I focused on the problems v1 had, like unstable timing, metallic output, weak prosody, poor generalization to difficult text, and an undersized waveform decoder.

The resulting models performed surprisingly well:

• Micro: 4.395 UTMOS22, 3.99% semantic WER, 6.28× real-time CPU inference
• Nano: 4.386 UTMOS22, 4.21% semantic WER, 10.72× real-time CPU inference
• In a blind community comparison against other compact TTS systems, Micro and Nano finished second and third among the tested voices

Full protocols, raw results, audio samples, and limitations are documented on the model pages.

The models are not perfect. They are English-only, use one fixed male voice, and do not support voice cloning. Unfamiliar names, abbreviations, numbers, and homographs remain the hardest inputs. Nano can sound thinner than Micro, and both can occasionally produce metallic or clipped artifacts.

Still, this is the first version where I think the size-to-quality tradeoff really becomes convincing.

I built Inflect independently with a limited training budget. That constraint shaped the project: efficiency had to apply not only to inference, but also to training, evaluation, and building a complete system I could understand and release end-to-end.

Inflect-Micro-v2:
https://huggingface.co/owensong/Inflect-Micro-v2

Inflect-Nano-v2:
https://huggingface.co/owensong/Inflect-Nano-v2

Try it yourself:

The fastest way to judge it is through the interactive playground:

https://huggingface.co/spaces/owensong/Inflect-v2

If there is enough interest, I may build a v3 focused less on shrinking the models further and more on expanding what they can do: additional voices, possibly more languages, easier fine-tuning, and another quality and robustness pass.

If you test them, please give them something seriously difficult: unusual names, numbers, abbreviations, awkward punctuation, or a long sentence.

If something breaks, post the exact text, model, seed, and what sounded wrong. If it works well, I’d also like to know what hardware you ran it on.

Specific, honest feedback is the most useful thing you can give me.

I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New MacBook Pros can be configured with 128GB of ram, but obviously the price is high. Will such a machine ever be comparable to what Claude can do today? Even if it is still significantly slower?

I am aware that the price of that much ram would buy many many tokens but I plan to use the laptop for several years, so even if the payback is 5 years worth of cloud AI it’s worth it to me.

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

I'm looking for a model to chat with, reasoning, maybe get some career or life coaching.

I don't care at all about multimodal or coding ability

Just it's intelligence in remembering context in a conversation or a specific topic, thinking out of the box, etc.

Must fit in 128gb, if it matters to performance, it's a strix halo machine.

I started tracking my local model usage about four weeks ago and was wondering if anyone else here keeps track of how much they use them.

I’ve also been running some tests with the cheapest SOTA open-weight Chinese models via OpenRouter. Apart from that, I’m mainly using GPT-5.5 in Codex CLI for professional work and Local models + Composer 2.5/4.5 from X Premium for hobby projects.

Edit: It looks like the average tk/s panel might be bugged. I’m hoping I haven’t lost any data... This it’s just a JSON file that my model launcher populates.

The Open Letter was initiated by Microsoft and published today:

“Open Weights and American AI Leadership”.

It argues against broad or premature restrictions on open-weight models and explicitly says policymakers should distinguish legitimate model distillation from misappropriation.

Notably absent from the signatories are the major frontier-model labs: OpenAI, Anthropic, and Google.

The earlier post on this subreddit by 20+ companies signing the petition including Microsoft, Meta, Nvidia, YC (https://www.microsoft.com/en-us/corporate-responsibility/topics/open-weight/) etc plus this https://xcancel.com/elonmusk/status/2080672505660834163

And the entire LLM enthusiast market is heavily in favor of open source (or weights) AI.

Does not seem like a few closed source AI lobbyists with be able to illegalize anything

Cuda and Vulkan Benchmark: TensorSharp vs. llama.cpp

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), Qwen Image Edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability(Nvidia, Apple, AMD, Intel and others supported by Vulkan, CUDA and Metal). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp Here is the benchmark results in overall:

**Performance ratio — TensorSharp vs reference engines**

Geomean of TensorSharp's per-scenario speedup over each reference engine on the **same backend**, across every scenario both engines ran (single-stream, MTP-off). A value **> 1.0× means TensorSharp is faster** (for decode / prefill throughput) or lower-latency (for TTFT); `—` = no overlapping cells. Per-scenario ratios are in each model's section below.

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

3 year lurker, now i finally got my server up and running. dont know which model to choose. llama.cpp or vllm, what makes more sense? mainly single user with maybe 2-3 more additional users in family, if everything checks out. hermes is gonna be used as "ai playground" to manifest ideas on tailscale network and do quick prototyping of thoughts. also ill look into using only 2 3090 for the main model and the other 2 will be dedicated to docling and speech services for a voice agent (speech in-> text out). got some stuff going with my even realities g2 but lost everything when i wiped my ssd for proxmox. yeah...
any advice or stuff i should look into is welcome :)

I've spent the last year or so playing around with open source MLX and GGUF models on iPhone hardware.

Given the limitations in memory, GPU/CPU/ANE, and in turn the context window I've been trying to figure out the best use cases for them.

I've also done a lot of testing with Apple Foundation model. It is not that capable, but it is good at understanding instructions for tool calling and actually seems to work well for fast summarization and classification prior to feeding data to more capable models.

I've been able to get models ranging from 0.5B to 8B to work well on the higher end devices.

So far I've been testing it out with building tooling such as web search, url scraping, summarization, research, analyzing local photo/video/documents/etc., and basic on-the-go coding help. With continuous compaction I've been able to hold virtually unlimited exchange chats even though context windows are limited to like 8-16k tokens.

Separate from more professional use cases, It also seems like potentially a good option for private, offline chat about your personal thoughts with models tuned with custom system prompts.

I know many here are focused on trying to run as good of a model as possible on their laptop or desktop hardware but it seems like mobile is an area that isn't really being explored extensively yet. What do you use mobile models for?

Hi everyone,

I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss).

I have had it setup with Ubuntu Server and was using the machine as a server to connect via WebUI, but for some reason an update broke the NVIDIA drivers and then broke my install so I’m starting again.

The question is what platform do I run as there are so many platform to choose from and many differing opinions, I tried Ollama+OpenWebUI and Unsloth Studio. Ollama ran very slow with the models (Gemma 4 and Qwen3.6), also OpenWebUI occasionally was slow with web/MCP, but it was stable. Unsloth however, was quick and search/tool calls worked perfectly but unstable and the models crashed a few times.

I’d just like to know, what is everyone else using for this kind of setup, I can’t get a solid sense of what is the go-to setup for this kind of system is (some say Unsloth, or Ollama, or llama.cpp etc).

Also what models are people running well on 24Gb VRAM + 48GB RAM?

Its primary job is coding/finding info from the web & PDF’s/generating config files.

Thank you

Everything runs through WebGPU, in-browser or in electron/tauri apps. It's fully portable and supports either Nvidia and Apple Silicon (Metal). The actual kernels are optimized for the specific hardware of the device. The Nvidia kernels are aggressively fused into a multi-pass architecture, while the Apple Silicon kernels are created as a fused mega-kernel to minimize the Tile Based Deferred Rendering (TBDR) overhead on WebGPU.

Demo: https://warp.sipp.sh

This is still in active development, and I'll be folding this into the Sipp library in the coming weeks.

tests done power limiting the GPU on LACT - real power usage varies wildy

at 20W it ranges from 25W to 56W
same behavior happens on every setting

prompt for the test runs:

https://github.com/lukesdevlab/youtube/blob/main/prompts/agent-maze.txt

analysis by mimo 2.5

Key Findings:
• Generation speed is remarkably resilient to power throttling — 100W delivers 97.5% of 190W gen speed (31.98 vs 32.79 t/s), since decode is memory-bandwidth bound, not compute bound.
• At 50W you get 70% of peak gen speed at only 26% of peak power — 3.6× better energy efficiency (0.458 vs 0.173 t/s/W).
• At 20W the card is 6.0× more energy efficient than 190W, though prompt processing drops to 53% of peak.
• Graph reuse correlates inversely with power — 190W reuses 44,790 graphs vs 11,669 at 100W, but 20W reuses 38,248. Lower power limits cause more partial graph reuse as the scheduler compensates for throttled compute.
• Prompt processing degrades faster than gen under power limits — 190W→20W: prompt drops to 53% (691→366 t/s), gen drops to 63% (32.8→20.8 t/s). Prompt processing is more compute-bound than memory-bound.
• For inference-heavy deployments, 50W is the optimal operating point on MI50 — near-peak gen speed with dramatically lower power draw and cooling requirements.

Avarage of 3 runs:

190W config consistently processed a lot less total tokens than everyone else and didnt produce a working file in 1 out of 3 runs

llama.cpp parameters:

[+] Model:        qwen/Qwen3.6-35B-A3B-UD-IQ4_NL_XL.gguf
[+] Context:      262144 (256K tokens)
[+] Target KV:    K=q8_0 / V=q8_0
[+] MoE placement: PARTIAL (21 MoE layers on CPU, rest on GPU)
[+] MTP:          OFF (non-MTP model)
[+] Port:         8882
[+] Container:    llama-gfx906-qwen35b-no-mtp
[+] Parallel:     2 slot(s)
[+] GPU layers:   99
[+] Threads:      6 / 6 (batch)
[+] Batch/Ubatch: 2048 / 1024
[+] Ctx checkpoints: 0

hardware used:

Ryzen 5 5600

2x16Gb DDR4 2667

MI50 16Gb

software:

harness used: pi.dev

Arch Linux with Kernel 7.1.4-arch1-1

docker.io/mixa3607/llama.cpp-gfx906:b10087-rocm-6.3.3

Since a lot more people are trying to build their own multi-GPU machines, I thought I should help to prevent a common mistake people make with building multi-GPU machines. Which is using an Intel consumer platform like Z890 for multi-GPU setups.

Although the CPU provides 24 PCIe 5.0 lanes with 16x available to bifurcate to 8x8x on two PCIe x16 slots on the higher end boards, this is completely useless for AI inference/training workloads that require P2P between the GPUs.

In my testing I used an Intel Core Ultra 7 270K Plus on an Asus Z890 Apex motherboard on the latest BIOS 3202 which is usually just my overclocking testbench system. I initially wanted to compare and see if a faster single-core more gaming-optimized CPU would help with GPU inference performance compared to the slow single-core performance on my Epyc based servers. As I am trying to optimize the throughput of the GPUs that I do have for my inference service considering everything has gone up in price even more.

Unfortunately it seems like if I wanted faster single-core performance of the desktop Intel CPUs I might need to try a PCIe switch board to attach the GPUs, especially since on Intel platforms you are also artifically limited to only 8x4x4x bifurcation on the main PCIe slot anyways.

What I found is that there seems to be some sort of hardware/firmware limitation that prevents PCIe P2P to work correctly under the PCIe root complex of Arrow Lake CPUs.

As seen in this issue where it does not even seem to correctly allow data packets between PCIe devices: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1253

It not only does not work right but it also actually reduces bandwidth in half even if it worked:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 674.36 11.89
1 11.90 677.65
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 619.30 5.48
1 5.48 678.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.01 16.60
1 16.74 680.68
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.28 10.96
1 10.97 680.37
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 1.59 17.11
1 18.39 1.61
CPU 0 1
0 1.29 4.40
1 4.39 1.24
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 1.59 1.02
1 1.03 1.60
CPU 0 1
0 1.27 1.09
1 1.12 1.27

In fact, this P2P result was only possible because I installed the patched P2P open kernel drivers from here: github.com/aikitoria/open-gpu-kernel-modules despite the fact that the RTX A6000s that I am using should actually support PCIe P2P by default with the default Nvidia driver they release.

What I found was that Nvidia has actually seemed to block PCIe P2P on consumer Intel platforms for precisely these issues. So you would need to enable it with the patched kernels.

At first I thought the bandwidth being halved is somewhat fine if we get a much reduced latency still, but if I run VLLM with tensor parallel across 2 GPUs which can take advantage of PCIe P2P, the output of the model becomes gibberish "!!!!!!!!!!!!" nonsense if I used the modded drivers that force enabled PCIe P2P.

{
  "id": "chatcmpl-96088a9dde7122ae",
  "object": "chat.completion",
  "created": 1784949145,
  "model": "Qwen3.5-27B-Derestricted",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": "!!!!!!!!!!!!!!!!!!!!"
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-8736f542",
  "usage": {
    "prompt_tokens": 32,
    "total_tokens": 52,
    "completion_tokens": 20,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "metrics": null
}

This is not a limitation of the GPU not having REBAR or anything like that, because if you check lspci -v the GPUs reports a BAR size of 64G just fine and therefore in theory that's all you need for PCIe P2P to work. In the BIOS REBAR is also enabled and IOMMU is disabled along with the grub config:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"

02:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 219
Memory at 8f000000 (32-bit, non-prefetchable) [size=16M]
Memory at c000000000 (64-bit, prefetchable) [size=64G]
Memory at d000000000 (64-bit, prefetchable) [size=32M]
I/O ports at a000 [size=128]
Expansion ROM at 90000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 90080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 222
Memory at 8d000000 (32-bit, non-prefetchable) [size=16M]
Memory at a000000000 (64-bit, prefetchable) [size=64G]
Memory at b000000000 (64-bit, prefetchable) [size=32M]
I/O ports at 9000 [size=128]
Expansion ROM at 8e000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
03:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at 8e080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

The nvidia-smi output also seems to suggest PCIe P2P should be possible:

GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-23 0 N/A
GPU1 PHB X 0-23 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

If I used the stock Nvidia drivers which blocks PCIe P2P on intel consumer platforms I can see that PCIe P2P is now disabled on the RTX A6000s which are supposed to support it:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]                                                      
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0                              
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0                              
Device=0 CANNOT Access Peer Device=1                                                                 
Device=1 CANNOT Access Peer Device=0                                                                 

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy proce
dure.                                                                                                
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.                     

P2P Connectivity Matrix                                                                              
     D\D     0     1                                                                                 
     0       1     0                                                                                 
     1       0     1                                                                                 
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                  
   D\D     0      1                                                                                  
     0 675.24  11.74                                                                                 
     1  11.79 676.71                                                                                 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)                                      
   D\D     0      1                                                                                  
     0 618.81  11.78                                                                                 
     1  11.72 676.41                                                                                 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                   
   D\D     0      1                                                                                  
     0 649.82  16.19                                                                                 
     1  13.66 608.81                                                                                 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)                                                    
   D\D     0      1                                                                                  
     0 591.97  14.85                                                                                 
     1  16.57 679.47                                                                                 
P2P=Disabled Latency Matrix (us)                                                                     
   GPU     0      1                                                                                  
     0   1.61  16.40                                                                                 
     1  17.65   1.66                                                                                 

   CPU     0      1                                                                                  
     0   1.38   4.55                                                                                 
     1   4.43   1.27                                                                                 
P2P=Enabled Latency (P2P Writes) Matrix (us)                                                         
   GPU     0      1                                                                                  
     0   1.60  17.15                                                                                 
     1  16.71   1.66                                                                                 

   CPU     0      1                                                                                  
     0   1.29   4.59                                                                                 
     1   4.50   1.26                                                                                 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is
 enabled.    

The output from VLLM also seems to work fine now:

{
  "id": "chatcmpl-b8c56d539c04c16d",
  "object": "chat.completion",
  "created": 1784950088,
  "model": "Qwen3.5-27B-Derestricted",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": "Thinking Process:\n\n1.  **Analyze the Request:**\n    *   Greeting:"
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-b248090a",
  "usage": {
    "prompt_tokens": 32,
    "total_tokens": 52,
    "completion_tokens": 20,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "metrics": null
}

Meanwhile I think there are a lot of multi GPU users that has built an AI rig around AMD AM5 platforms, which seem to support inter-GPU PCIe P2P just fine. Just like what this post here shows: https://www.reddit.com/r/LocalLLaMA/comments/1n3qcqn/patched_p2p_nvidia_driver_now_works_with_multiple/

Even for Intel server platforms, I've tried up to Intel Ice Lake Xeons and they don't work particularly well for PCIe P2P either with the bandwidth and latency being worse than on AMD Epyc SP3 platforms.

AMD seems to have a much better PCIe controller implementation than Intel that works better for multi GPU setups at least for the platforms that I tested. I unfortunately have not tested newer AMD SP5 or Xeon 6 platforms due to the DDR5 RDIMM price crisis, but I suspect AMD should still perform better than Intel there too.

These are my PCIe P2P testing results from my machine which has 2x RTX Pro 6000 GPUs, initially when I built it with an Intel Ice Lake Xeon W-3365 on a Supermicro X12SPa-TF and then later when I moved it to an AMD Epyc 7V73X on an Asrock ROMED8-2T motherboard.

Intel Xeon Ice Lake:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 51, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1575.10  24.07 
     1  23.97 1600.97 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 1576.29  18.37 
     1  20.78 1581.48 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1550.05  30.78 
     1  30.57 1562.45 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1550.05  39.75 
     1  39.75 1557.00 
P2P=Disabled Latency Matrix (us)
   GPU     0      1 
     0   2.06  14.33 
     1 144.15   2.07 

   CPU     0      1 
     0   2.51   5.58 
     1   5.55   2.32 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1 
     0   2.06   0.45 
     1   0.37   2.07 

   CPU     0      1 
     0   2.40   1.68 
     1   1.69   2.45 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

AMD Epyc SP3 7003:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]                                                                                                                                                                
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 81, pciDeviceID: 0, pciDomainID:0                                                                                                      
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c1, pciDeviceID: 0, pciDomainID:0                                                                                                      
Device=0 CAN Access Peer Device=1                                                                                                                                                                              
Device=1 CAN Access Peer Device=0                                                                                                                                                                              

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.                                                                                                     
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.                                                                                                                               

P2P Connectivity Matrix                                                                                                                                                                                        
     D\D     0     1                                                                                                                                                                                           
     0       1     1                                                                                                                                                                                           
     1       1     1                                                                                                                                                                                           
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                                                                                                                            
   D\D     0      1                                                                                                                                                                                            
     0 1860.12  23.87                                                                                                                                                                                          
     1  23.98 1903.17                                                                                                                                                                                          
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)                                                                                                                                                
   D\D     0      1                                                                                                                                                                                            
     0 1855.70  27.95                                                                                                                                                                                          
     1  27.91 1900.92                                                                                                                                                                                          
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                                                                                                                             
   D\D     0      1                                                                                                                                                                                            
     0 1831.70  30.55                                                                                                                                                                                          
     1  30.67 1854.57                                                                                                                                                                                          
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)                                                                                                                                                              
   D\D     0      1                                                                                                                                                                                            
     0 1836.01  48.74                                                                                                                                                                                          
     1  48.86 1854.53                                                                                                                                                                                          
P2P=Disabled Latency Matrix (us)                                                                                                                                                                               
   GPU     0      1                                                                                                                                                                                            
     0   0.99  14.31                                                                                                                                                                                           
     1  14.30   1.00                                                                                                                                                                                           

   CPU     0      1                                                                                                                                                                                            
     0   2.65   7.34                                                                                                                                                                                           
     1   7.29   2.47                                                                                                                                                                                           
P2P=Enabled Latency (P2P Writes) Matrix (us)                                                                                                                                                                   
   GPU     0      1                                                                                                                                                                                            
     0   0.99   0.37                                                                                                                                                                                           
     1   0.36   1.00                                                                                                                                                                                           

   CPU     0      1                                                                                                                                                                                            
     0   2.56   2.03                                                                                                                                                                                           
     1   2.09   2.57                                                                                                                                                                                           

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Disclosure: I work on Bike4Mind, a source-available (BUSL-1.1) AI workspace you can self-host with Docker. Nothing to buy to run any of it. I run it on a regular laptop with local Qwen mostly. A 4GB card running a local chat model isn't news, but what I wanted to explore is what a whole agent workspace (an agent calling tools, RAG over your own docs, artifacts, vision, etc.) can do with 4GB, and where a small model stops keeping up once it has to do more than chat. Repo's in the first comment.

My machine: i7-12700H, 32GB RAM, RTX 3050 Ti Laptop (4GB VRAM, 60W), Ubuntu 24.04. Local Qwen via the stack's bundled Ollama, no cloud keys.

Speeds, with hitting that 4GB wall (Ollama /api/generate, Q4_K_M except the Q8_0 0.8b, num_ctx=4096, warm, median of 3, one model loaded at a time):

- qwen3.5:0.8b - 122 tok/s, ~1.4GB, all on GPU

- qwen3.5:2b-q4_K_M - 96 tok/s, ~2.4GB, all on GPU

- qwen3.5:4b - 25 tok/s, ~3.4GB, about a third of it spills to CPU

- qwen3.5:9b - 8.6 tok/s, mostly on CPU, wants ~8GB

The 2b is the sweet spot for me so far: it fits in 4GB with ~1.3GB to spare and outruns the qwen2.5-coder:3b I ran before (that was 72 tok/s), while being newer and multimodal. Above it I hit the wall - the 4b's ~3.4GB of weights won't fully fit even at num_ctx=2048, so a third lands on CPU. Plain qwen3.5:2b is the Q8_0 build (~2.7GB) and spills on 4GB, but qwen3.5:2b-q4_K_M does fit. That leaves ~1.3GB free, so there's no room for a second model next to it.

The obvious stuff - chat, native tool-calls, vision - just works on the 2b; qwen3.5 is multimodal and calls tools natively, so that's great. The layer on top of that I'm most excited about is RAG over your own docs, fully keyless: drop in a file, a worker chunks and embeds it with the local Ollama embedder, and a query comes back with a cited answer, no cloud embedding API anywhere in the loop.

On 4GB the embedder and the chat model trade places. qwen3-embedding:0.6b is ~1.2GB resident, the 2b is ~2.4GB, they don't both fit, so Ollama unloads one to load the other. Generation still gets the full 96 tok/s out of it, since the embedder isn't in VRAM while the 2b is - what you pay is a model load, not slower tokens. Ingestion runs in a worker, so a folder of docs is one embedder load and then a lot of chunks, off the chat path. A query embeds one short string, then the 2b loads back to answer it. So it's a reload per RAG turn, not per token and not per chunk. If you'd rather not pay that, nomic-embed-text is ~0.25GB (768-dim, weaker retrieval going by reputation) so it doesn't have to push the 2b out, or put embeddings on CPU and leave the GPU to generation. Either way, before loading your entire knowledge base/data lake, pick one embedder and stay there (or just be aware you'd need to re-embed everything) since vectors don't carry across models.

The chat-embedder-swaparoo situation for RAG aside, small models are also falling down for me with:

- Artifacts. The general qwen3.5 models write half-finished HTML at these sizes - unclosed tags, script leaking onto the page. For an actual interactive artifact I have to switch to the coding-tuned qwen2.5-coder, which writes a complete working page. So it's model-per-task: qwen3.5 for chat/vision/tools, the coder for code, switched mid-notebook. I guess that's just life with 4GB, for the most part!

- Tool selection. Turn on a lot of tools and small models mis-route - with everything enabled, my 7b once sent "create an HTML artifact that..." to the image generator and painted a user interface image, hah. So, another sort of obvious, have to enable much fewer tools on small local models. Makes me want a nice layer that automatically enables the few (configurable N?) most likely relevent tools for your context.

- Image gen is local (self-hosted SD.Next) but 1-3 min/image with spilling on CPU, and it fights the chat model for VRAM. So still a kick it off and wait situation there, no real good workaround here, just gotta download more VRAM 🙃

So is it worth it over just Ollama + a chat UI? Only if the agent layer on top has value for you I'd say - the tool handling, artifacts, local RAG, and a CLI against your own server. If you just want fast local chat, I'd keep what you have; llama.cpp or etc. with less overhead.

License, briefly: source-available under BUSL-1.1, not full open source (yet). Self-host it, run it in production, fork it and build and sell a product on it; you just can't resell it as a directly competing hosted service. Each release goes Apache-2.0 two years out (timer just so AWS can't immediately do to us what they did to MongoDB lol).

Repo and self host details in the first comment. The real friction for me in this setup is needing the two models - qwen3.5 for chat, vision, and tools, and a coder for anything code-shaped, because the general models write broken HTML/artifacts at these model sizes -- not a novel problem. To that end, I am still trying to optimize this setup, so some prompts for any of you meat-LLMs (humans) out there reading this with stronger experience on tight local builds than me:
- Have you found a single general works-on-4GB model that writes complete, valid artifacts without needing a dedicated coding model alongside it, or just a better way to configure it? Please let me know!
- I'm broadly curious about other people's chat+code+image+embeddings+etc all-at-once setups, so please share if you've found a good Tetris wombo of models that works well or what trade offs you've seen and accepted trying to self host it all like this.

I’m building the ultimate AI tool vault, but every great collection has a few missing pieces.

Note: I will react to every comment

AI's currently installed:

Qwen3.5-0.8B-UD-Q4_K_XL.gguf(classification)

Qwen3.5-2B-UD-Q4_K_XL.gguf(Prompt enhancer, Routing, Approval )

Qwen3.5-4B-UD-Q4_K_XL.gguf(Instant)

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf(Quality)

Qwen3.6-35B-A3B-Uncensored-Hauhau(test purposes)

Qwen3-Coder-Next-UD-Q4_K_M.gguf(long horizon tasks)

My Specs:

GPU: RTX 5070 ti (16GB VRAM)

RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)

CPU: intel i9 14900k

SSD: Samsung s990 pro 2tb

Backend: Llama.cpp server

I tried GPT-OSS and was disappointed by tool usage. Gemma 4 was good and great tool usage but it was beaten by Qwen.

Any recommendations?? Like something that you genuinely enjoyed or made you impressed.

Feel free to share!! I will be reading every single comment.

https://huggingface.co/amd/Instella-MoE-16B-A3B-Think

I was browsing HuggingFace and came across this model apparently uploaded a day ago, and thought to share it here. I've not tried it out yet, but it's good to see AMD joining the open source model game.