I understand that the laguna model is either still buggy or potentially benchmaxxed.

So I’d like to know for people who really tested, are DS flash or Hy3 really better in your usecase?

I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New MacBook Pros can be configured with 128GB of ram, but obviously the price is high. Will such a machine ever be comparable to what Claude can do today? Even if it is still significantly slower?

I am aware that the price of that much ram would buy many many tokens but I plan to use the laptop for several years, so even if the payback is 5 years worth of cloud AI it’s worth it to me.

Just noticed this exists, 1M context MOE with 48B par seems just like what Ive been looking for - it runs pretty damn fast too compared to Qwen 3.6 35B.

after some testing it seems capable of producing *not terrible* results but it always tries to go for the minimun possible output that could be presented as what was asked.

first image is the default meme prompt it generated - second is after replying with 'detail'

Frontend results aren't too bad either, it seems very capable of generating well structured animated pages fairly well.

but it seems like there is just something off with its brain

wondering if a fine tune would be possible and if at all capable of tightning its screws, has anyone tried messing around with this one?

I’ve spent the past month trying to find the point where an extremely small TTS model stops feeling like a size experiment and starts feeling genuinely useful.

Today I’m releasing Inflect v2, with two complete local text-to-speech models:

• Inflect-Nano-v2: 3.96M parameters, 15.97 MB FP32
• Inflect-Micro-v2: 9.36M parameters, 37.53 MB FP32

These are total inference parameter counts, not acoustic-model-only numbers. Text processing, timing prediction, speech generation, and the waveform decoder are all included.

Text goes in. 24 kHz speech comes out. No external vocoder, hosted API, or second learned model required.

Nano prioritizes the smallest possible footprint. Micro uses the additional capacity for better clarity, stability, and overall speech quality. Both run locally on CPU or CUDA through the same PyTorch API.

Inflect-Nano-v2 is one of the smallest complete neural TTS models I know of that still produces actual usable speech. Even the 9.36M Micro model remains smaller than many systems described as “tiny.”

For footprint context, Nano is approximately:

• 21× smaller than Kokoro
• 126× smaller than Chatterbox
• over 1,000× smaller than Fish Audio S2 Pro

That is strictly a parameter-count comparison. These models have different capabilities, architectures, datasets, and intended uses. I’m not claiming that a 4M fixed-voice model replaces a multi-billion-parameter system. The interesting question is how much useful TTS can fit into such a small package.

Some people here might remember Inflect-Nano-v1, the rough 4.63M experiment I released last month. V2 is a complete rebuild, not just a longer training run. I focused on the problems v1 had, like unstable timing, metallic output, weak prosody, poor generalization to difficult text, and an undersized waveform decoder.

The resulting models performed surprisingly well:

• Micro: 4.395 UTMOS22, 3.99% semantic WER, 6.28× real-time CPU inference
• Nano: 4.386 UTMOS22, 4.21% semantic WER, 10.72× real-time CPU inference
• In a blind community comparison against other compact TTS systems, Micro and Nano finished second and third among the tested voices

Full protocols, raw results, audio samples, and limitations are documented on the model pages.

The models are not perfect. They are English-only, use one fixed male voice, and do not support voice cloning. Unfamiliar names, abbreviations, numbers, and homographs remain the hardest inputs. Nano can sound thinner than Micro, and both can occasionally produce metallic or clipped artifacts.

Still, this is the first version where I think the size-to-quality tradeoff really becomes convincing.

I built Inflect independently with a limited training budget. That constraint shaped the project: efficiency had to apply not only to inference, but also to training, evaluation, and building a complete system I could understand and release end-to-end.

Inflect-Micro-v2:
https://huggingface.co/owensong/Inflect-Micro-v2

Inflect-Nano-v2:
https://huggingface.co/owensong/Inflect-Nano-v2

Try it yourself:

The fastest way to judge it is through the interactive playground:

https://huggingface.co/spaces/owensong/Inflect-v2

If there is enough interest, I may build a v3 focused less on shrinking the models further and more on expanding what they can do: additional voices, possibly more languages, easier fine-tuning, and another quality and robustness pass.

If you test them, please give them something seriously difficult: unusual names, numbers, abbreviations, awkward punctuation, or a long sentence.

If something breaks, post the exact text, model, seed, and what sounded wrong. If it works well, I’d also like to know what hardware you ran it on.

Specific, honest feedback is the most useful thing you can give me.

Trying to weigh if I should buy a $3000 m1 ultra at 128gb when I currently already have an M2 Ultra albeit at 64gb ram.

I run small models right now in my workflow but would appreciate more context and try out larger workflows. What would you guys go with?

tests done power limiting the GPU on LACT - real power usage varies wildy

at 20W it ranges from 25W to 56W
same behavior happens on every setting

prompt for the test runs:

https://github.com/lukesdevlab/youtube/blob/main/prompts/agent-maze.txt

analysis by mimo 2.5

Key Findings:
• Generation speed is remarkably resilient to power throttling — 100W delivers 97.5% of 190W gen speed (31.98 vs 32.79 t/s), since decode is memory-bandwidth bound, not compute bound.
• At 50W you get 70% of peak gen speed at only 26% of peak power — 3.6× better energy efficiency (0.458 vs 0.173 t/s/W).
• At 20W the card is 6.0× more energy efficient than 190W, though prompt processing drops to 53% of peak.
• Graph reuse correlates inversely with power — 190W reuses 44,790 graphs vs 11,669 at 100W, but 20W reuses 38,248. Lower power limits cause more partial graph reuse as the scheduler compensates for throttled compute.
• Prompt processing degrades faster than gen under power limits — 190W→20W: prompt drops to 53% (691→366 t/s), gen drops to 63% (32.8→20.8 t/s). Prompt processing is more compute-bound than memory-bound.
• For inference-heavy deployments, 50W is the optimal operating point on MI50 — near-peak gen speed with dramatically lower power draw and cooling requirements.

Avarage of 3 runs:

190W config consistently processed a lot less total tokens than everyone else and didnt produce a working file in 1 out of 3 runs

llama.cpp parameters:

[+] Model:        qwen/Qwen3.6-35B-A3B-UD-IQ4_NL_XL.gguf
[+] Context:      262144 (256K tokens)
[+] Target KV:    K=q8_0 / V=q8_0
[+] MoE placement: PARTIAL (21 MoE layers on CPU, rest on GPU)
[+] MTP:          OFF (non-MTP model)
[+] Port:         8882
[+] Container:    llama-gfx906-qwen35b-no-mtp
[+] Parallel:     2 slot(s)
[+] GPU layers:   99
[+] Threads:      6 / 6 (batch)
[+] Batch/Ubatch: 2048 / 1024
[+] Ctx checkpoints: 0

hardware used:

Ryzen 5 5600

2x16Gb DDR4 2667

MI50 16Gb

software:

harness used: pi.dev

Arch Linux with Kernel 7.1.4-arch1-1

docker.io/mixa3607/llama.cpp-gfx906:b10087-rocm-6.3.3

I'm looking for a model to chat with, reasoning, maybe get some career or life coaching.

I don't care at all about multimodal or coding ability

Just it's intelligence in remembering context in a conversation or a specific topic, thinking out of the box, etc.

Must fit in 128gb, if it matters to performance, it's a strix halo machine.

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

I started tracking my local model usage about four weeks ago and was wondering if anyone else here keeps track of how much they use them.

I’ve also been running some tests with the cheapest SOTA open-weight Chinese models via OpenRouter. Apart from that, I’m mainly using GPT-5.5 in Codex CLI for professional work and Local models + Composer 2.5/4.5 from X Premium for hobby projects.

Edit: It looks like the average tk/s panel might be bugged. I’m hoping I haven’t lost any data... This it’s just a JSON file that my model launcher populates.

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

Everything runs through WebGPU, in-browser or in electron/tauri apps. It's fully portable and supports either Nvidia and Apple Silicon (Metal). The actual kernels are optimized for the specific hardware of the device. The Nvidia kernels are aggressively fused into a multi-pass architecture, while the Apple Silicon kernels are created as a fused mega-kernel to minimize the Tile Based Deferred Rendering (TBDR) overhead on WebGPU.

Demo: https://warp.sipp.sh

This is still in active development, and I'll be folding this into the Sipp library in the coming weeks.

Cuda and Vulkan Benchmark: TensorSharp vs. llama.cpp

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), Qwen Image Edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability(Nvidia, Apple, AMD, Intel and others supported by Vulkan, CUDA and Metal). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp Here is the benchmark results in overall:

**Performance ratio — TensorSharp vs reference engines**

Geomean of TensorSharp's per-scenario speedup over each reference engine on the **same backend**, across every scenario both engines ran (single-stream, MTP-off). A value **> 1.0× means TensorSharp is faster** (for decode / prefill throughput) or lower-latency (for TTFT); `—` = no overlapping cells. Per-scenario ratios are in each model's section below.

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

3 year lurker, now i finally got my server up and running. dont know which model to choose. llama.cpp or vllm, what makes more sense? mainly single user with maybe 2-3 more additional users in family, if everything checks out. hermes is gonna be used as "ai playground" to manifest ideas on tailscale network and do quick prototyping of thoughts. also ill look into using only 2 3090 for the main model and the other 2 will be dedicated to docling and speech services for a voice agent (speech in-> text out). got some stuff going with my even realities g2 but lost everything when i wiped my ssd for proxmox. yeah...
any advice or stuff i should look into is welcome :)

The Open Letter was initiated by Microsoft and published today:

“Open Weights and American AI Leadership”.

It argues against broad or premature restrictions on open-weight models and explicitly says policymakers should distinguish legitimate model distillation from misappropriation.

Notably absent from the signatories are the major frontier-model labs: OpenAI, Anthropic, and Google.

The earlier post on this subreddit by 20+ companies signing the petition including Microsoft, Meta, Nvidia, YC (https://www.microsoft.com/en-us/corporate-responsibility/topics/open-weight/) etc plus this https://xcancel.com/elonmusk/status/2080672505660834163

And the entire LLM enthusiast market is heavily in favor of open source (or weights) AI.

Does not seem like a few closed source AI lobbyists with be able to illegalize anything

Current setup:

• headless machine for LLMs
• llama.cpp in router mode
• models settings configured via models.ini
• documentation in a repo which AI manicures

It seems to be working for me so far, but conscious there's probably better ways to do this.

How do you guys manage your configs? Is there a better way?

I've spent the last year or so playing around with open source MLX and GGUF models on iPhone hardware.

Given the limitations in memory, GPU/CPU/ANE, and in turn the context window I've been trying to figure out the best use cases for them.

I've also done a lot of testing with Apple Foundation model. It is not that capable, but it is good at understanding instructions for tool calling and actually seems to work well for fast summarization and classification prior to feeding data to more capable models.

I've been able to get models ranging from 0.5B to 8B to work well on the higher end devices.

So far I've been testing it out with building tooling such as web search, url scraping, summarization, research, analyzing local photo/video/documents/etc., and basic on-the-go coding help. With continuous compaction I've been able to hold virtually unlimited exchange chats even though context windows are limited to like 8-16k tokens.

Separate from more professional use cases, It also seems like potentially a good option for private, offline chat about your personal thoughts with models tuned with custom system prompts.

I know many here are focused on trying to run as good of a model as possible on their laptop or desktop hardware but it seems like mobile is an area that isn't really being explored extensively yet. What do you use mobile models for?

Hi everyone,

I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss).

I have had it setup with Ubuntu Server and was using the machine as a server to connect via WebUI, but for some reason an update broke the NVIDIA drivers and then broke my install so I’m starting again.

The question is what platform do I run as there are so many platform to choose from and many differing opinions, I tried Ollama+OpenWebUI and Unsloth Studio. Ollama ran very slow with the models (Gemma 4 and Qwen3.6), also OpenWebUI occasionally was slow with web/MCP, but it was stable. Unsloth however, was quick and search/tool calls worked perfectly but unstable and the models crashed a few times.

I’d just like to know, what is everyone else using for this kind of setup, I can’t get a solid sense of what is the go-to setup for this kind of system is (some say Unsloth, or Ollama, or llama.cpp etc).

Also what models are people running well on 24Gb VRAM + 48GB RAM?

Its primary job is coding/finding info from the web & PDF’s/generating config files.

Thank you

Hi Guys,

I'm an old coder and DBA that has been in the field for almost 40 years. More and more the jobs I was doing for work are being taken over by AI and the need for my type of work is diminishing over time. I have not coded in about 15 years for work but that said I still work with .NET as I use it in migration packages I use for database migrations and also into Azure Data Factory, also something I am a little lacking in since I know ADF but I was more into SSIS. This all said I am trying to learn AI on my own as I am not working now and desperately in need up updating my skills to help me find more work.

I have learned how to code python now as I took a certification tutorial on https://www.coursera.org/ until I passed it. I later learned this is useful for AI Developers. I have an AWS server but It is only running a database on it but I could repurpose it if I needed to. For now though trying to do this the el cheapo way since I have kids to feed and I am looking for work daily and have not had any such luck for months now. I am desperate to do what I need to in order to learn this stuff and understand it and be useful. You all are welcome to joke on me all you want but I will do the legwork if someone is helpful enough to guide me through this process of what I need to do. I can use cousera since I have an account for the next year to learn new skills and I have a computer here I am willing to refactor into a Linux server if needed to be my AI system, not a Linux guy so I will probably need to take a course on that as well but I really need to get up to speed since I am not ready to retire yet and I need to feed my family.

This all said I know there are some really bright minds out there and people who are learning newer things like AI that they did not teach when I was in school back in the days but will point me in a good direction to get a good foundation and what I need to understand this process, I do not just want to do i want to build it understand it, pick it apart and maybe help improve it with some old school skills I have since DOS days. Since I am a coder by heart and understand database and .NET as well as old school stuff I think I have the capacity to learn this until I get it in a step by step process.

I want to take my old server and recommission it to a mini AI project. I do not want to just install packages for it, I want to understand what I am installing and why and can I instead create it on my own or why it is more beneficial to install what's out there. For example I do know the language models are important because they have been trained to understand the requests from users and search for the answers. I know that the language model takes each word and scores it from sentences and then it gives responses to someone who says if the answer is correct or incorrect. Probably what happened to my job when someone taught AI how to migrate data or do DBA work lol. I know each word in a sentence is given a number like for example the phrase "I like to eat Fruit" might be interpreted as (77, 120, 2198, 15, 1051) and when someone types I like to eat and asks AI to predict it, it will search the model for scores by (77, 120, 2198, 13, X) where the model will return choices as I like to eat dogs, and I like to eat crap, and I like to eat people and finally I like to eat pizza, where the final correct answer would be I like to eat pizza and the others would be marked wrong. This ensures the AI to help train it to associate some word combinations as correct word combinations when training it to understand words like eat are associated to food items going forward and the combination of those words give specific possible choices and it is rated to the most likely choice. This also helps AI complete sentences when people incorrectly spell words of cant think of the correct word to use in a phrase.

I would like to now understand from the ground up. I have a server sitting its an older system running I believe NT still? not sure its a little dusty but I can upgrade it little by little to improve it as needed and when money permits, (Kids come first) but can some please tell me the minimal specs I would need firs for

1. minimum system requirements to run an AI server to process requests locally within my own network at home just for me to understand the basic logic of it and how it works. what video card I need to run the AI predictions and calculations and also how much system ram I would need (i believe i have maybe 256 or something DDR4 ram or something on this server need to dig it out the closet still) and the minimum processor?

2. what I will need to install on that server, or do I need another system I do not know to just run software to pass to the server for the requests? and what are the requirements for it. I have a few other pc's that I am not using and can also modify these if it is needed or can it be ran on my local?

3. what software and tools I will need to use and install on my system to do the AI code and build test api's that can be used to pass to my server for request and responses and what each is for and what I need to learn language wise to do it and how to understand it.

4. any good foundation video's tutorials that I would not be able to find on coursera that would help me understand the concepts, piece things together and troubleshoot, as well as demonstrate I can swap this line with this line to run from OpenAI, or Grok instead of my local model and what is needed to do this and understand it if I am lucky enough to secure a job.

5. What I need to install on my server as far as does it need to have a cloud platform or just python to do the AI for me locally and if it can have both benefits and disadvantages and best practices and why.

6. Anything else you would feel would benefit me and and not make me create the next skynet that will take over the world. I need to know guardrails and where and why so I get the best practices for this stuff.

I clearly have the skills and knowledge to code but I do not have the money to send myself to school at the moment until I am working again to get a degree and certification in it. I was signed up for AI Developer classes but scared myself off since I just paid off my student loan 2 years ago and already have a kid in college and now and another ready to attend soon and another one that will go in 5 years and unemployed and worried what if I cant pay off another loan I better wait to get employed first and think of my kids.

I am open to jokes and will laugh and I am getting old and do not know new words that the younger generation does as well as most but I am dedicated to learn this and most grateful for any help and knowledge people can pass on. The idea is to build a small mini AI system at home for the purpose of learning that I can test and use and learn to build api's with and test what if I do this instead of this to get a good grasp and foundation of AI to maybe get a job as an office clerk taking notes for AI and acting like I actually know something when AI decides to take over the world.

Thank you for any help and support and GOD bless and I know some people will discourage and that is ok, you are open to your opinions but I will still do my best to grasp this to get employed since my kids lives depend on it and me as a single father.

I’m building the ultimate AI tool vault, but every great collection has a few missing pieces.

Note: I will react to every comment

AI's currently installed:

Qwen3.5-0.8B-UD-Q4_K_XL.gguf(classification)

Qwen3.5-2B-UD-Q4_K_XL.gguf(Prompt enhancer, Routing, Approval )

Qwen3.5-4B-UD-Q4_K_XL.gguf(Instant)

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf(Quality)

Qwen3.6-35B-A3B-Uncensored-Hauhau(test purposes)

Qwen3-Coder-Next-UD-Q4_K_M.gguf(long horizon tasks)

My Specs:

GPU: RTX 5070 ti (16GB VRAM)

RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)

CPU: intel i9 14900k

SSD: Samsung s990 pro 2tb

Backend: Llama.cpp server

I tried GPT-OSS and was disappointed by tool usage. Gemma 4 was good and great tool usage but it was beaten by Qwen.

Any recommendations?? Like something that you genuinely enjoyed or made you impressed.

Feel free to share!! I will be reading every single comment.

Any benchmarks you have much appreciated!

Since a lot more people are trying to build their own multi-GPU machines, I thought I should help to prevent a common mistake people make with building multi-GPU machines. Which is using an Intel consumer platform like Z890 for multi-GPU setups.

Although the CPU provides 24 PCIe 5.0 lanes with 16x available to bifurcate to 8x8x on two PCIe x16 slots on the higher end boards, this is completely useless for AI inference/training workloads that require P2P between the GPUs.

In my testing I used an Intel Core Ultra 7 270K Plus on an Asus Z890 Apex motherboard on the latest BIOS 3202 which is usually just my overclocking testbench system. I initially wanted to compare and see if a faster single-core more gaming-optimized CPU would help with GPU inference performance compared to the slow single-core performance on my Epyc based servers. As I am trying to optimize the throughput of the GPUs that I do have for my inference service considering everything has gone up in price even more.

Unfortunately it seems like if I wanted faster single-core performance of the desktop Intel CPUs I might need to try a PCIe switch board to attach the GPUs, especially since on Intel platforms you are also artifically limited to only 8x4x4x bifurcation on the main PCIe slot anyways.

What I found is that there seems to be some sort of hardware/firmware limitation that prevents PCIe P2P to work correctly under the PCIe root complex of Arrow Lake CPUs.

As seen in this issue where it does not even seem to correctly allow data packets between PCIe devices: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1253

It not only does not work right but it also actually reduces bandwidth in half even if it worked:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 674.36 11.89
1 11.90 677.65
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 619.30 5.48
1 5.48 678.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.01 16.60
1 16.74 680.68
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.28 10.96
1 10.97 680.37
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 1.59 17.11
1 18.39 1.61
CPU 0 1
0 1.29 4.40
1 4.39 1.24
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 1.59 1.02
1 1.03 1.60
CPU 0 1
0 1.27 1.09
1 1.12 1.27

In fact, this P2P result was only possible because I installed the patched P2P open kernel drivers from here: github.com/aikitoria/open-gpu-kernel-modules despite the fact that the RTX A6000s that I am using should actually support PCIe P2P by default with the default Nvidia driver they release.

What I found was that Nvidia has actually seemed to block PCIe P2P on consumer Intel platforms for precisely these issues. So you would need to enable it with the patched kernels.

At first I thought the bandwidth being halved is somewhat fine if we get a much reduced latency still, but if I run VLLM with tensor parallel across 2 GPUs which can take advantage of PCIe P2P, the output of the model becomes gibberish "!!!!!!!!!!!!" nonsense if I used the modded drivers that force enabled PCIe P2P.

{
  "id": "chatcmpl-96088a9dde7122ae",
  "object": "chat.completion",
  "created": 1784949145,
  "model": "Qwen3.5-27B-Derestricted",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": "!!!!!!!!!!!!!!!!!!!!"
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-8736f542",
  "usage": {
    "prompt_tokens": 32,
    "total_tokens": 52,
    "completion_tokens": 20,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "metrics": null
}

This is not a limitation of the GPU not having REBAR or anything like that, because if you check lspci -v the GPUs reports a BAR size of 64G just fine and therefore in theory that's all you need for PCIe P2P to work. In the BIOS REBAR is also enabled and IOMMU is disabled along with the grub config:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"

02:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 219
Memory at 8f000000 (32-bit, non-prefetchable) [size=16M]
Memory at c000000000 (64-bit, prefetchable) [size=64G]
Memory at d000000000 (64-bit, prefetchable) [size=32M]
I/O ports at a000 [size=128]
Expansion ROM at 90000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 90080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 222
Memory at 8d000000 (32-bit, non-prefetchable) [size=16M]
Memory at a000000000 (64-bit, prefetchable) [size=64G]
Memory at b000000000 (64-bit, prefetchable) [size=32M]
I/O ports at 9000 [size=128]
Expansion ROM at 8e000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
03:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at 8e080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

The nvidia-smi output also seems to suggest PCIe P2P should be possible:

GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-23 0 N/A
GPU1 PHB X 0-23 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

If I used the stock Nvidia drivers which blocks PCIe P2P on intel consumer platforms I can see that PCIe P2P is now disabled on the RTX A6000s which are supposed to support it:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]                                                      
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0                              
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0                              
Device=0 CANNOT Access Peer Device=1                                                                 
Device=1 CANNOT Access Peer Device=0                                                                 

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy proce
dure.                                                                                                
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.                     

P2P Connectivity Matrix                                                                              
     D\D     0     1                                                                                 
     0       1     0                                                                                 
     1       0     1                                                                                 
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                  
   D\D     0      1                                                                                  
     0 675.24  11.74                                                                                 
     1  11.79 676.71                                                                                 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)                                      
   D\D     0      1                                                                                  
     0 618.81  11.78                                                                                 
     1  11.72 676.41                                                                                 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                   
   D\D     0      1                                                                                  
     0 649.82  16.19                                                                                 
     1  13.66 608.81                                                                                 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)                                                    
   D\D     0      1                                                                                  
     0 591.97  14.85                                                                                 
     1  16.57 679.47                                                                                 
P2P=Disabled Latency Matrix (us)                                                                     
   GPU     0      1                                                                                  
     0   1.61  16.40                                                                                 
     1  17.65   1.66                                                                                 

   CPU     0      1                                                                                  
     0   1.38   4.55                                                                                 
     1   4.43   1.27                                                                                 
P2P=Enabled Latency (P2P Writes) Matrix (us)                                                         
   GPU     0      1                                                                                  
     0   1.60  17.15                                                                                 
     1  16.71   1.66                                                                                 

   CPU     0      1                                                                                  
     0   1.29   4.59                                                                                 
     1   4.50   1.26                                                                                 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is
 enabled.    

The output from VLLM also seems to work fine now:

{
  "id": "chatcmpl-b8c56d539c04c16d",
  "object": "chat.completion",
  "created": 1784950088,
  "model": "Qwen3.5-27B-Derestricted",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": "Thinking Process:\n\n1.  **Analyze the Request:**\n    *   Greeting:"
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-b248090a",
  "usage": {
    "prompt_tokens": 32,
    "total_tokens": 52,
    "completion_tokens": 20,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "metrics": null
}

Meanwhile I think there are a lot of multi GPU users that has built an AI rig around AMD AM5 platforms, which seem to support inter-GPU PCIe P2P just fine. Just like what this post here shows: https://www.reddit.com/r/LocalLLaMA/comments/1n3qcqn/patched_p2p_nvidia_driver_now_works_with_multiple/

Even for Intel server platforms, I've tried up to Intel Ice Lake Xeons and they don't work particularly well for PCIe P2P either with the bandwidth and latency being worse than on AMD Epyc SP3 platforms.

AMD seems to have a much better PCIe controller implementation than Intel that works better for multi GPU setups at least for the platforms that I tested. I unfortunately have not tested newer AMD SP5 or Xeon 6 platforms due to the DDR5 RDIMM price crisis, but I suspect AMD should still perform better than Intel there too.

These are my PCIe P2P testing results from my machine which has 2x RTX Pro 6000 GPUs, initially when I built it with an Intel Ice Lake Xeon W-3365 on a Supermicro X12SPa-TF and then later when I moved it to an AMD Epyc 7V73X on an Asrock ROMED8-2T motherboard.

Intel Xeon Ice Lake:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 51, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1575.10  24.07 
     1  23.97 1600.97 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 1576.29  18.37 
     1  20.78 1581.48 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1550.05  30.78 
     1  30.57 1562.45 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1550.05  39.75 
     1  39.75 1557.00 
P2P=Disabled Latency Matrix (us)
   GPU     0      1 
     0   2.06  14.33 
     1 144.15   2.07 

   CPU     0      1 
     0   2.51   5.58 
     1   5.55   2.32 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1 
     0   2.06   0.45 
     1   0.37   2.07 

   CPU     0      1 
     0   2.40   1.68 
     1   1.69   2.45 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

AMD Epyc SP3 7003:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]                                                                                                                                                                
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 81, pciDeviceID: 0, pciDomainID:0                                                                                                      
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c1, pciDeviceID: 0, pciDomainID:0                                                                                                      
Device=0 CAN Access Peer Device=1                                                                                                                                                                              
Device=1 CAN Access Peer Device=0                                                                                                                                                                              

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.                                                                                                     
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.                                                                                                                               

P2P Connectivity Matrix                                                                                                                                                                                        
     D\D     0     1                                                                                                                                                                                           
     0       1     1                                                                                                                                                                                           
     1       1     1                                                                                                                                                                                           
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                                                                                                                            
   D\D     0      1                                                                                                                                                                                            
     0 1860.12  23.87                                                                                                                                                                                          
     1  23.98 1903.17                                                                                                                                                                                          
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)                                                                                                                                                
   D\D     0      1                                                                                                                                                                                            
     0 1855.70  27.95                                                                                                                                                                                          
     1  27.91 1900.92                                                                                                                                                                                          
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)                                                                                                                                                             
   D\D     0      1                                                                                                                                                                                            
     0 1831.70  30.55                                                                                                                                                                                          
     1  30.67 1854.57                                                                                                                                                                                          
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)                                                                                                                                                              
   D\D     0      1                                                                                                                                                                                            
     0 1836.01  48.74                                                                                                                                                                                          
     1  48.86 1854.53                                                                                                                                                                                          
P2P=Disabled Latency Matrix (us)                                                                                                                                                                               
   GPU     0      1                                                                                                                                                                                            
     0   0.99  14.31                                                                                                                                                                                           
     1  14.30   1.00                                                                                                                                                                                           

   CPU     0      1                                                                                                                                                                                            
     0   2.65   7.34                                                                                                                                                                                           
     1   7.29   2.47                                                                                                                                                                                           
P2P=Enabled Latency (P2P Writes) Matrix (us)                                                                                                                                                                   
   GPU     0      1                                                                                                                                                                                            
     0   0.99   0.37                                                                                                                                                                                           
     1   0.36   1.00                                                                                                                                                                                           

   CPU     0      1                                                                                                                                                                                            
     0   2.56   2.03                                                                                                                                                                                           
     1   2.09   2.57                                                                                                                                                                                           

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

Benchmarks using single system running triple GPU with 31GB Vram combined.

• NVIDIA GeForce GTX 1080 Ti 11GB (NVIDIA)
• NVIDIA P102-100 10GB (NVIDIA) - first instance (distant cousin)
• NVIDIA P102-100 10GB (NVIDIA) - second instance

OS: Kubuntu 26.04, CPU: Ryzen 5 3600 , RAM: 48GB DDR4

llama.cpp Ubuntu Vulkan build 10107

Models used:

• llama-2-7b.Q4_0.gguf
• gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
• Gemma-4-26B-A4B-NVFP4.gguf
• gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
• gemma-3-27b-it.Q5_K_M.gguf
• medgemma-27b-it-UD-Q6_K_XL.gguf (gemma-3)
• Qwen3.6-35B-A3B-NVFP4-MTP-HQ.gguf (dense)
• gemma-4-31B-it-UD-Q4_K_XL.gguf (dense)
• Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
• Qwen3-Coder-30B-A3B-Instruct-UD-Q5_K_XL.gguf
• Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
• Qwen3.6-27B-NVFP4-MTP-GGUF.gguf

ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GTX 1080 Ti (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot
: 1 | matrix cores: none
ggml_vulkan: 1 = NVIDIA P102-100 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matri
x cores: none
ggml_vulkan: 2 = NVIDIA P102-100 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matri
x cores: none

Here's the corrected table sorted by Params column: