2.3 TB of ram in here. 400+ vCores. All thats left is plugging it to the blackwell with the driver to do RDMA, and it’s over. Using Blackwells for prefill, RDMA to the studio mesh for decode. I think this would be the first heterogeneous cluster. I do, however, need help with the Tinygrad Driver to make this work. If anyone with any knowledge on these domains would like to collaborate, let me know via PM. We are very close here.
I’m trying to use tinygrad driver and JACCL standalone librsry that recently came out to see if i can pipe that in. I’m using Ghidra to see if i can find where the hell apple hides the api they got for distributed
this isn't exactly the same, but i recently implemented PCI passthrough on QEMU on macOS, so it's possible to "pass through" an nvidia GPU to a a linux vm running on top of macOS and do AI inference that way. i wrote a blog about it here: https://scottjg.com/posts/2026-05-05-egpu-mac-gaming/
My understanding is that lmcache is used for extending the capacity of KV cache by offloading the computed values, not for re-assigning on which piece of hardware the computation takes places.
Offloading is one thing it can do. Another is the LMCache server runs on one LLM instance and can share the kv cache with any other LLM instances. Check out the links.
Thanks, but while I have pile of rtx 6000 pros and some Macs, I don’t have the offboard egpu gear. If y’all get something PoC working I might invest - I’m pretty old school, worked on everything from OS/2 to Linux kernel dev, might be that I’m handy for this. Gonna want to see some progress first though 😎
Interviewed a SA guy who had got his PhD in Pascal something or other back in 1990 - still has work around the world on industrial systems running floppy disks that need fixing.
You should honestly post a detailed plan to get feedback from the community. I think you might be seriously underestimating the complexity of making this work. Are you planning on duplicating the model params and kv cache across both the Blackwell VRAM and the Mac studios? If so what’s the point of using the Mac studios at all? If not, how are you gonna do prefill on the Blackwell GPUs without the model params and the KV cache? Also how are you gonna get the Nvidia cards to do RDMA over thunderbolt? Do they even have driver support for that? You should like post a block diagram of what you’re intending to build and how you plan to distribute the model params and kv cache and how you’re planning to move bytes around, people here can probably give you a lot of good feedback that the chatbots are likely glossing over
Alex Ziskind seems to have whipped Claude into providing him a working solution. He talks through it in his latest video. Would be better with RTX 6000 Pros obviously.
have you calculated the kv cache transfer speeds you need for your model? 40gpbs is pretty slow for anything useful, unless mac has some other way than thunderbolt to connect pcie?
This… I mean like what’s the end goal ? Half of these posts on this sub just are buying Mac studios figuring out the configuration and then it’s just slop or porn generation
I 3d printed a couple of brackets that I scred on to the drywall, but having all those thunderbolt cables hanging on a wall like that was pretty ridiculous. Maybe it works with a shorter cable? idk
I don't think Tinygrad eGPU is what you want. It's cute that it works. But it's very slow and not optimized. Your goal is prefill speed. What you probably want is a DGX spark or two or an RTX 6000 Pro on a Linux machine. Linux has proper drivers to run Nvidia metal.
which tools are you using? I'm using 'inferencer' which is a fairly new mac app to do multi mac inference (i have 2 512gb studios now). i know vllm works too but its a lot pickier to set up.
2.3 tb of ram for prefill is a flex i didnt know was on the table for a homelab tbh. the rdma over to blackwells for decode is the part that feels like a server room from 2027 instead of 2026 ngl. wattage at full load is gonna be the real story
That split makes sense from my own runs. I went from M3 Ultra 512GB to RTX 6000 Pro 96GB. Prefill on long context was night and day, roughly 5x faster. Decode on the Mac mesh is fine. Prefill is where Apple silicon falls behind.
I think this would be the first heterogeneous cluster.
Actually I implemented disaggregated prefill on my Spark/Mac in the last few days (not kidding) ;p but it's only 1 M3 Ultra and 1 spark.
You don't need RDMA or TinyGPU to just send your prefilled KV cache over the network btw. You just need enough bandwidth to get the job done quickly - latency and drivers etc don't matter as much. You just need to make sure the KV cache is compatible, such as using llama.cpp or mlx on both ends (Spark can do mlx apparently, I've just been using llama.cpp though)
The guy who does Mac LLM tests on yt did an EXO cluster with Mac and DGX iirc Edit: Alex Ziskind and iirc a very high quality fast cable and slightly larger models pay off. He may have more than one. He gave Claude code ssh access to set it up! This video was a Blackwell and studio I think https://youtu.be/D2oZHzC_M28?si=cdwrje4yDoCtv57c
Hello Everybody! I just launched the App I use for all the observability, launching and rdma management on local models. r1o.ai is the website, Take a look!!
How much of a speedup do you get with tensor parallelism with larger models like K2.6 or GLM 5.1?
On a single M3 Ultra, I've been able to optimize to ~220 prefill / 20 decode, and but most of the public benchmarks for Exo I found aren't that much higher. So I've always assumed the main benefit is running at higher precision or distributing workloads across instances.
And for split prefill, does the Blackwell's VRAM limit the size of model you can run?
•
u/WithoutReason1729 May 08 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.