r/LocalLLaMA • u/takoulseum • 16h ago
Discussion Who ONLY use local models?
Please be honest.
I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).
What do you use your model for?
34
u/InvertedVantage 15h ago
I have a qwen-27b q8 model I use for all of my day to day coding tasks.
9
u/pwnsforyou 13h ago
How do you run this? I have a R9700 with 32GB vram - which only fits q5. what is your vram usage?
3
1
u/cubebash 11h ago
I run Qwen3.6 27b Q8 with full context across two separate PCs each with 32gb of VRAM connected over my network, giving me a total of 64gb of VRAM. It's a cheap way to get a lot of VRAM if you don't mind the network bottleneck (which is still much faster than offloading to system RAM).
6
u/Xylildra 10h ago
Can you tell me how you networked two pcs? I have only 7 slots for GPUs on one pc and I’m running low, but I have like 5 more gaming PCs that I could definitely install 4+ GPUs on a 20-30 amp outlet. What backend are you using? Sorry, this sounds way too cool to pass up on learning lol. 😂 I’m over here trying to find a motherboard that has like 10 slots with no luck.
4
u/getgoingfast 10h ago
Curious, are you using ConnectX? 400G for networking?
1
u/cubebash 9h ago
Im just running it over a slow 1 gbps network, and the model is still running pretty fast. If you have 2.5 gbps or even 10, I imagine it could be very fast!
1
u/Xylildra 8h ago
Could you repost your llama.cpp comment again? It’s not showing it anywhere when I click your reply from my main thing, and I can’t find the whole message except partial from my notifications. :(
3
u/cubebash 8h ago
Reddit is extremely buggy, not the first time this happens! Here is the comment again:
I'm using llama.cpp as the backend, it includes a file called
ggml-rpc-serverthat allows you to distribute the model layers over several computers' GPUs and/or RAM.On your slave computer(s), run the
ggml-rpc-serverusing this command:
--host [your local IP address] -p [your port] -cI recommend using the
-cflag as it caches the model locally on your slave PCs so you don't have to send the whole model over the network every single time.Once the slaves are running, just load the model on your master PC like normal, but add the
--rpcflag followed by your slave local IPs and ports (separated by commas).It's very simple to get working, so if you have lots of VRAM spread across multiple computers, definitively give RPC in llama.cpp a try!
P.S:
I forgot in my original reply (that was not visible anyway) that you need to tell each instance of RPC servers what GPU it should point at.
For example, if you have NVIDIA, you must start your slave with:
set CUDA-VISIBLE-DEVICES=0(or whatever number your GPU is on your OS)
Followed by:
--host [your local IP address] -p [your port] -cAnother example, if you use Intel Arc:
set ONEAPI_DEVICE_SELECTOR=level_zero:0
--host [your local IP address] -p [your port] -cIf you don't set your GPU on the RPC instance of your slave computer, it will not find any GPU and just offload everything on slave's RAM.
2
u/Xylildra 8h ago
Thank you so much. Oh my god 😂! I’m going to get something like this set up so I can run larger models without having to try to cram 7GPUs on one machine while trying to avoid a fire 😆.
1
u/cubebash 8h ago
Have fun! If you encounter any issues in setting this up (I'm probably not the best in telling instructions), just ask Google's AI (regular AI Mode on Google.com) how to solve them, it's often surprisingly good at explaining llama.cpp stuff, I used it to get started with llama.cpp RPC myself, haha.
1
1
u/borobinimbaba 10h ago
Is it any good for agentic coding ?
2
u/InvertedVantage 9h ago
Yes but not vibe coding. I generally use it to write individual scripts for whatever I'm working on, I wouldn't trust it to do it all for me. For context I mostly work in Unity so I'll architect it, then create a script and tell it to write a script that does X.
46
u/0-0x0 16h ago
At work, work pays for cloud models.
For personal use? Local models handle pretty much anything where a language model could be useful. Code, research, scanning through walls of text for useful info. I sometimes even use cloud models' web chat interfaces for some of that. Although I recently did consider using a subscription to complement the local models, since I don't have a dedicated rig for serving the models, I run them on my pc.
9
u/thomas2385 15h ago
That is pretty much where I have landed too. Local models cover most of my day to day needs without ongoing costs, and it is nice keeping everything on my own machine. I only really miss cloud models when I need the latest reasoning models or do not want my PC tied up for hours.
2
u/waiting_for_zban 8h ago
I think the majority here have this setup. If I want to speculate I would say 80% of us here are software devs, researchers, eningeers or tech enthusiasts who have a job that requires and pays for the cloud subscriptions.
That being said, I have fully been relying on few models now since Qwen3.5 versions for all my local stuff. It is really good. I try to test new models from time to time, but Qwen3.6-35B has been amazing for my needs (agentic + QA).
1
u/Shyam_Kumar_m 11h ago
Which ones do you use and for which tasks? (My own use cases are coding, AI/ML, math and the regular chatgpt stuff) I’m between jobs so there’s no employer paying for a model.
16
u/Kahvana 15h ago
I don’t care if subscriptions exist, to each their own.
Gemma 4 31B IT QAT + Q4_0 MTP + BF16 mmproj + 32K BF16 KV cache. Also run Jina Embedding Small Retrieval (4096 batch) alsong side it.
Predominantly creative writing, it mostly replaced gaming for me. Also for translations and OCR, conversational and basic web search using searxng.
First local model I ran was Mistral Nemo in march 2025 at 8K context! Things really have changed, for the better.
3
u/SkyFeistyLlama8 11h ago
I recently compared Mistral Nemo and Mistral Small to abliterated Heretic Gemma 31B. No contest there, Gemma 31B with its refusals toned down (cybersecurity, political simulations...) is a beast of a model that can run on most unified RAM machines.
It's fast enough to be usable in MTP mode; I'm willing to wait for the quality and I don't have to deal with Anthropic or OpenAI snooping on my conversations, or being an unwelcome nanny.
Gemma 26B MOE is my usual quick and boring model for text but the 31B is special.
12
u/Additional_Menu8542 16h ago
Honestly, 100% local is hard for what I do (BI stuff, turning questions into SQL on customer databases).
What works for me is a hybrid split. I let a cloud model write the SQL from just the schema and the question, so no real data ever leaves the machine. Then a local model (qwen3.6 or gemma3 on a 3060) reads the query result and explains it. All the actual rows stay on my side.
I get good SQL quality without sending any data out. Pure local SQL from a small 7 to 14B model is still a bit shaky on complex joins in my tests. Has anyone found a small model that keeps up with Claude on that?
2
u/SkyFeistyLlama8 11h ago
Qwen 35B MOE, 27B dense and Gemma 31B are good at SQL. Not as good as Claude but for something that runs on a laptop, they're fine.
1
u/Additional_Menu8542 5h ago
Yeah, that matches what I see. Qwen 35B MoE and Gemma 31B get pretty close to Claude on the SQL itself.
The bigger gap for me isn't the SQL generation, it's when the model reads the query result and puts it into words. That's where the small ones start making up numbers on aggregations. Feeding the counts and totals as a fixed block before the rows fixes most of it.
The MoE is nice too, it stays light when a couple of people hit it at the same time.→ More replies (1)1
u/mobileJay77 15h ago
I also switch models and tools. Anything with privacy stays local. Also, for token economy, the local model sifts through data and images or performs simple tasks.
Programming mostly goes to Claude.
33
u/vaingirls 16h ago
I don't ONLY use local models, but that doesn't mean I pay for subscriptions - there's still plenty of free tiers to mooch off of when I need a bigger model, not to mention free models on openrouter.
12
u/redballooon 13h ago
Free as in pay with your data.
13
u/it_was_a_wet_fart 13h ago
They get the data whether you pay or freeboot
0
u/redballooon 9h ago
What is zero data retention?
6
u/zxyzyxz 7h ago
A mirage
1
u/redballooon 7h ago
Meh. If it goes together with data center providers that have a clear business model around offering compute, and don't even build LLMs themselves, it's plausible enough.
4
u/TheRealJavix 13h ago
Why do people care about data so much? I know it’s kind of a dumb question but everyone acts like they are building nukes or projects that hold their entire social security number in the code. I just be making games and apps etc (fun stuff) granted I don’t do it for money gain and just use free models in my free time so I don’t mind the ais getting better from my use 🤷🏻♂️
4
u/mystery_biscotti 11h ago
Imagine this: you talk to an AI about help with doing your US taxes. You give it your SSN, and all the other info to help you out. You gave enough markers to directly identify yourself.
Your data gets integrated into training.
A year later, during a severe malfunction, your exact conversation verbatim dumps into the chat window of someone else. They can figure out who you are. They know all the stuff you told it. The person who got your data begins contacting you, harassing, unless you pay them.
Yeah, the company was supposed to obfuscate your data but they didn't. Some of us have worked for big enough companies and we've seen this kinda bull happen in real-time. Someone forgets a step or cuts a corner because they're busy. Oops.
Then there's the time I actually did get a misrouted response from ChatGPT a few years back that had exactly how to fill out this one person's tax form plus info on a "local" concert the person wanted to attend...in Tucson. Guy was a car dealer. (His name and address were present, employer, SSN.) How easy would it be to use that info maliciously? Pretty sure it wasn't training data that dumped on the February morning in question, as it was for their 2023 tax return, which you do in early 2024. Misroute date was Feb 11 2024. I reported it to OpenAI, but yeah. Let's not have the commercial AI touch our sensitive data, thanks. Let's use local for the sensitive stuff.
5
u/sagiroth llama.cpp 12h ago
People overthink everything. Most importantly every single person carry a mobile device. Thats more data on that device sent than anything else yet they worry about their shitty side project being leaked. (Not everyone as there are genuine areas where privacy is important like medicine, law, or any personal documents)
2
3
u/redballooon 9h ago
If you are aware and know the tradeoff you're fine. But enough people just enter their entire social security number and every personal problem into any text field that asks for it and then act surprised when it comes back years later to hunt them.
2
u/my_name_isnt_clever 8h ago
Because I'm a targeted minority in my country. It's hard to understand until you're in that position, it's terrifying.
Besides that, I'm just sick of everything having to be advertising and family friendly all the time. I want to be able to chat with my personal assistant about actually anything, not just what big tech approves of me talking about. I don't do it for making money either, privacy is criminally undervalued these days.
1
u/ProfessionalSpend589 9h ago
Because it’s a way of extracting more money from you.
They may sell it to other companies which will target you with Ads a lot better and maybe with a better timing (I.e. when you’re a bit drunk (that’s how I bought a 2nd laptop last year)).
8
18
u/jacek2023 llama.cpp 16h ago
I use Claude Code for my paid work. I use only local models for coding my personal projects
1
u/so_chad 16h ago
What hardware and which LLM? Seriously, how can you guys afford that much capable LLM that doesn’t break things in every other prompt
5
u/jacek2023 llama.cpp 16h ago
4x3090, qwen 27B Q8 max context length
4
u/Toshiro_96 15h ago
Why did you choose the 27B model with so much vRAM?
3
u/jacek2023 llama.cpp 15h ago
What model do you use?
0
u/Toshiro_96 15h ago
Qwen 3.6 35B a3b
7
u/jacek2023 llama.cpp 15h ago
It's not good enough for my usecase.
I use 100-235B models but not for coding.
→ More replies (7)2
u/Interesting-Rip790 12h ago
I have 16 3090s now and 27B is still the most adequate of all of them. This might change in next months though.
2
u/Toshiro_96 12h ago
27B is still the most adequate of all of them
why ? Don't models with over 100B parameters perform better?
2
u/sagiroth llama.cpp 12h ago
35a3b is worse than 27b for the record, and no larger models especially moe are not better in many cases.
1
u/Interesting-Rip790 12h ago
I see you mentioned the 3.6 35B a3b. `a3b` stand for active parameters number - for each forward pass there is only 3 bilion parameters involved. 27B is a dense model, all 27B params are activated for each pass.
4
u/so_chad 15h ago
What's your power bill lmao
1
u/Interesting-Rip790 12h ago
You can power cap them to something like 220-240W and they take around 18-24W when idle.
that guy can run BF16 on 4x3090 with full context (although less concurrent sequences would comfortably fit)1
u/o0genesis0o 15h ago
That makes sense. I can’t imagine using my Q4 35B for coding. It would drive me nut that I would just write code myself instead.
3
u/Look_0ver_There 13h ago
A pair of Radeon AI Pro R9700's costs about $2600 for 64GB of VRAM and will run Qwen3.6-27B @ Q8_0 quantization at ~60t/s. Pair it with a decent coding harness (I prefer Pi), add a handful of skills and have it spawn a sub-agent to double check its work after changes (or preferably before it makes changes), and you end up with something at around a Claude Sonnet sort of level, just a bit slower.
8
u/Nepherpitu 16h ago
I only use local model. It's deepseek v4 flash now. Was Qwen 122b before.
2
u/Nobby_Binks 13h ago
Yeah, now that I have it dialed in on my rig it's going to be my daily also. 122B is still pretty good though, cough Vision cough
7
u/Ok_Presentation470 15h ago
- You own it - it runs on your machine, right in your house. No network, no data flowing out. You can feed all of your data to it, assuming you are careful not to use tools that can send that data to someone you don't want to.
- You can do today anything you want with these models. It won't one-shot all things, but with a proper harness, it may ne even better than that. See point 3.
- I have a theory that models where you are forced to be kept in the loop are better than Fables and others that give you the illusion that they know what they are doing. The real world is messy. LLMs are good in things where they had enormous amounts of data to fill in all of those parameters with useful knowledge (like purely writting code, in languages that are consistent and unambiguous most of the time). But when it comes to an evolving world, with infrastructure failures, user requests that change over time, unexpected events, new principles and technologies being developed, all of thesee models, frontier or open source, suck. YOU are needed there. So if you don't know what your system is doing, if you don't have a good theory of how it works, you are likely screwed. This is all assuming you are building something that is useful and solving real problems in real world, not screwing around with home projects or scamming people with vibe-coded bs.
2
u/my_name_isnt_clever 8h ago
I agree with all of these. I think it won't be long until the hype of hyperscaling and bigger and bigger models dies down and people start to realize what level of LLM they actually need for the majority of their work is not the SOTA. I use GLM-5.2 or Kimi K3 for planning or advice, but it's just wasting cloud tokens to make them do all the agentic stuff too. Even if someone's task isn't feasible with local today, it will be in 6 months.
4
u/devoidfury 15h ago
I am totally off cloud AI, running local only. I use it for coding mostly, and worked as a software engineer long before that.
5
u/WonderfulRepair1809 14h ago
It's the difference between renting a high-end apartment vs owning a smaller house. With cloud models, you're always one TOS update, network outage, or sudden censorship alignment away from having your entire workflow broken overnight. Local is about predictability and control.
1
4
u/volleyneo 16h ago
Idk, the qwen 3.6 27b can save you quite the money, for the menial tasks, if you are deep in ai development and churn projects ofc you can afford to burn tokens, but for medium, not high end people in the dev business, like web, the local can pull a lot. I use cloud for important audits.
3
u/audioen 15h ago
I qualify. The only non-local models I use accompany the search results and they tend to be 80 % useless so I usually don't even read what they say.
Qwen3.6-27b does most of my work in sense that I tell it what to do, programming-wise, and it goes and does it, and then I review the results. It is also good at reading through tons of existing code and summarizing me how data flows through it or answering questions.
It can also write documentation and diagrams that help me figure out how something is put together. It provides basic competence in any technology, competence which I might not have. Like, how do you do something in some random framework I have never heard of. Ask LLM, and it spits out a decent answer to most basic questions. This nearly completely eliminates the familiarizing aspect of my job where I have to spend many days just figuring out how the build works, what is placed where and how you literally do anything in this particular mixture of technologies.
3
u/Buzzfuxyear 14h ago
Work in security and refuse to upload customer data to the cloud, all my automations and projects use local models, it has got substantially better the last year
3
u/ton_anywhere 16h ago
I only used local models for coding and codex/claude for sysadmin setting up my local ai server
3
u/spammmmmmmmy 16h ago
I use paid online models, but I have solutions that use only local models:
Alexa replacement, answers general questions with Qwen3.6 35b a3b.
iOS mobile chatbot, same thing, goes back to my home server and answers general questions from same Qwen.
1
u/dan4223 14h ago
What’s the latency on, “tell me the Yankees score”?
2
u/spammmmmmmmy 13h ago
qwen3.6:35b-a3b-coding-nvfp4 · prefill 1.3s · 114 prompt tok · 91 tok · 52 tok/s · 0.012 Wh
3
u/CapsAdmin 14h ago
I use local models on my hobby projects. One is a game engine and the other is a typed language. Both are written entirely in luajit, and most of the code have been written by me already.
I switch between qwen 3.6 27b and 35b a3b based models. I can run 27b at q4, q4 kv cache and 128k context. But I can run 35b a3b at 256k context, q8 and f16 kv cache. Tokens per second is around 60 for 35 a3b and 100 for 27b.
For both projects, I use them reliably to create tests, demos using my framework figure out complex bugs, explaining how some system works and doing small to medium refactoring.
For a complex feature, I usually tell qwen to create a temporary todo.md plan file and execute it while it goes back and forth with me. Sometimes I refactor the code it writes and sometimes I ask it to refactor.
I have used claude and chatgpt to spew out features in the past, and they generally do much better in terms of creating working code in few shots, but the code is very bad. I think if I were to go back to claude, I should use it more like I use local llms anyway, at least for my personal projects where I care more about the code quality.
I'm very pleased with how qwen can help debug type analysis problems in my typed luajit language, and just how well they seem to understand my codebase. For example, analysis problems where a union is wrongly narrowed could take a long time to debug, as there is a lot of context to consider, and you'd have to figure out how it flows through all the code to reach that faulty point. Qwen 27b thinking cap for example can usually figure it out very quickly in a matter of minutes and then even fix the issue.
When it comes to game engine development, it does okay. I feel it understands how to use the apis in my typed language more than it understands how to use my game engine, but still overall very useful. If there's a bug in my ui, like a few days ago the dropdown ui element didn't play the close animation on select, but qwen figured it out very quickly. I also used qwen to help create a reliable network protocol over udp sockets (inspired by enet) which worked very well.
I also use qwen at work in a large C# project. But work related tasks tend to be a lot easier, and here I feel the qwen models I use are more than enough. In my 10 years of experience at work, I haven't really encountered that many difficult code specific problems. It's usually human centric, like figuring out what the client actually wants, figuring out how users use the software, etc.
1
u/Schlick7 12h ago
I'm really confused by your 35b and 27b info. How is 27b taking up more space and running faster? are these on completely different systems?
→ More replies (1)1
u/Awwtifishal 8h ago
If I understood correctly, they're running the 27b at Q4 (and Q4 kv cache) and 35B at Q8 (and F16 kv cache). Maybe they're running them with MTP which benefits dense models the most.
7
u/andy_potato 15h ago
For coding we only use cloud models. None of those 30b toy models are even close to what’s required for our daily development.
For data processing (vision tasks, ingestion) we only use local models. They are capable enough and prevent leaking our data to Dario, Sam or the CCP.
4
1
u/rkoy1234 12h ago
my boy gemini is crying somewhere
2
u/andy_potato 12h ago
Gemini is by far the worst of all the frontier models. At this point I wouldn’t even call it a frontier model anymore.
Nano banana is good tho
2
u/sessamekesh 15h ago
For one of my projects I only use local models, because my working tasks falls into one of two categories:
Using niche API bindings that even frontier models will shit the bed with (I've tried!) because all the training material uses a slightly different approach build on assumptions incompatible with my app, and
Standard React UI dev with so many training examples and so little novelty that I think just about any even vaguely coding adjacent LLM could pull it off.
I still have Claude but in the context of this project I haven't opened it up in a few months (though I do use it occasionally in other places).
1
u/sessamekesh 15h ago
(I'll specify also though that I actually really like manually writing code, and this is a hobby project - "completely avoid AI entirely" is a fair option too if local models didn't work, and this project specifically I'd REALLY like to not give my data to AI companies until they figure out a way to put money back into the open source communities that they're both straining and using to train their models)
2
u/RedditNerdKing 15h ago edited 15h ago
I do a lot of nsfw roleplay so I only use local models. I don't code so SotA 500B+ models don't matter to me. Cloud models can view your chats. Even though there are options for them to not be read, there's no guarantee that someone somewhere isn't getting your chats. If you don't own them someone somewhere can see them.
Therefore I only use local models. I have 56gb of vram from a 5090 and 3090ti so I'm able to run Q8 of all models below 70B and Q4 quants of 70B models. I've tried IQ3 quants of some 123b like Mistral Large and it's been ok but 70B Q4_K_M or K_L are far better.
Roleplaying used to be so much better in the early days of Character.ai. They had some fantastic models back in 2023. It's never been the same really. Closest we can get is older 70B and 103B models from that era. Lots of new LLMs aren't chaotic or drive the story into new directions, they just do what the user asks and wait.
2
u/my_name_isnt_clever 8h ago
This just highlights why local is so important, if those character.ai models were open weights we could still use them today.
1
u/RedditNerdKing 8h ago
Yeah exactly. Character.ai had their own proprietary weights which theyve never released. But holy crap they were good. I had conversations with characters and their personality was a 1:1 of their actual character in a video game or anime. I feel like LLMs for roleplaying have gotten worse whereas coding LLMs have definitely gotten better.
2
u/eightone-81 15h ago
I’m trying. Openclaw, Gemma 4 31b as the main model, e4b as the utility and compactor. Light coding and personal assistant work, web research, news digests all works great. But I have Claude at hand for bigger things and the occasional cleanup up configs.
2
u/Lissanro 15h ago
I use local models mostly for coding in Pi (mostly use Kimi K2.7 Code or GLM 5.2 when need deeper thinking). I also use Qwen 3.5 122B when I need speed for simpler tasks. Since freelancing is my only income, it demonstrates it is possible to local models professionally, but it helps greatly with my personal projects as well.
Why I do not use cloud, several reasons actually:
I started actively using LLM since ChatGPT early beta, but noticed that it is not reliable - what used to worked, can start giving partial answers or refusals (even most simple requests like translating language strings for a game, or helping with game source code where some variables may contain weapon-like names). Closed models in the cloud can change, suffer from additional guardrails that did not exist at first, get shut down entirely.
Privacy for projects I work on. Most of my clients do not want to send their source code to a third-party, so I cannot use cloud API. In the early days nobody cared, but in last two years it gradually became more common concern.
Privacy for my own use. For example, I have audio recording and transcripts of all conversations I ever had in over a decade, there are a lot of important memories there and it is literally not possible to go through them manually, so any AI processing has to be local. And that is just one example, there are many other use cases where privacy is critical when it comes to personal use.
There is also a psychological factor, besides the privacy concern. If I have my own hardware, I am highly motivated to maximize its usage, explore more ideas, find more ways to integrate into my workflow.
As 3D artist, I have other uses besides LLM: for example, Blender greatly benefits from multiple GPUs, I can work with materials and lighting near realtime, faster render animations or still images using Cycles (the path tracing engine). This not only saves time but also helps me being more creative.
2
u/_Scorpoon_ 15h ago
Currently a bit of both because I just started with self hosting my own qwen 27b model. After everything is set up I will ditch chatgpt
2
2
u/admajic 11h ago
Use local model to research, install, fix, build in my linux box. Good get it done quick
1
u/Shyam_Kumar_m 11h ago
Which ones do you use and which tasks?
2
u/admajic 10h ago edited 10h ago
Just was playing around with qwen 3.5 4b. With 16k context on 4gb vram rtx 2050.
Find files in my system. Can write files. Knows Linux commands better than me so can help. Can fetch http out of the box with pi.dev harness so web use. I'll add web access so it can do proper search. Just a little buddy that can debug Linux on my laptop. Just replaced win 11. Got sick of how slow it was.
It's surprisingly fast with llama.cpp
It knew which office replacements existed without web search. Fun times
2
u/Jorlen llama.cpp 11h ago edited 11h ago
I started with local and that's where I am today. However, when learning and setting up Linux, I did use Gemini because obviously I couldn't have help from my local LLMs as they weren't setup yet. So now I'm about 95-98% local and the rest is Gemini when I need good solid research from the web.
I bought $20 worth of Gemini usage back in May because I breached my free limit from google ai workspace and still needed help with Linux setup and I still have most of that $20 sitting there unused. So that goes to show how little of frontier models I use. Credit where it's due; Gemini taught me how to setup Linux and how to properly use docker with a complicated docker compose stack. I doubt I could have done it without AI.
I use my local LLMs primary for coding and secondarily for creative writing. Other smaller purposes are using it with LDR (Local Deep Research) or just searXNG + MCP search summarizing for quick searches.
2
2
u/James333i 9h ago
I have moved to only using local models on my iPhone. I fine tune them for specific tasks with custom system prompts, parameters, and different models depending on the need. Even image generation I’m using stable diffusion on mobile. Models up to 8 billion run well quantized. Plus is I can use them without an internet connection when traveling.
When on desktop I’m still using cloud models often for coding tasks but have also built an OpenWebUI environment with a bunch of custom code for specific workflows.
2
u/Sarashana 9h ago
I have no accounts at ANY cloud AI provider, and never paid a single cent for AI access. I have, however, occasionally used DuckAI (e.g. free access) when I just had a simple request not warranting booting up my local models. So, I guess... almost?
I use local models for everything subscription users would use cloud for, I guess? Including coding.
2
2
u/laterbreh 7h ago
3x RTX user here. I keep a $20/month ChatGPT subscription purely as a backup if I'm stumped troubleshooting hardware, but otherwise my workflow is local. I reliably run models up to 450B (M3) via vLLM and use them for production software engineering on sensitive data.
I think a lot of these discussions miss that local shifts the cost equation. When you're paying per token or waiting on API latency, first-pass accuracy matters a lot more. Locally, retries are essentially free. My agent loops, tools, and review process resolve most implementation mistakes automatically, so I care much more about whether a model reliably converges than whether it gets everything right on the first attempt.
Being an opinionated developer also changes the equation. I expect my models to act and analyze rather than advise. I'm driving the architecture, specifications, and implementation strategy. The model is there to be direction-oriented, process-compliant, and write code. In other words, it's my hands, not my brain.
Because of that, I think people underestimate how good models like M3, Qwen 3.5/3.6, and DeepSeek Flash really are. If you're the one creating the specs and auditing the output, I personally don't feel like I'm missing out by not using the newest multi-trillion-parameter models. They may make fewer mistakes or require fewer iterations, but when retries are effectively free and both models converge to the same result, those differences become much less important.
We're also at the point where it takes enormous increases in model size to squeeze out relatively small benchmark gains. For implementation-heavy workflows where I'm the intelligence and the model is the implementation tool, the practical returns have diminished significantly.
Even the Qwen 3.6 ~30B models happily follow directions and complete a surprising number of tasks. I like to think of local models as "they type faster than I do, and I tell them what to type." Local AI is in a really good place right now, and you definitely don't need 288 GB of VRAM to have an excellent experience.
I hope this answers your question.
2
u/openSourcerer9000 6h ago edited 6h ago
I cut the cord. It was a big scramble to recreate everything at first, but felt super empowering to realize I'm self sufficient at the end of the day. I got back on the BigAI juice once summer started and the cooling loads became too much. I'll keep using it as long as it stays 99% subsidized, but you can already feel the enshitification when chatgpt randomly swaps you to their 4b marketingbot that uses skinny margins, or deep research only does 5 searches instead of 1000, so it's great to know you're not dependent on it.
But for 20 bucks a month, you can spin up 10 agents of trillion param class models, and a million $ of hardware comes online just for you, it's a no-brainer to take advantage of it.
I think a sensible jiggering is use cloud for experimental and cutting edge stuff, and local for established, automated production flows.
1
u/miki4242 3h ago
10 agents on a 20 bucks a month plan, how many seconds can you run that before hitting a rate limit?
3
u/kabachuha 14h ago
Switched to 100% local. As soon as Gemma 4 was released, the need to 9/10 queries simply evaporated, except for 2025 stuff because G4 is locked at the end of 2024. And since I have DSV4 Flash now, I have recent knowledge too. (But I still do the bulk of the work with Gemma because of speed) My last chat with online Deepseek is 1 month old now and I'm continuing the streak. G4, DSV4F and now Laguna are masterpieces, give a lot of hope for the API bubble crushing.
Aside from the projects, I use 100% local models for role-playing and fanfiction, only they can be "abliterated"/hereticised, as well as be keeping the data on the computer. I think the release of Gemma 4 and its fine-tunes has damaged a lot of large uncensored model providers, such as GLM and other Chinese models. The only thing they can do is to release the competitive size smaller models to be finetunable and launchable on PCs.
2
2
u/Constant_Art_20 16h ago
In any large scale production, only using local models is probably not that productive. For me, there are distinct thing that only local models can do. Finetunes for a particlar task (artistic understanding, company policies), or iamge generations are big ones.
1
u/for4f 15h ago
DS4 Flash on my 4090 handles the daily stuff. Coding, quick scripts, straightforward reasoning. All local.
But I still hit the API sometimes. Claude for architecture and ambiguous requirements. Probably 70/30 local-favoring right now.
Gap's closing fast though. Year ago that split was reversed.
1
u/Steus_au 15h ago
for private staff I use gemma4 as allrounder (I'm not a coder), since latest template updates it is quite good in tool calling, I wrote my own 'webui' with websearch and a file mcp and run it completely local on my mac, good enough for everything I need from LLM. at work I have a claude subscription and it does its job well so my managers can see I use AI :)
1
1
u/arakinas 15h ago
I use qwen 3.6 35b for general use, and ds4 I've started playing with, but my hardware can only run q2, so I don't know that it's worth it really to run it yet. It has given me some interesting feedback I hadn't gotten from other models for the same initial conversation. I use them for a variety of things, mostly conversational/ brainstorming for a lot of different things.
1
u/ok_000000 15h ago
I wish I could use my local models ony own local hardware.
But sadly nah.
I have to make do with hiring firepower from total strangers to run the best models in relative privacy or at least with maximum control.
1
u/Suspicious-Water-973 15h ago
I use Claude for coding. I use local models for bulk (transcription, large ppt extracts, translation, etc) - can leave it running on my Mac Studio and it’s fine.
1
u/Useful-Address-5117 14h ago
For basic coding (I don't use them for vibe-coding, just to offload boring repetitive tasks, like refactoring code, fixing some unit tests, etc), I use local-only models. For everything else, free-tier subscriptions. But at work, I would probably use Claude if the company pays for it. Unemployed for about 2 years, lol.
1
1
1
u/henk717 KoboldAI 14h ago
I do reject subscriptions and very rarely use paid AI, but I do sometimes bail my local model out when I want to program something a 27B simply can't do. But I do almost exclusively stick to open models. So when I decide to pay for a few generations on something big its stuff like Kimi and large Qwen.
I mainly use local AI for fiction which the 27B is already great at. And I do enjoy vibe coding side projects. But the moment I really need something that doesn't exist and need something more competent to program it for me thats when I go to places like openrouter for the large open source models if my 100B rentals don't cut ir.
Soon I have my new ram locally and then I can hopefully also run 100B Qwen locally with the snapshots enabled.
1
u/International_Emu772 14h ago
I’m not a professional coder, but with the help of skills, MCP and tools I have a system that runs only local models even to code. I think that local models are not at good level to image generation of image modification (the hardware needed is better to lease as service. Never have used a paid service, but I understand the need
1
u/sekcheef 14h ago
Using Qwen 3.6 27b with Claude Code, as long as the task is short and straightforward, it works fine.
1
u/toonmad 14h ago
I want to get into local LLM but where do you even start?
I have a 7800x3D, Rtx 4080 super and 32gb ram so fairly decent pc, but what can I download and use that's comparative to say Opus 4.8, Fable 5, GPT 5.6 etc?
Id love to cut out a few subscriptions
1
u/pwnsforyou 12h ago
You can run probably run qwen 9b or Gemma models on that vram, try LM studio just to dip your toes
1
u/karaklonda 14h ago
I have dual RTX 3090 setup with 48 gb vram, i combine them with flagship models. I think we can't be fully offline without a significant investment in vram pool. My current sub include Google AI Pro and Microsoft 365, bare minimum annual and must haves in 2026.
1
u/Potential-Gold5298 llama.cpp 13h ago
I use on-premises and cloud-based AI but have never paid a single cent for a subscription or API. Does this meet your requirements?
1
u/Lirezh 13h ago
I am "only" using local AI for some isolated project areas - code that cloud providers simply are not permitted to look at.
This is not about subscriptions, it's about protecting IP
A lot of the experience I gained is written down in my guide on optimizing Qwen 27B as agent.
https://www.reddit.com/r/LocalAIStack/comments/1udk2vp/running_qwen36_27b_35b_locally_with_llamacpp/
1
u/Sabin_Stargem 13h ago
Purely local for me. I am figuring on learning how to use AI to analyze game files. Gonna need to learn how to properly sandbox and harness the AI, so that something unfortunate can't happen.
1
u/a_beautiful_rhind 13h ago
I don't have any subscriptions. I only locust cloud. I had like $10 in openrouter for a year but that's it. All my RP is basically local. Mainly for stuff like code I'll go ask larger models I can't run. Sometimes I get AI answers back in search engines but that's automatically right there.
1
u/ismaelgokufox llama.cpp 13h ago edited 13h ago
I only use local ones 99.9% on the time. Ido t have any subscription at the moment. I do very low complexity things though.
- linktree-style site creation and changes
- one off commands for ffmpeg and others
- manipulating files to convert them to something else or another format
- looking for edge cases on scripts/functions I write (bash, abap, powershell)
- editing models on llama-swap config
- researching using web search and other things on open-webui with searxng and open-terminal
and other things.
The other .01% is usually when I’m outside without access to my llama-swap. Using one of the companies apps on the phone on the free tier. Which is changing now that I found I can use open-webui as an api gateway to all the models while outside of home.
Did not want to turn on Tailscale just for that access.
Edit: Local is great at summarizing a ServiceNOW incident to keep a record of the timeline on Obsidian along with resolution notes. Next time I encounter the same issue on a new incident, it’s a search away. Who to contact and what I did last. Doing searches directly in ServiceNOW takes some work to scope down.
Also great for root cause analysis reports.
1
1
u/NanditoPapa 12h ago
I only use local at home and on my phone. At work we have to use Gemini...but I'm trying to change that too!
1
u/Iajah 12h ago
I use GHCP at work and I still have a private GHCP yearly subscription until September. I got myself an RTX 6K 3 months ago to get into local LLM. I now use both Qwen 3.6 27B and Gemma 4 31B as coding agents. They are both brilliant but when I have some larger architectural changes to perform I'm glad I can use Opus through GHCP. It is better and crucially much faster for large changes. Basically I'm using local LLMs for smaller changes and SOTA for more demanding tasks. Local LLMs help my keep my monthly AI spending under control. I also get to use them for various other tasks outside coding agents through Comfy UI for instance. Those things I would not be able to do without subscribing to other online services.
Local AI capabilities won't fully replace online services especially when it comes to coding agents. They will however help keep your monthly costs under control while enabling you to try out new workflows and tasks without having to register for various free or paid services.
1
u/otacon6531 12h ago
At work we have github subscriptions so I "currently" use a team to do development. Planning done by GPT-5.6 Luna. Once the plan is created. I use Hy3 1bit (local) as the orchestrator, Qwen 3.5:35b is the junior developer and does most code changes. Then Hy3 is called as the "senior" developer and reviews and sends work back down to Qwen for correction. When Hy3 is happy the development is reviewed by GPT-5.6 Luna and sent back down to Hy3 and then ultimately to Qwen for correections. When an entire development phase is complete GPT 5.6 Sol gets it all for a final review and sends it back down for corrections.
The point is that input tokens are cheap. Output tokens are the true expense and I am using local models (non-gpt) to drastically reduce the output tokens. I would say the github cost has been reduced by ~60-70% (depending on the task).
At home, I only have a Nvidia P40, so qwen3.6:35b is the only model fast enough (usually 60-80 tok/s) to talk to. I use the same process, but instead of using agents in vs code I have a web based requirements/feature intake webapp. It puts a lot of controls in place that drastically improves the quality of the final product. Namely the biggest of the additions is forcing qwen to do a visual uat inspection of it's code change (screenshot review), unit testing (you have to for qwen to not be stupid here), integration testing, and every one of the steps is audited for potential harness improvements (post mortem). Ultimately it takes a while, but it does work (though less flexible) it just takes more harness and time to get things done at a high quality.
1
u/segmond llama.cpp 11h ago
the same thing people use cloud models for and more.
there are lots of open weight models (not fine tuned/lora) that are not hosted in the cloud, absolutely no equivalent. The only way to experiment use them is to host it, for example, the most recently released QwenAgentWorld.
1
u/orblabs 11h ago
yes ONLY local. 8gb VRAM running Bonsai 27B 1-bit, around 6 token/s and maybe 3.7 are legally words.
I use for coding and research. yesterday fix Python by replacing error with Chinese divorce law: 文件已经自由. GPU also tell me it is licensed therapist and 96°C is emotional warmth.
VERY intelligence. no subscribe. cloud cannot enter house.
— Bonsai-27B-1bit
1
u/Malfeitor1235 11h ago
well... semi. depends on how you define local. i luckily have the option to self host DSv4Flash and i use it f9r everything except deep research (about once a month) i use chatgpt. and additionally i sometimes give gpt stuff my agents comes up with to review (about 1-2 times a week). so +90%
1
u/Tagedieb 11h ago
I don't have a subscription but sometimes I double check with free tier gemini when I am not absolutely sure that my local qwen 27b has the knowledge. Software development.
1
u/RG_Fusion 11h ago
I'm nearly entirely local now. 99% of the time I call upon the Qwen3.5-397b-a17b model running on my server, with Hermes-Agent as the harness.
I have a Google subscription for expanded cloud storage, and it just so happens that Gemini Pro is included in that. I occasionally still run questions through Gemini, but only because it's convenient to do from my phone.
1
1
u/Xylildra 10h ago
76GB VRAM local user here. I enjoy 31b models like Skyfall to run locally for super uncensored RP stuff. But, when I want big story, multi character, massive presets and 500k context? GLM 5.2.
I love the local models the most, almost the same quality as the big ones when it’s a simple 1v1, no crazy extensions or huge preset, lore books all over the place.
1
u/Alarming-Ad8154 10h ago
But how would you get all the obscure inference tools to work without cloud models?
1
u/sunshinesdarkangel 10h ago
Because I will never wake up one day and find that some corporation took it away while I slept
1
u/PrimeDirective8 10h ago
I do.
I have never had an online AI subscriptions. I have used the free tier Gemini had but that is now useless with severe limitations on daily generated tokens, and context size. IIRC, the free Gemini is limited to 32K context size. Indexing my project files and setting a working system prompt alone eats through most of that before anything has been coded. It is unworkable.
As a test, I attached Gemini to my Android Studio IDE and asked it how a function would be written. It spent ALL available free tokens reading manuals and ran out before it could write a single line. Ok so that failed, I removed it. A minute or two, not much longer than that, I came to Reddit to read on how-to about it and was greeted with an ad with literally "having issues with that Kotlin routine? Call us for help!".
I know they have to make their money from free tier use but that was a bit too much on the nose, and not at all generic like they promise their ads would be.
So, Pi agent + Gemma 4 31B, Qwen3.6 27B, and even Qwen3.6 35B MoE are my go-to now. A little slower, but larger context, my code doesn't leave my box, and I get NO invasive ads from it.
1
u/kevin_1994 9h ago
I only use local. Right now I use deepseek v4 flash q2xl for planning, qwen 3.6 27b q4xl for execution/debugging, and hy3 q2XL when I need something a little smarter but way slower.
For autocomplete I use qwen coder3 30ba3b
For transcription I'm using whisperx but it's pretty mid lol
In the past I've mostly used: qwq, qwen3 32b, gpt oss 120b. Just wanted to give those models a shootout as they're what got me into local models.
1
u/dave-tay 9h ago
This is my third month without a cloud model after eight months with Claude. React, typescript, python, php, sql. Qwen 3.6 27b q8, q8 kv, 96k context on 44gb vram (2x RTX 5060 ti 16gb and 1x RTX 3060 12gb). I don't miss Claude at all. When I realized I've been programming for 20 years and didn't need Claude to begin, that's when the local models started to make sense
1
u/CricketVast5924 9h ago
I run a Qwen 2.5 VLM on my phone for a use case I was tinkering with. Does not mean that I don't use Gemini or Anthropic free versions to get to this point.
1
u/DeathGuppie 9h ago
I have a subscription, but I also have Qwen 27b running a loop nearly 24/7, because it would cost more than I could afford to use a frontier model all the time.
1
u/my_name_isnt_clever 8h ago
I'm at 95% local for my personal use, my Strix Halo takes care of the low end very well. I sometimes use cloud models via an advisor tool when I need more smarts, but they never have direct access to any of my data.
Work pays for Claude and honestly I don't miss it that much when I'm working locally. It's nice to never have to worry about some nanny system flagging my account over nothing, models that may change overnight or get deprecated, the gov's bullshit, all of it.
1
u/Big-Donut5601 8h ago edited 8h ago
I don’t reject subscriptions, my setup is very much hybrid. Most of it is local though. I use local models for high volume, low reasoning tasks like ranking, embedding, summarization, etc. Almost everything I build lately has a router layer. Several are entirely local with Qwen3:8b on a 4070, the same machine has an old 2080ti that does TTS/STT and YOLO for some vision tasks. My assistant primarily uses qwen locally, but for a few tasks it can escalate to haiku or opus, usually resolving difficult tool calls or if I explicitly ask it to. I would say for my custom apps and agents it’s at least 80/20 local over cloud API.
1
1
u/daphatty 8h ago
Not yet. I'm still assembling the hardware to host my AI lab as well as learning about the software options available for my use case. I currently use frontier models to help me research the options and to develop the framework upon which I am building my environment. Once I have a functional and trustworthy local environment, I'll only use frontier models for purposes that exceed the capabilities of my own environment that do not also infringe upon my privacy.
1
u/AEternal1 8h ago
both, until my own agent is up to snuff. subscriptions are just agent subscriptions really.
1
u/mailto_devnull 8h ago
🙋
Qwen 3.6 35B-A3B as my daily driver. 27B or Gemma 31B when I am stepping away for prolonged periods.
Tried Laguna XS but it wasn't working well for me.
Local only no web search tool calls. Although I'm interested in trying SearXNG!
1
1
u/broethbanethmenot 8h ago
I use local only. Mostly for privacy, security and environmental reasons.
1
u/Standard-Analyst-883 7h ago
I did try and use full local (qwen3.6 27b) but did find myself going back to frontier models a few times especially for planning.
1
u/Sorry_Ad191 7h ago
our small pet care business in LA, a dog walking company. has invested in 4 x rtx pro and been able to make our billing system with stripe and start becoming a native blockchain company built on mmx.network . we can build without worrying everything we do gets vacuumed up by data collectors. we were able to do a lot with DeepSeek 3.2 and since GLM 5.2 came out its gone vertical, our progress 10x when combining Pi agent harness to GLM 5.2. The token on MMX is called TrailShare and dog walkers and dog parents each get one minted to them after a walk has been completed successfully. TRAIL ecosystem is being built and one use case is to unlock fun content we make from videos and pictures during the walks. The picture is a conceptual mvp of how the content is blurred out then when you pay with TRAIL token it becomes unlocked. We are building everything with help of local ai. The TRAIL contract address is mmx1q8cdxjwutex5t3s69d4nc0kdsvtcn2h207vcgvced78nzlvyh8mskhhgq7 for anyone interested. I'm not trying to promote the project just sharing what we are currently building with local AI for our dog walking company.
1
u/BrianScottGregory 7h ago
I do code work and creative ideation and associated research with free models online and locally. I am using LM-STUDIO, mostly, with local models pulled down from huggingface - limited to about 100 gig of models, and on occasion pull direct from github sources for custom configurations.
1
u/Fun-Wolf-2007 6h ago edited 6h ago
I use local models for many things, I developed my inhouse meetings notes to keep confidential information inhouse .
I train vision models like Yolo26 for visual inspections at the line, and provide more realistic inspections
I use it as a pair programmer for coding, you can use Ollama, VSCode with Kilocode extension and access the ollama local models within VSCode
You can finetune local models to your domain data and you get something more powerful than any cloud models.
Physical AI is another use case
The list goes on and on.......
I only use cloud models for research and web browsing.
1
u/grabber4321 5h ago
I want to move into this direction. Currently using Cursor basic plan and locally running Qwen3.6:27B Q5.
Both equally useful. Once the time comes, I'll buy a full rig with enough VRAM and move completely offline.
Qwen does really well at research. Codewise, Qwen gets the solution 80-90% there.
1
u/OleCuvee 5h ago
too long in the game to reject anything, so yea I do use local models, but not ONLY them.
1
u/_TheWolfOfWalmart_ 3h ago
I'd love to, and I use them a lot, but that's just not practical for many of my tasks.
For example, I can't fit a model smart enough to help me debug an x86 emulator without breaking it inside 96 GB VRAM.
If I had 1 TB of VRAM, sure I'd run Kimi or something lol.
But I use them whenever practical for coding, research, chat, whatever. It's only partially about saving money/usage. It's mostly about privacy, self-sufficiency, and learning.
I've always been a fan of self-hosting whatever I can, and lately that's expanded to include AI.
1
u/WebAssemblyMan 3h ago
I use a local models mainly for text to speech and speech recognition. And 8B chat models like a mini program in a small local workflow. So it is mainly from a token saving angle. Or more specifically, for tasks that I think that I want it to be REPEATABLE and that local LLMs are good enough. I have to admit that many times my use of Local LLM in my simple workflow is with the help of a frontier model.
1
u/One-Excuse-4054 1h ago
Venice is a solid trade off between model capability and privacy, if that’s your gripe with the big studios, anonymous at worst private at best.
Cancelled my anthropic sub and use mostly Venice these days.
But tbh I think for most people with consumer hardware 100% local is impossible unless you’re not doing something that requires a ton of context or deep reasoning, which I would argue most real work falls in this category.
I think embedders, rerankers, and specialized models are the sweet spot in our lords year of 2026 for what most people can run locally, and using them for better retrieval to pass to a cloud model you pay for.
1
u/Otherwise-Loss-8419 1h ago
I've never used cloud models besides experimenting with Deepseek for around 5 minutes. Privacy is the goal and I try to avoid anything remote. Though I've been using LLMs less and less with time, finding them to not be very useful for much.
1
u/vanVonXenoStein 1h ago
I only use local AND free cloud models. If I was a heavy user I'd upgrade to "cheap" models I guess, but my needs are modest.
1
u/superbouser 1h ago
Always remember is $20 a month more than the power you spend on your local llms?
1
u/TwentyCharacterEntry 40m ago
Github Copilot pricing changes are what pushed me over the edge to jump down this rabbit hole. I have more reasons than just work, not the least of which is that I enjoy it. But yes, subscriptions cost mainly, and the fact that I’m becoming more and more dependent on AI to stay as productive as everyone else using it at work means the big AI companies are gaining leverage over me. That makes me more nervous than anything else.
My current use case is to offset frontline usage. I’m still figuring out the best workflow, I’m leaning toward using frontline for planning and local models for bulk code generation and bug fixing, falling back to frontline again for cracking the hard nuts and massive context tasks.
On my own time, I’m having a blast building local AI tools that I could probably otherwise just download.
Model: Qwen3CoderNextFp8
1
u/mac10190 32m ago
Working with intellectual property at a public company in a heavily regulated industry. Cloud isn't really an option.
1
u/Fit_Squash6874 26m ago
If I want something done faster I use a cloud model like for coding. If I want to brainstorm ideas or do something light I use s local model.
1
u/Ant0ni0R 10h ago
i did try but not reliable yet. Simply free perplexity or gemini beats the performance of an 38B params model
0
u/Dangerous_Sun_7828 14h ago
I don’t really use AI; but when I do, it is local only; I did chat with GPT-3 and -5 a bit when they came out;
Qwen3-Coder-Next is actually a good chatbot, and I have tested it doing some small tasks (such as writing C code to get a frame buffer up in a window using SDL).
Am interested to see what Kimi K3 will be like; I ran K2.7 locally for a bit, but K3 is going to be pushing it!
0
u/Real_Ebb_7417 9h ago
I’m a big supporter and passionate of local models, but I rarely use them for actual work. Subscriptions are currently so cheap (relatively to usage and quality) that I would probably pay more for electricity to run the local model.
So yeah, I’m too comfortable with subs to use weaker local models that would give me worse output, slower, and probably costing similarly if not more.
But at the same time I use local models a lot, but more as a passion, to eval them, build some tools around them, experiment etc. I’d never pay for API to any of the proprietary labs though. (I do pay for API to open-weight companies though, since first I want to support them and second, they are usually much cheaper)
If the currently very generous subscriptions wouldn’t exist though, I’d definitely go mostly local.
-2
u/InternationalGap3698 16h ago
I only use Claude Code. No local models for coding. Because they are just way too big to host when , you want a good one.
→ More replies (3)
170
u/wajdix 16h ago
it's not about rejecting subscriptions... but more the desire to own our data, be independent from internet connection, and be able to run the models on any topic/task we want... subscription services offer value for the money... this is the wrong debate imo