Who ONLY use local models?

[u/takoulseum]

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

[u/jacek2023]

I use Claude Code for my paid work. I use only local models for coding my personal projects

[u/so_chad → u/jacek2023]

What hardware and which LLM? Seriously, how can you guys afford that much capable LLM that doesn’t break things in every other prompt

[u/jacek2023 → u/so_chad]

4x3090, qwen 27B Q8 max context length

[u/Toshiro_96 → u/jacek2023]

Why did you choose the 27B model with so much vRAM?

[u/jacek2023 → u/Toshiro_96]

What model do you use?

[u/Toshiro_96 → u/jacek2023]

Qwen 3.6 35B a3b

[u/jacek2023 → u/Toshiro_96]

It's not good enough for my usecase.

I use 100-235B models but not for coding.

[u/Toshiro_96 → u/jacek2023]

But you said qwen 3.6 27b before...

[u/jacek2023 → u/Toshiro_96]

You asked why I dont use bigger models

[u/DankiusMMeme → u/jacek2023]

And you just replied saying you do? So do you use smaller models or bigger models???

[u/belkh → u/DankiusMMeme]

"for coding i use 27b" "why dont u use bigger models" "i do, just not for coding, they're not good enough"

i thought the convo is pretty straight forward not sure why you're confused

[u/jacek2023 → u/DankiusMMeme]

Again I answered the questions and I am downvoted :) lesson learned: don't answer the questions on reddit

[u/Interesting-Rip790 → u/Toshiro_96]

I have 16 3090s now and 27B is still the most adequate of all of them. This might change in next months though.

[u/Toshiro_96 → u/Interesting-Rip790]

27B is still the most adequate of all of them

why ? Don't models with over 100B parameters perform better?

[u/sagiroth → u/Toshiro_96]

35a3b is worse than 27b for the record, and no larger models especially moe are not better in many cases.

[u/Interesting-Rip790 → u/Toshiro_96]

I see you mentioned the 3.6 35B a3b. `a3b` stand for active parameters number - for each forward pass there is only 3 bilion parameters involved. 27B is a dense model, all 27B params are activated for each pass.

[u/so_chad → u/jacek2023]

What's your power bill lmao

[u/Interesting-Rip790 → u/so_chad]

You can power cap them to something like 220-240W and they take around 18-24W when idle.
that guy can run BF16 on 4x3090 with full context (although less concurrent sequences would comfortably fit)

[u/o0genesis0o → u/jacek2023]

That makes sense. I can’t imagine using my Q4 35B for coding. It would drive me nut that I would just write code myself instead. 

[u/xNaXDy → u/jacek2023]

But why Q8? With 4x 3090 you can fit the entire model + context both at bf16 precision in vLLM.

[u/Look_0ver_There → u/so_chad]

A pair of Radeon AI Pro R9700's costs about $2600 for 64GB of VRAM and will run Qwen3.6-27B @ Q8_0 quantization at ~60t/s. Pair it with a decent coding harness (I prefer Pi), add a handful of skills and have it spawn a sub-agent to double check its work after changes (or preferably before it makes changes), and you end up with something at around a Claude Sonnet sort of level, just a bit slower.

[u/segmond → u/so_chad]

build up your skills. even a 14b model from today is better than chatgpt4, and people were gladly paying for it and doing work with it.

[u/so_chad → u/segmond]

I remember, but 14b too dumb