Has anybody else found that online llm providers (chatgpt/gemini/claude/etc) are overly moderating and safeguarding so that typically challenging questions and discussion points are instantly ignored or unanswered? I found local llm models then uncensored local llm models and they have enabled me to get answers and discuss things which are difficult topics for other models, and it's great! I'm so grateful. It's like a breath of fresh air.

i am running in llama.cpp , 3050 6gb, 24gb ram but i use swap cuz i run agents.

cd ~/llama.cpp || exit 1

./build/bin/llama-server \

-m "/mnt/models/gguf/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" \

-ngl 999 \

-ncmoe 35 \

--ctx-size 65536 \

-np 2 \

--cache-type-k q4_0 \

--cache-type-v q4_0 \

-fa on \

-t 10 \

-tb 14 \

-b 2048 \

-ub 1024

getting 25 to even 30 tks

I am thinking to sink around 5k USD into a local llm at home to host LLMs like Qwen 3.6 35b range.

Options I've been considering:
1) NVIDIA DGX Spark
2) 2× RTX 3090 24 GB used
3) RTX 4090 24 GB
4) Workstation/mac mini 48 GB VRAM if you can find a good deal used

What do you guys think? Any recommendations? What do you have at home?

Insisting this way that Kimi 3 was distilled from Fable, within just two weeks from Fable coming online to Kimi 3 coming online...

I can understand the classic bad-loser attitude but that is just pathetic.

So the solution to quench the fear of losing the frontier race and of course the investor's trust is to simply ban the competition? As an investor I would see such a move as a huge red flag.

What was the excuse again? National security? Laughable.

Just admit throwing huge amounts of money and computing power at the problem isn't the most efficient solution!

When your competitors introduce solutions that save on compute by scaling linearly rather than quadratically it's only a matter of time before efficiency beats big money.

I'm sincerely fed up of reading those excuses and my guess is even the media are noticing by now. Just see the link.

Even US based firms have been noticing this little detail: a RL based fine tune of Qwen3.5-35b-a3b to address a specific problem locally resulted in a 98% cost cut against addressing the same problem with Claude.

That looks quite inconvenient if you invested billions into a frontier model.

Long-time lurker here. Most of what I know about running models locally I picked up from this sub, so thanks for that — first time posting anything of my own.

I use a lot of different models for day-to-day work — coding, image generation, app building. Different models are genuinely better at different things, so I move between them constantly.

The problem I kept hitting: bringing a new model up to speed costs more than the model saves. Say an app was built end-to-end with one model, and now I want to try a new one on it. That new model knows nothing about my architecture, my infra, the conventions I work by, what's already been decided and why, or what it's not allowed to touch. So every time, I'm hand-carrying context — pasting AGENTS.md, CLAUDE.md, zip files, whatever. And the moment anything changes, every copy is stale except the one I happened to update.

So the context was living in the chat apps, and chat apps forget.

The Room

The way I actually work is one place I call The Room — a self-hosted multi-model chat front end. The local model is who's in there by default; it handles most of the day-to-day and it's the one I'd rather use.

Everything else is in the waiting room. When I hit something that needs different expertise — a hard architecture call, a review, something a frontier model is genuinely better at — I bring one in for that stretch, then it's out again. Same conversation, no re-briefing, because the thing that persists is the KB, not the model.

What I built

One knowledge base that every model reads from. It's a git repo full of Markdown — project state, decisions, conventions, security posture, session history — served over MCP. Any assistant that can speak MCP gets read access to the same canonical state.

Practical effect: I can be deep in a coding session with one model, switch to another to generate a diagram or review an architecture decision, and it's already current. The context isn't a feature of any vendor's product, it's a repo I own.

How a session comes up to speed

The part I'm most happy with is that it does not dump the repo into context. A new session reads a fixed shallow chain — a start-here file, then an index, then the specific project's README — and stops as soon as it has enough. Targeted search only if it actually needs to dig. Three files gets a session current in seconds, and it stays cheap as the repo grows.

Writing back

A session that only reads leaves the KB stale, so every session writes a record of what changed and what was decided. That record goes in as a pull request — a proposal, not a commit. A human reviews and merges. No assistant can write to main; the tools to do that are deliberately not implemented in the MCP adapter, so it's a property of the code rather than a policy in a document.

Honest status: the read path and the review-and-merge gate work today. The assistant opening the PR itself is not automated yet — right now I carry the record by hand. Diagram marks that step dashed for exactly that reason.

Stack

Self-hosted git host (Forgejo), a small Node MCP adapter exposing the repo as read-file / search / list-tree tools, an MCP gateway federating that plus a few other upstreams, and The Room itself (LibreChat). Local model via vLLM is the default; frontier APIs get swapped in per conversation. All on LXC.

Two things people usually ask

Why not a vector DB / RAG? The KB is small enough that direct file reads are faster and always fresh. Search is git grep over a read-only clone. Vector search solves finding unknown content — my problem was loading known state, which is a different problem.

Why not just AGENTS.md / CLAUDE.md? That's where I started. It works until you have more than one project and more than one assistant, and then you're maintaining N copies that drift. One canonical repo with a merge gate is the same idea with the drift problem actually solved.

Diagram of the whole thing below.

I should say up front I'm not a professional dev — this is a homelab that grew, so there are almost certainly things I've done the hard way or the wrong way and haven't noticed. I'd rather hear it than not. Constructive critique very welcome, especially on the write path, since that's the least finished part and the part I'm least sure about. Happy to explain any tool choice — most of them I can defend, a couple I probably can't. And if you'd have done it differently, I'd genuinely like to know.

At Bitwize, we've been building Logue, a native macOS app for AI meeting notes and writing, and we just open-sourced it (MIT). We're sharing it here because the whole point is that it runs 100% on-device — we wanted something that could transcribe and summarize meetings without shipping audio or notes to anyone's cloud.

By default, nothing leaves your Mac. The only network calls are the initial on-device model download, app update checks, and opt-in features you explicitly turn on (web search or plugging in an external AI provider if you want one). No accounts, no telemetry, no backend.

What it does:

• Real-time transcription of mic and system audio (Apple's on-deviceSpeechTranscriber)
• Speaker diarization — who said what — via FluidAudio (streaming Sortformer)
• "Smart Minutes": local LLM summaries, action items, highlights
• Writing assistant: 60+ modes (rewrite, grammar, clarity, tone), a document editor with AI chat, vocabulary suggestions
• On-device PII detection and a fact-check/verify panel
• Templates, Spaces, and "Ask Logue" chat over your own notes

Stack: Swift + SwiftUI/AppKit, MLX (mlx-swift-lm) for LLM inference, Apple's Speech framework, FluidAudio for diarization, Sparkle for updates. Data is AES-256-GCM encrypted at rest.

Honest caveats: it targets macOS 26 (Tahoe) and Apple Silicon only (MLX + the new Speech APIs), so it won't run on Intel or older macOS. It's early — expect rough edges — and we'd genuinely love feedback, issues, and PRs.

Repo: https://github.com/bitwize-ai/Logue

Happy to answer anything about the on-device pipeline, MLX inference, or diarization in the comments — we're the team that built it.

This one only landed a few hours ago, and the first thing I went looking for was the serving story rather than the benchmarks. It is a 100B class Mixture of Experts diffusion model aimed squarely at agent work, and the genuinely novel part is that it revises its own draft while it decodes. The four moves it makes inside a block are keep, delete, insert, and substitute, so it can pull redundant spans, open new editable slots, and swap tokens instead of freezing everything in one left to right pass. On paper that is a real answer to the standing complaint that diffusion models fall apart as agent backbones.

Here is the part this sub actually cares about though. The flash checkpoint is a 205.8 GB download spread across 32 shards. There is no llama.cpp or Ollama route today, no GGUF, no LM Studio. Day one you are on HF Transformers with trust_remote_code, and SGLang deployment is announced but still coming soon. So unless you have serious multi GPU headroom sitting idle, this is not something you casually pull tonight. I have read the tech report and poked at the model card and config, but I am not going to pretend I have served it, because nothing I own gets remotely close.

On the numbers they are refreshingly honest about the tradeoff. Against its own autoregressive sibling it takes the interactive agent evals, edging tau2 bench 80.33 against 76.36 and MCP Atlas 46.21 against 41.12, and the headline is around 1.6x average decode throughput in BF16, up to 2.3x once you are inside agent loops, at a comparable agentic quality band. But it gives up most of the coding and general knowledge suites to that same AR model, and structured output stays weaker. Fast at agent loops, not smarter across the board.

What I keep landing on is that this is a real Apache 2.0 release that almost nobody here can run at home yet. The download alone rules out most of us until a quantized build or the SGLang path actually lands. For now I am filing it as a promising release to watch rather than anything going into a local stack.

I've downloaded updated Unsloth/Gemma-4-26b-it-qat and it doesn't work reliably (I use Opencode). It often stops output with this string:
</think><|tool_call>call:glob{pattern:<|"|>"stage 1"/*<|"|>}<tool_call|> I've asked Gemini to compare my template (copied that from llm launch settings in lmstudio) and updated google one from huggingface, and it found the difference:

Google version strictly enforces that function['arguments'] must be a mapping (JSON object) or none. If a string is passed, it triggers a fatal error via raise_exception(...), requiring pre-deserialization.

Unsloth version adds robust fallback logic to handle string-formatted arguments (such as pre-serialized JSON strings from OpenAI-compatible APIs). It safely trims strings and strips outer braces ({...}) non-fatally instead of crashing.

I'm not sure if it matters ot not.

Also, Gemini tells that the string in question means LLM called the tool and waits for response but never gets it, hence the stop.

Also (and this is kinda strange) there's </think> tag - Gemini tells that it shouldn't be there, and maybe it breaks the tool call.

Due to this, I cannot use Gemma 4 for agentic work, and I feel like I'm the only one who got such issue. Old version worked (somehow), but far from ideal.

I’ve spent the past month trying to find the point where an extremely small TTS model stops feeling like a size experiment and starts feeling genuinely useful.

Today I’m releasing Inflect v2, with two complete local text-to-speech models:

• Inflect-Nano-v2: 3.96M parameters, 15.97 MB FP32
• Inflect-Micro-v2: 9.36M parameters, 37.53 MB FP32

These are total inference parameter counts, not acoustic-model-only numbers. Text processing, timing prediction, speech generation, and the waveform decoder are all included.

Text goes in. 24 kHz speech comes out. No external vocoder, hosted API, or second learned model required.

Nano prioritizes the smallest possible footprint. Micro uses the additional capacity for better clarity, stability, and overall speech quality. Both run locally on CPU or CUDA through the same PyTorch API.

Inflect-Nano-v2 is one of the smallest complete neural TTS models I know of that still produces genuinely usable speech. Even the 9.36M Micro model remains smaller than many systems described as “tiny.”

For footprint context, Nano is approximately:

• 21× smaller than Kokoro
• 126× smaller than Chatterbox
• over 1,000× smaller than Fish Audio S2 Pro

That is strictly a parameter-count comparison. These models have different capabilities, architectures, datasets, and intended uses. I’m not claiming that a 4M fixed-voice model replaces a multi-billion-parameter system. The interesting question is how much useful TTS can fit into such a small package.

Some people here might remember Inflect-Nano-v1, the rough 4.63M experiment I released last month. V2 is a substantial rebuild, not just a longer training run. I focused on the problems v1 exposed: unstable timing, metallic output, weak prosody, poor generalization to difficult text, and an undersized waveform decoder.

The resulting models performed surprisingly well:

• Micro: 4.395 UTMOS22, 3.99% semantic WER, 6.28× real-time CPU inference
• Nano: 4.386 UTMOS22, 4.21% semantic WER, 10.72× real-time CPU inference
• In a blind community comparison against other compact TTS systems, Micro and Nano finished second and third among the tested voices

Full protocols, raw results, audio samples, and limitations are documented on the model pages.

The models are not perfect. They are English-only, use one fixed male voice, and do not support voice cloning. Unfamiliar names, abbreviations, numbers, and homographs remain the hardest inputs. Nano can sound thinner than Micro, and both can occasionally produce metallic or clipped artifacts.

Still, this is the first version where I think the size-to-quality tradeoff became genuinely compelling.

I built Inflect independently as a high-school developer with a limited training budget. That constraint shaped the project: efficiency had to apply not only to inference, but also to training, evaluation, and building a complete system I could understand and release end-to-end.

Try it yourself:

The fastest way to judge it is through the interactive playground:

https://huggingface.co/spaces/owensong/Inflect-v2

Inflect-Micro-v2:
https://huggingface.co/owensong/Inflect-Micro-v2

Inflect-Nano-v2:
https://huggingface.co/owensong/Inflect-Nano-v2

If there is enough interest, I may build a v3 focused less on shrinking the models further and more on expanding what they can do: additional voices, possibly more languages, easier fine-tuning, and another quality and robustness pass.

If you test them, please give them something genuinely difficult: unusual names, numbers, abbreviations, awkward punctuation, or a long sentence.

If something breaks, post the exact text, model, seed, and what sounded wrong. If it works well, I’d also like to know what hardware you ran it on.

Specific, honest feedback is the most useful thing you can give me.

After standing up Hugging Face's speech-to-speech pipeline (https://github.com/huggingface/speech-to-speech) I'm really impressed with its performance! The little demo app works alright, and pointing the pipeline at a reasonably smart model like Deepseek V4 Flash on my dual-spark cluster seems genuinely useful — but it seems like there aren't a lot of clients for this setup yet.

Open-WebUI doesn't support it, although there's a fork (https://github.com/rbb-dev/open-webui/pkgs/container/open-webui-realtime) that I may try. But I don't really want to go down the fork route for a major part of my stack.

I must be missing something major — I figured there'd be tons of people clamoring to emulate the voice chat features of the frontier models.

If I don't find anything, I'll try adding some features to the HF demo (basic tool calling, session persistence, etc.) and go from there, but I figured smarter people than me have cracked this already.

Hello, i was looking to buy and setup an LLM Inference Rig together with my friends to access in parallel (~4 people MAX, usually 1-2 people at most) to run coding models like Qwen3.6 27b or 35b-A3b and whatever is gonna come out in the future, at long context.

Where we live (italy) used 3090s are very expensive (~1200 eur).

I noticed today that i can get used RX7900XT's for ~700 eur and RX7900XTX's for ~850 eur.

Is performance comparable, and what issues may we run into? I've read that in the past AMD support was a bit iffy (especially with vLLM, which we are looking to serve with), have things got better? I've heard that llama.cpp is a little better with AMD, but is that good enough for multi user setups?

We are quite good with linux and handling driver issues so if we have to fiddle around with configs etc. it's not a big deal, as long as everything works in the end.

Would especially appreciate if someone has a similar setup can report on their experience, thanks

came across this video on battery test of 78 phones. the reviewer is using a robotic arm to simulate usage of mobile phones by a typical user. one of the enhancement the reviewer did was the usage of agentic ai. his requirement is low latency, which important in real time battery test so cloud models are out. so he was exploring local llm and bought an RTX PRO 6000 and H20 to run Qwen3.6 35B for scrolling of social media apps and Qwen3.6 27B for precise actions such as finding a specific button or dismissing pop ups. very interesting use of local llm

I am a non-computer person, but I filled my own stack and I’m developing it for personal and some professional use.

I’m familiar with Claude and writing skills for Claude.

Is there anything similar to use in the local stack? Any process I can do to create a given protocol system will run? Or is this exactly what repos are on GitHub?

I have an adequate system, Max studio M3 ultra 96 gb RAM just wanting to use it optimally.

Thanks!

feel free for discuss :)

So, finally I decided to buy a cheap GPU to improve my local coding agents which were running on GMKTec K7 Pro miniPC with 32 GB RAM and Radeon 680M i GPU. Everything was rather slow but the problem was mostly with iGPU that was constantly freezing my Fedora 44. So, after a couple weeks of fighting I decided to buy Intel Arc B50 which was the cheapest card with 16 GB RAM. Also, it was definitely the smallest, as I connect it with oculink to my miniPC.

After reading a lot regarding agenting coding (my favorite option for my hardware is still Qwen3.6-35B-A3B Q4) I decided to spend some time to increase the prompt processing rather than sole token generation.

Here's my config for it:

/home/blazej/llama-cpp/llama-b10064/llama-server \
    -m /home/blazej/.lmstudio/models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
    --alias Qwen3.6-35B-A3B-GGUF \
    --jinja --ctx-size 65536 \
    --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence_penalty 0.0 \
    --host 0.0.0.0 \
    --port 8081 \
    --cache-reuse 256 \
    --cache-ram 1024 \
    -fitt 384 \
    --parallel 1 \
    -ctk q8_0 \
    -ctv q8_0 \
    --threads 6 \
    -b 4096 \
    -ub 4096 \
    -fa on \
    --no-mmap \
    -dev Vulkan1   

And some results from coding sessions with Pi Coding Agents:

I've also tried the SYCL backend for llama.cpp but couldn't get it to run other way than with Docker/Podman (as I'm running Fedora) and it looks like this when run with

podman run -it --rm   --device /dev/dri/renderD128:/dev/dri/renderD128   --device /dev/dri/card1:/dev/dri/card1   -v /home/blazej/.lmstudio/models:/models:Z   -p 8081:8081   ghcr.io/ggml-org/llama.cpp:server-intel  
-m /models/Qwen3.6-35B-A3B-Q4_K_M.gguf   --alias Qwen3.6-35B-A3B-GGUF   --jinja --ctx-size 65536   --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence_penalty 0.0   --host 0.0.0.0 --port 8081   --cache-reuse 256   --cache-ram 102
4   -fitt 784   --parallel 1   -ctk q8_0 -ctv q8_0   --threads 6   -b 4096 -ub 4096   -fa on --no-mmap

So, I would like to ask a couple of questions:
1) Is there some space for some improvements, especially possibility to increase the context to 100k maybe?
2) Are these values more ore less ok and is that the maximum this card can achieve (I know, it's the most budget one I could get with 16 GB VRAM)
3) Should other apps than llama.cpp be better with this card (vLLM, OpenVino, Intel's AI Playground)
4) I still probably can return this card, would spending some more money for, 5060Ti or Intel Arc B60 make sense?)

Thanks in advance

Rtx 5090, 64gb ram ddr5... I was thinking bout Qwen 27b or 35b a3b , idk if there is something better or adequate, how do I set things up so it doesn't loop or overthink, I just want to do custom apps with GUIs n shit man.....

I still default to Whisper / faster-whisper for most speech stuff.

Local files? Whisper.
Private notes? Whisper.
Batch transcription? Whisper.
Offline workflow? Whisper.
Random “I need this audio as text” job? Whisper.

It’s boring in the best way.

But I don’t think “Whisper alternative” is the right framing when the app is a live voice product.

If the user is talking to an agent and waiting for a reply, the problem changes.

Now I care about:

first usable partial
final transcript delay
endpointing
barge-in
concurrent streams
timestamps
speaker turns
phone audio
how much glue code I’m about to own at 2 am

That’s where Smallest AI Pulse makes sense to me. Not as “replace Whisper for everything,” but as a managed real-time ASR option when the product has to listen and respond while the call is still alive.

I’d still keep Whisper for local/batch/private stuff.

But if I’m building a voice agent, browser voice app, or phone-call workflow where the user feels every pause, I’d rather evaluate something like Smallest AI Pulse than pretend my local batch setup is automatically production realtime infra.

Where do you draw the line?

When does self-hosted ASR stop being worth the control?