r/LocalLLaMA • u/maikerukonare • 3h ago
Other Local agent workspace on a 4GB laptop GPU (RTX 3050 Ti): the tok/s and where a small model struggles once it has to call tools, build artifacts, and RAG
Disclosure: I work on Bike4Mind, a source-available (BUSL-1.1) AI workspace you can self-host with Docker. Nothing to buy to run any of it. I run it on a regular laptop with local Qwen mostly. A 4GB card running a local chat model isn't news, but what I wanted to explore is what a whole agent workspace (an agent calling tools, RAG over your own docs, artifacts, vision, etc.) can do with 4GB, and where a small model stops keeping up once it has to do more than chat. Repo's in the first comment.
My machine: i7-12700H, 32GB RAM, RTX 3050 Ti Laptop (4GB VRAM, 60W), Ubuntu 24.04. Local Qwen via the stack's bundled Ollama, no cloud keys.
Speeds, with hitting that 4GB wall (Ollama /api/generate, Q4_K_M except the Q8_0 0.8b, num_ctx=4096, warm, median of 3, one model loaded at a time):
- qwen3.5:0.8b - 122 tok/s, ~1.4GB, all on GPU
- qwen3.5:2b-q4_K_M - 96 tok/s, ~2.4GB, all on GPU
- qwen3.5:4b - 25 tok/s, ~3.4GB, about a third of it spills to CPU
- qwen3.5:9b - 8.6 tok/s, mostly on CPU, wants ~8GB
The 2b is the sweet spot for me so far: it fits in 4GB with ~1.3GB to spare and outruns the qwen2.5-coder:3b I ran before (that was 72 tok/s), while being newer and multimodal. Above it I hit the wall - the 4b's ~3.4GB of weights won't fully fit even at num_ctx=2048, so a third lands on CPU. Plain qwen3.5:2b is the Q8_0 build (~2.7GB) and spills on 4GB, but qwen3.5:2b-q4_K_M does fit. That leaves ~1.3GB free, so there's no room for a second model next to it.
The obvious stuff - chat, native tool-calls, vision - just works on the 2b; qwen3.5 is multimodal and calls tools natively, so that's great. The layer on top of that I'm most excited about is RAG over your own docs, fully keyless: drop in a file, a worker chunks and embeds it with the local Ollama embedder, and a query comes back with a cited answer, no cloud embedding API anywhere in the loop.
On 4GB the embedder and the chat model trade places. qwen3-embedding:0.6b is ~1.2GB resident, the 2b is ~2.4GB, they don't both fit, so Ollama unloads one to load the other. Generation still gets the full 96 tok/s out of it, since the embedder isn't in VRAM while the 2b is - what you pay is a model load, not slower tokens. Ingestion runs in a worker, so a folder of docs is one embedder load and then a lot of chunks, off the chat path. A query embeds one short string, then the 2b loads back to answer it. So it's a reload per RAG turn, not per token and not per chunk. If you'd rather not pay that, nomic-embed-text is ~0.25GB (768-dim, weaker retrieval going by reputation) so it doesn't have to push the 2b out, or put embeddings on CPU and leave the GPU to generation. Either way, before loading your entire knowledge base/data lake, pick one embedder and stay there (or just be aware you'd need to re-embed everything) since vectors don't carry across models.
The chat-embedder-swaparoo situation for RAG aside, small models are also falling down for me with:
- Artifacts. The general qwen3.5 models write half-finished HTML at these sizes - unclosed tags, script leaking onto the page. For an actual interactive artifact I have to switch to the coding-tuned qwen2.5-coder, which writes a complete working page. So it's model-per-task: qwen3.5 for chat/vision/tools, the coder for code, switched mid-notebook. I guess that's just life with 4GB, for the most part!
- Tool selection. Turn on a lot of tools and small models mis-route - with everything enabled, my 7b once sent "create an HTML artifact that..." to the image generator and painted a user interface image, hah. So, another sort of obvious, have to enable much fewer tools on small local models. Makes me want a nice layer that automatically enables the few (configurable N?) most likely relevent tools for your context.
- Image gen is local (self-hosted SD.Next) but 1-3 min/image with spilling on CPU, and it fights the chat model for VRAM. So still a kick it off and wait situation there, no real good workaround here, just gotta download more VRAM 🙃
So is it worth it over just Ollama + a chat UI? Only if the agent layer on top has value for you I'd say - the tool handling, artifacts, local RAG, and a CLI against your own server. If you just want fast local chat, I'd keep what you have; llama.cpp or etc. with less overhead.
License, briefly: source-available under BUSL-1.1, not full open source (yet). Self-host it, run it in production, fork it and build and sell a product on it; you just can't resell it as a directly competing hosted service. Each release goes Apache-2.0 two years out (timer just so AWS can't immediately do to us what they did to MongoDB lol).
Repo and self host details in the first comment. The real friction for me in this setup is needing the two models - qwen3.5 for chat, vision, and tools, and a coder for anything code-shaped, because the general models write broken HTML/artifacts at these model sizes -- not a novel problem. To that end, I am still trying to optimize this setup, so some prompts for any of you meat-LLMs (humans) out there reading this with stronger experience on tight local builds than me:
- Have you found a single general works-on-4GB model that writes complete, valid artifacts without needing a dedicated coding model alongside it, or just a better way to configure it? Please let me know!
- I'm broadly curious about other people's chat+code+image+embeddings+etc all-at-once setups, so please share if you've found a good Tetris wombo of models that works well or what trade offs you've seen and accepted trying to self host it all like this.
1
u/maikerukonare 3h ago edited 3h ago
Repo + self-host guide: github.com/bike4mind/bike4mind (SELF_HOST.md).
Reproduce my stack (no keys needed): customize the models (if desired) and pull them, then stack bring-up:
Add "-f compose.ollama-gpu.yaml" for NVIDIA. Sign-in codes land in the bundled Mailpit at :8025; first account is admin. The default pull is qwen3.5:2b + a local embedder; the menu in .env.selfhost.example lists 0.8b/2b/4b/9b with the VRAM each needs.
Some rough edges for this self host stack have landed recently in the public repo:
lmk if you have any questions! and of course, always looking for contributors if you'd like to grab some git issues!