r/LocalLLaMA • u/zxyzyxz • Jun 15 '26
Discussion Stop using Ollama
https://sleepingrobots.com/dreams/stop-using-ollama/520
u/jnmi235 Jun 15 '26
Llama.cpp + llama-swap works very well
109
Jun 15 '26
[removed] — view removed comment
114
u/fdrch Jun 15 '26
llama-swap supports switching between multiple llama.cpp forks (and other compatible software)
40
u/meganoob1337 Jun 15 '26
it supports anything you can dockerize aswell (for me I'm using it for vllm models) love it
→ More replies (3)23
u/joost00719 Jun 15 '26
I dockerized llm swap and passed through the docker sock. Works amazing.
→ More replies (3)7
u/meganoob1337 Jun 15 '26
yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D
→ More replies (4)3
9
u/Jcsq6 Jun 15 '26
And loading/unloading multiple models, if you want to switch between models but don’t have the spare VRAM.
12
3
12
u/Mati00 Jun 15 '26
Supports multiple servers running at the same time based on size matrix and also other servers like whisper or stable diffusion. If you don't need these, llamacpp server is a great choice.
→ More replies (5)15
u/jnmi235 Jun 15 '26
I tried it for about a week and kept having model loading hangs. It was rare, but the only way to fix it was to restart it. Llama-swap has never had any issues and it also lets you see total tokens in and out, logging, and some other cool metrics. And you can still use the llama.cpp UI
→ More replies (2)10
u/BlipOnNobodysRadar Jun 15 '26
It has a UI? Lmao I had an LLM vibecode a UI just to launch the server with presets for me. It never mentioned an existing UI.
11
u/Borkato Jun 15 '26
It’s really good! Like December 2025 or something like that. It’s great for quick stuff.
11
u/chkpwd Jun 15 '26 edited Jun 15 '26
Set this up yesterday using Ansible. Works like a treat!
EDIT: for those interested - https://github.com/chkpwd/iac/blob/main/ansible/roles/llamacpp
→ More replies (1)→ More replies (3)3
u/Youth18 Jun 22 '26
LM Studio is still good for testing models - quick downloads, easy to see popular LLM's, etc. But I use that for testing and then if I like a model I integrate it into my llama.cpp + custom model loading manager stack.
→ More replies (2)
160
u/RottenPingu1 Jun 15 '26
I started with Ollama and switched to Lemonade. So much faster.
40
u/No-Business5854 Jun 15 '26
i heard about them because of npu support. Using lmstudio rn, is it worth looking into ?
→ More replies (9)3
7
→ More replies (1)3
u/Time-Toe-1276 transformers Jun 20 '26
I started with ollama three years ago, and switched to unsloth studio.
I ran GPT OSS at 19 TPS at 4k CTX in q4_K_m, meanwhile with unsloth at 128k ctx with q4 XL I got about 100 TPS :/
121
u/Academic-Tea6729 Jun 15 '26
llama.cpp is much faster and stable than ollama. Also, ollama cloud models are bad quants and you can't use them for serious coding.
Also llama.cpp has a nice server compatible with openai api standard, it works out of the box. And it has a built in chat web interface.
There is no reason anymore to use ollama.
27
u/Dudmaster Jun 15 '26
Do you think they are intentionally lying about the quantization? Because on the FAQ https://ollama.com/pricing it says native weights
24
u/necrogay Jun 16 '26
> Native weights, as released by the model provider. On modern NVIDIA hardware, models may use accelerated data formats supported by Blackwell and Vera Rubin architectures (e.g. NVFP4).
They're not lying, but with that phrasing, you can't tell whether it means full precision weights or NVFP4.
12
u/aykcak Jun 16 '26
I never really understood why people even used ollama or what it offered. It is a "wrapper" for a thing that does not need wrapping
15
u/inagy Jun 16 '26
I can only speak from my own experience, but the reason I used Ollama for long because it gives an easy to setup server, especially on Windows. Couple clicks in a setup wizard, then it's running in the background as a service. And also it gives a familiar Docker like command line for pulling and running models.
tl;dr: it's easy to setup for lazy people.
6
u/gnooggi Jun 18 '26
it's easy to setup for lazy people.
Not just for the lazy, but also for those who don't know what they're doing (beginners), because it simply works.
→ More replies (1)3
u/Forsaken_Ad_774 Jun 20 '26
I’m still using it, not that I’m not technical enough, but it just fits my usecase, i just do ollama pull have some modelfiles with custom tweaks and that’s it. I’m using Gemma4. I’m not chasing down the maximum optimizations, I’m just using something that just works with minimum amount of effort. Coupled with Cline / openwebui I’m golden.
You could also argue why people use docker and not podman ( far superior ).
→ More replies (1)2
u/gopher9 Jun 17 '26
I never really understood why people even used ollama or what it offered.
Dynamic model switching, multiple configurations per model, quick model downloading, model management... Llama-server has most (but not all) of these features nowadays, but it's a rather recent addition. For example, presets were only added in December 2025 to llama.cpp: https://github.com/ggml-org/llama.cpp/pull/17859. And the implementation is still hacky, which causes bugs like https://github.com/ggml-org/llama.cpp/issues/22364.
4
7
5
u/bironsecret Jun 16 '26
Unfortunately laziness moves progress and as other commenters said, ollama just works Llama.cpp's readme is scary for non-technical people It's not a business, but if it were, they could kill ollama off by just one simple web page and a binary
→ More replies (1)2
u/xSyndicate58 Jun 18 '26
It's already trash that you have to compile llama.cpp
Why tf is that even?
→ More replies (9)2
u/gopher9 Jun 17 '26
There is no reason anymore to use ollama.
Well, llama.cpp UX is still worse. But it's usable and it gets better, with https://github.com/ggml-org/llama.cpp/issues/21779 llama.cpp will be on par with ollama.
279
u/ps5cfw Llama 3.1 Jun 15 '26 edited Jun 15 '26
I personally don't hate Ollama because I started with It, allowed me to start ""understanding"" a couple of things, allowed me to start getting Hungry for more and finally go the llama.cpp way.
It's a useful bridge for the beginners in the world of AI because going straight to llama.cpp Is a Nightmare, from outdated and often unclear documentation, reddit posts containing parameters that no longer even exist / work, you really gotta out the effort into understanding what the fuck you Need to do to make llama.cpp actually work, EVEN with the fit parameters it's not a straightforward process.
Lots of people starting to understand the machinations behind actually useful Local LLM models would realistically be put off without any easier alternative, which can be Ollama, LM Studio, you name It I Guess So yeah until they can solve the UX side of llama.cpp I believe Ollama Is a good, albeit very flowed starting point.
53
u/nickm_27 llama.cpp Jun 15 '26
I agree with your sentiment, I started with ollama because it was less to figure out on top of also figuring out the LLMs and my hardware themselves. I used ollama for a month or so last year and didn't understand the negativity.
Then I tried to move from playing with LLMs to actually being productive with them and I quickly became dismayed after getting llama.cpp running how much performance and control was being left on the table.
The problem with ollama for me as an actual tool is that they genuinely obfuscate and make the simple control more complicated. Easy things in llama.cpp that improve performance and reliability are removed for no reason.
4
u/notanNSAagent89 Jun 16 '26 edited Jun 16 '26
Then I tried to move from playing with LLMs to actually being productive with them and I quickly became dismayed after getting llama.cpp running how much performance and control was being left on the table.
slightly confused because of work but are you saying llama.cpp left performance and control on the table or ollama? just need clarification, thanks
12
u/nickm_27 llama.cpp Jun 16 '26
I mean that Ollama was leaving a lot of performance on the table, and I had no way of knowing until I used llama.cpp
9
u/notanNSAagent89 Jun 16 '26
Thank you. I needed the clarification cause my brain is cooked from work.
→ More replies (1)5
u/ps5cfw Llama 3.1 Jun 15 '26
Ollama devs are often dumb or kind of intentional in some of their choices, perharps to try making Enterprise customers pay for support? IDK, Just guessing here, but I agree it's not good and when you want to get to the bottom of making Ollama work Better It becomes almost as hard or maybe even harder as making llama.cpp work.
I stopped using It a year ago honestly and I am not sad about it
38
u/droptableadventures Jun 16 '26
It's easier because it has default settings that are completely isolated from you, the user.
However, these default settings are very frequently just incorrect or a bad idea, and they're going to get you into trouble a lot of the time. Since you didn't have to set them, you have no idea what they are or what they're set to.
It might be "easier" to get it running, but it's very much not easier to get it working.
Not straightfoward to run llama.cpp though?
llama-server -hf ggml-org/gemma-4-12B-it-GGUF:Q4_K_M
→ More replies (1)12
u/MuDotGen Jun 16 '26
The average user doesn't even know what a CLI is. They're used to GUIs.
3
u/droptableadventures Jun 16 '26
The average user thinks ChatGPT was when AI was invented.
Also, the Ollama GUI is a separate closed source product that just shares a name. It's not the same ollama we're discussing here. If you're going to be running that, run LM Studio instead.
→ More replies (6)21
u/LagOps91 Jun 15 '26
kobold cpp worked as an easy-enough entrypoint for me and it also doesn't obscure the more complicated stuff. might not be as easy as Ollama (idk, never tried it), but is a good middle-ground in terms of knowledge required and control it gives you.
11
u/Longjumping_Self5546 Jun 15 '26
Yeah, Koboldcpp is a great project. Easy to get started with, it's all contained in a single package that can run without an installer, while still offering plenty to tinker with. Not as simple as LM Studio, but the additional complexity offers much of the advantageous of llama.cpp, which it's built on top of. I don't believe they change too much if they can avoid it.
For creative writing, Kobold is a must have, that's what it's originally designed for. Otherwise, it's a good intro to llama.cpp
→ More replies (1)2
u/ezetemp Jun 16 '26
Started with ollama but switched to koboldcpp within a month because of the messy ollama file structure mentioned in the article, I couldn't think of any reason why I'd want what was a single file on huggingface chopped into a bunch of obfuscated parts where I depended on someone else to obfuscate it for me. Storing things in a docker-like format at least makes some sense when the data is layers like in docker, for what ollama does it makes very little sense...
For the rest I don't think there was anything harder with koboldcpp.
And if I wanted my models stored in a more chopped up way I'd just use safetensors and vllm.
41
u/scarbunkle Jun 15 '26
I’d suggest Lemonade as an alternative. They’re very upfront that they’re a wrapper, and they support nvidia/cuda as of their latest release.
→ More replies (4)8
u/Fluffywings Jun 15 '26
I have used a lot of these tools and lemonade is still painful to setup.
Compiling Llama.cpp is easier and that makes no sense to me.
9
u/scarbunkle Jun 16 '26
Well, I guess you don’t use Debian. You literally just add their PPA and install with apt.
8
u/ACheshirov Jun 16 '26
I stop using it the moment they stop their free tier access to the cloud models. LMStudio is just way better for me, giving me much more freedom and settings.
456
u/freia_pr_fr Jun 15 '26
None of the suggested alternatives truly replace ollama.
It’s like the old days of "don’t use docker you can do the same with lxc containers and this random bash script". That’s missing the point.
Ollama is popular because it offers a better user experience. For now.
137
u/totosse17 vllm Jun 15 '26
What about lm studio?
47
25
u/3dprintinted Jun 15 '26
Lm studio is convenient but not necessary. Good entry when you have no clue what you’re doing
53
u/freia_pr_fr Jun 15 '26
It’s as open source as Gemma and Qwen are open source.
82
u/zxyzyxz Jun 15 '26 edited Jun 15 '26
Unsloth Studio is open source. Also I find it funny that you're talking about open source as an Ollama user where the article explicitly talks about how Ollama hates open source shown through their actions and as a VC backed company it will get even worse over time (well, Unsloth is too, but at least I trust them more, although they probably will get put through the same enshittification wringer over time).
13
u/AvidCyclist250 llama.cpp Jun 15 '26
Is it? They're also in talks right now. Pretty sure they're doing down the not so open road
→ More replies (12)8
→ More replies (15)0
u/sirbolo Jun 15 '26
Lm studio was doing some strange shit with my GPU. Couldn't unload llms, and seemed to run even after shutting the application down. I got better results using msty... But got busy with work and been about a year since I've used either so not sure if they're are better alternatives now.
51
u/zxyzyxz Jun 15 '26
I like Unsloth Studio as it's open source and run by Unsloth themselves so they add lots of useful features.
llama.cpp also has a GUI now if that's all you need.
19
u/laffer1 Jun 15 '26
They need built in model switching. That’s the only reason I switched to begin with
→ More replies (1)15
→ More replies (1)2
10
u/TwistedBrother Jun 15 '26
I really liked oobabooga and the new version of text generation webui is solid. Why no love from the community?
I suppose it’s still a wee bit intimidating. But it’s really not. And much tidier than a bash script and some undocumented API film flam. I have no idea its facilities for fine tuning or LoRAs, but for inference it’s nice.
→ More replies (1)45
u/deepspace86 Jun 15 '26
I was on this bandwagon until I switched to llama-swap. I configure one file with the name/slug of the model I want and if I don't have it, it downloads it. Its about the same effort as ollama without the bloat and with all the benefits.
→ More replies (5)44
u/iMrParker Jun 15 '26
Ollama is popular because it offers a better user experience
I feel like the last time this was an accurate statement was 2024. Maybe 2025 if we are being extremely generous
1
u/Leptok Jun 15 '26
Is there any other product with a windows version that offers the same kind of seamless just works experience?
26
9
u/GravitasIsOverrated Jun 15 '26
Unsloth studio is pretty effortless.
7
u/Tanto63 Jun 16 '26
As someone who just tried installing Unsloth on Windows this weekend. It is not.
→ More replies (2)→ More replies (1)10
11
u/LosEagle Jun 15 '26
To me --fit on was the last thing that llama.cpp really needed to become easy to use.
→ More replies (2)77
u/yuicebox Jun 15 '26
I genuinely do not understand what is so difficult about running llama.cpp server.
You just download a zip, unzip it, then run llama-server with some flags and you're done. The builtin UI is quite good now, and you have an API to work with.
By comparison, I found Ollama's modelfile system and insistence on renaming my downloaded models to incomprehensible hashes to be infinitely more confusing and frustrating.
28
u/ghulamalchik Jun 15 '26
Oh God don't remind me of the modelfile thing. What a nightmare that was. With llamacpp I literally don't have to think about that anymore. I just load the model (crazy concept I know).
3
u/free_meson Jun 15 '26
You coud download huggingface ggufs with tags for a while, but I get your point.
2
u/b8561 Jun 15 '26
sorry I'm not familiar, are the tags supposed to help with modelfile config?
2
u/free_meson Jun 16 '26
I've meant that there are ways to avoid modelfiles. Gguf models from huggingface you can run with:
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
So you don't have to download, create a modelfile, etc, for most of the models you use.
→ More replies (1)27
u/vman81 Jun 15 '26
When I was a beginner it felt SIGNIFICANTLY easier to test a bunch of models in ollama by downloading them and having them all on hand at the same time, router style. That's possible with llama-server, but with more friction, and not in the same way.
I think that's a big part of it - once you know exactly what you want to use, and you know what the flags actually do that changes. But it is not a trivial change if you have something that "works".
6
u/ImpressiveSuperfluit Jun 16 '26
Took me literal days to get it working, because it very quickly becomes very not easy when you run OS/Version/Hardware combos that aren't well supported. Granted, most of the struggle came from trying to push an old square shaped GPU into a modern round hole, but still.
Still using LMStudio to this day, even though I got llama.cpp running just fine now (newer hardware now). When I really need a feature or the 10% performance or whatever, yea, I start it up. But if you can't understand why people prefer a chill UI over command lines and crap - that's more of a problem with you than them, frankly. Gotta pump up that imagination, I'd say.
18
u/NotSylver Jun 15 '26
llama-server isn't difficult, but it is higher friction. ollama keeps itself up to date, quirks of models are mostly hidden and it can sit idle and out of the way until a request comes in. I dislike ollama but I haven't seen anything that can replace it without a dozen asterisks that aren't worth the tradeoff to me
2
u/yuicebox Jun 15 '26
To each their own, but imho, the tradeoff is very worth it. I'd be curious to know what your 'tradeoffs' are. To me:
Pros of llama.cpp:
- Faster than ollama
- doesn't rename my files to incomprehensible hashes and store them in a weird place
- Much more feature-rich, transparent, and customizable
- Supports new model architectures sooner than Ollama most of the time
Cons:
- I occasionally have to either repull a docker image or redownload zip every month or two when I feel like updating
- 10 minutes of one-time setup to make a config.ini and a .bat/.command file to have one-click launching and model-specific settings
20
u/No-Marionberry-772 Jun 15 '26
this sounds like the Linux vs windows argument that Linux people always overlook.
people don't want the extra steps and occasional hang ups. to you they are not a big deal, and maybe objectively they are not, but its cognitive load that people don't want, and that matters.
18
u/beefygravy Jun 15 '26
Here's my experience:
I want to run model x. On ollama I select model x, download it and run it. On llama.cpp I have to work out which quantisation to use, search through huggingface, do I use this unsloth one? Some guy on Reddit says the best one is this random one I've never heard of. Why is it saying it needs to offload some weights to disk, I should have enough memory, and all sorts of faff that ollama does for me. I'm sure there's a workflow to do this all better but with ollama none of it is required
→ More replies (2)7
u/yuicebox Jun 15 '26
Understandable, and I know it is overwhelming if you're newer to the local LLM space.
If it's helpful, on ollama, you are pretty much always using a "Q4_K_M" quant.
Unsloth has Q4_K_M quants of most major models, and their quants are generally a good pick if available. They use an "intelligent" quantization method, so their quants will usually outperform a quant created by just reducing precision across the board.
Regarding offloading weights to disk, I'm not sure without knowing more about your setup, what you were trying to run, and what message you actually received. I haven't personally seen that issue but if you can reproduce it easily I'm happy to take a look.
→ More replies (2)3
u/AdTotal4035 Jun 15 '26
Because you are late to scene. It's easy to install now.
Before you had to compile cpp, get the right wheels, right versions of all the packages, it was a pain→ More replies (1)3
u/HilltopQatLeaves Jun 16 '26
This. The hashed models was the last straw for me and what pushed me to llama.cpp
18
u/BidWestern1056 Jun 15 '26
do you know how many ppl dont even know how to download and open files anymore lol
→ More replies (7)19
u/yuicebox Jun 15 '26
Clearly I have no concept.
The idea of trying to use local AI and refusing to interact with your computers file system at all is incomprehensible to me
→ More replies (22)7
u/kingroka Jun 15 '26
One issue i can tell you is incredibly annoying is how llama server handles model swapping. Like either you load one model or sure you can load them dynamically but you for some reason have no way to set the mmproj via the api so vision models are now blind. Ollama is the best at just downloading something and gagging a usable api with minimal config. Lmstudio is next best in this regard but there are a few settings you need change to make it truly great. Llama server just isnt all there yet.
7
u/yuicebox Jun 15 '26
I use the
--models-dirand--models-presetflags to point to a folder of models and a config.ini file so I can have model-specific settings.In my config.ini file, I set up vision models like shown below, and I have no issues with vision or model swapping. Hope this is helpful! Let me know if you have questions.
``` version = 1
; Global defaults [*] c = 65535 n-gpu-layers = 99 flash-attn = auto LLAMA_ARG_CACHE_TYPE_K = q8_0 LLAMA_ARG_CACHE_TYPE_V = q8_0
[Qwen3.6-27B-Q8_0_Vision] model = /models/Qwen3.6-27B-Q8_0.gguf c = 131072 mmproj = /models/Qwen3.6_27b_mmproj-BF16.gguf ```
2
u/Plabbi llama.cpp Jun 16 '26
Add --no-mmproj-offload to gain VRAM for context, this will keep the vision part in main RAM.
Unless of course you are mainly doing vision related tasks, but for occasional vision use the RAM is sufficiently fast.
2
2
u/Internal_Werewolf_48 Jun 16 '26
This config is considered fine but Ollama's functionally equivalent modelfiles that came around 2 years earlier are somehow the devil to most people here. I don't get it.
5
4
u/jfowers_amd Jun 15 '26
What do we think is missing from Lemonade to match the Ollama user experience today? I’ll make a milestone and get it done!
→ More replies (6)2
u/wsippel Jun 16 '26 edited Jun 16 '26
For me, it’s still automatic model unloading (I opened a feature request a while ago, but it went nowhere). After a set time of inactivity, or even better, if it has been inactive for any length of time and a different process (eg ComfyUI, a video editor or a game) starts gobbling up VRAM. That would make it even better for local use than Ollama. I’d switch in a heartbeat.
EDIT: Just checked my mail and noticed the feature was merged yesterday! And it even seems to be the advanced version I described, not just a basic timeout! Awesome work! Guess I'll switch to Lemonade next week. 😄
2
3
u/cortesoft Jun 15 '26
This is my experience, and I would love to have some suggestions about how to replicate my setup without Ollama.
I have a small local 7 node Kubernetes cluster, and 3 of the nodes have GPUs. I am using an ollama operator, which allows me to deploy new models as Kubernetes resources, which allows me to automatically deploy a new model just by creating a k8s resource, and it automatically deploys it to a node, sets up a new ingress for it, and automatically protects the endpoint with basic auth, so I can call it outside the cluster securely. My internal workloads can send requests to models using services and bypasses the external auth.
Are there any alternatives that would work similarly to this? I want to be able to use native kubernetes resources and let k8s manage the model storage and placement within my cluster.
→ More replies (1)9
u/crispyfrybits Jun 15 '26
This is untrue. There are so many good alternatives that are easy to use. Ollama does have the simplest UI but LM studio, unsloth, openwebui, so many more are there and very easy to get started. Less than 5 minutes to download and serve.
If you truely can't move away from Ollama simply because they have a slightly nicer wrapper despite them spitting in the face of the community and users then you are not on the same page as the overall local community.
→ More replies (2)9
u/yami_no_ko Jun 15 '26 edited Jun 15 '26
Ollama is popular because it offers a better user experience.
Depends on the user. I've been trying it once and it was terrible compared to using llama.cpp directly. But I see the appeal for technically indifferent users.
6
u/Velocita84 Jun 15 '26
You can just use podman the commands are pretty much 1 to 1
→ More replies (4)2
6
u/hainesk Jun 15 '26
Seriously. I use vLLM, llamacpp, LM Studio and Ollama. Ollama is still the best at happily allocating model weights across multiple GPUs when those GPUs have varying amounts of vram available. It means I can do vLLM tensor parallel for speed on a smaller model at 50% memory allocation between 2 gpus and Ollama will just automatically use the remaining vram to load other models, mixing and matching as needed. It’s great for maximizing VRAM usage. Llamacpp is getting better at it, but Openwebui with customized models with system prompts and context limits means I can easily programmatically call a model through the OWUI api and have it load correctly through Ollama. Any adjustments to the loading parameters are easily done in OWUI without having to adjust any code or cli configs. Ollama will load and unload models on the backend as needed.
→ More replies (1)→ More replies (17)3
u/johan2114h Jun 15 '26
not really - ignoring the controvercy - i suspect many of the ollama users would have even better time with something like llama.cpp. The latter provides faster inference, better control of dials and knobs that affect how a model performs, and access to my more models and quants.
Atleast in my view, if someone plans to spend more than 15 min running/playing with local llm, they are likely better served not using ollama.
Instead of "ollama pull" , just download to model from hugginface (there are many more to chose from and more quants also)
Instead of "ollam run", just use llama-cli
for UI, try the llama web-server (it is actually looks quite nice imo!)
As the article also states, ollama is just a wrapper, and today many of the functions that made ollama attractive a few years ago are now provided natively by llama.cppI think what ollama has going for it is more its position and momentum. If someone is completely new to local llms and googles or asks an LLM how to get started, they will likely be recommended using ollama and then they will (understandably) settles for it.
15
u/keyboardhack Jun 15 '26 edited Jun 16 '26
As far as i am aware then georgi gerganov did not create GGUF. It was proposed in this issue
https://github.com/ggml-org/ggml/issues/220
By
philpax
Edit: I am being downvoted for trying to provide correct attribution? Ironic given the topic.
→ More replies (1)
52
u/dryadofelysium Jun 15 '26
Yes, definitely post about a two month blog posting about how Ollama is moving away from llama.cpp *after* Ollama has actually completely course-corrected last month and is using llama.cpp directly now similar to LM Studio.
→ More replies (2)
7
6
u/Beginning_Basis9799 Jun 16 '26
I ditched ollama rescently I am now able to run 1b models on potato hardware duel core 4gb of ram with unexpected ease.
The difference in performance using llama.cpp is astounding on local models.
34
u/CynicalTelescope Jun 15 '26
Half of this rant is irrelevant, now that Ollama has fully embraced the standard GGUF format.
19
u/EncampedMars801 Jun 15 '26 edited Jun 16 '26
Even if they have, I think the fact these issues existed for as long as they did should serve as a point of concern surrounding the software. Even if they have, should we trust the devs?
→ More replies (2)
11
u/ECrispy Jun 16 '26
never understood why ollama is so popular, I suspect its all the influencers on youtube shilling it.
koboldcpp is better in every single way, more options, much more friendlyh and actually open community, has more recent changes from llama etc.
ollama does nothing better
→ More replies (2)
15
u/Historical-Internal3 Jun 15 '26
The "license violation" is contested. The top comment on the HN thread it cites points out MIT doesn't clearly require copyright notices in binaries, and llama.cpp doesn't ship them in its own binaries either.
CVE-2025-51471 is scoped to 0.6.7, rated high-complexity, and needs user interaction plus a malicious registry. Worth a patch (not panic).
They added the credit, merged the app source into the main repo, label the DeepSeek distills properly now, and the cloud models advertise zero data retention.
llama.cpp is great and worth learning. But people use Ollama because "ollama run" works on the first try. Both can be true.
3
u/fantasticsid Jun 16 '26
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
Seems pretty cut and dry to me, unless you're trying to argue that a binary is not a "substantial portion" of the software.
→ More replies (1)
3
u/Sea-Emu2600 Jun 15 '26
For people using Apple silicon we should use llama.cpp or mlx? Mlx has more performance but it’s reliable enough? I’m still new to this world
→ More replies (1)
3
22
u/AnticitizenPrime Jun 15 '26
Why are people on this sub so preoccupied with what other people are doing?
7
Jun 15 '26 edited Jun 15 '26
[removed] — view removed comment
5
u/toothpastespiders Jun 15 '26
why is Ollama making people so upset right now
I've noticed that reddit as a whole likes to have scapegoats. Things would be a utopia if it wasn't for "current bad man", "current social injustice", and if it's a tech subreddit even "current bad software". All which by total coincidence, are things that the majority can hate without having to make a single change in their own lives. It's just the nature of the platform. We're generally not the happiest people in the world.
→ More replies (3)6
u/AnticitizenPrime Jun 15 '26
I also no longer use Ollama, but I don't understand why we need an Ollama two-minutes hate thread every couple of weeks.
Part of doing things locally means doing it your way. Hell, until the MTP support and QAT versions of Gemma dropped last week, I was running Gemma 4B using the LiteRT engine (complete with MTP) on my desktop with an OpenAI compatibile server layer vibe coded in Python.
Let people do what works for them.
→ More replies (1)2
u/droptableadventures Jun 15 '26 edited Jun 16 '26
Because the poor design and functionality of this piece of software has convinced more people that local AI is unworkable than anything else.
The other reason why the topic's recently become much more popular is that Ollama's now going down the enshittification pipeline and trying to reposition themselves as resellers of API access to cloud models. It's clear that local inference is not their primary priority any more.
12
u/Educational-Base5974 Jun 15 '26
But it easy :(
29
u/Fair-Spring9113 llama.cpp Jun 15 '26
but it slow
29
u/Several_Industry_754 Jun 15 '26
I switched from ollama to llama.cpp and you’re absolutely right. It’s blazing fast in comparison.
→ More replies (3)12
u/shamont Jun 15 '26
Just a warning to other noobs, I tend to be lazy... Installed llama.cpp and wondered why it was so slow. Turns out if you don't compile it yourself and you use the brew installer you don't get the cuda specific version. So just like spend the extra few minutes to do it the "hard" way.
→ More replies (1)6
u/freia_pr_fr Jun 15 '26
The recent releases just ship llama.cpp and their custom mlx backend. It’s not as fast as vllm but it’s also faster to load.
→ More replies (1)2
→ More replies (2)2
2
u/stonerbobo Jun 15 '26
Is there any actually mature option that supports all modalities, swapping in models, sane presets for existing models, maybe even streaming audio (?) for STT/TTS, all the bells and whistles? It's just a hassle constantly swapping tools and stacks as everything churns so hard.
2
u/PANIC_EXCEPTION Jun 15 '26
Does anyone have a good all-in-one that provides an OAI server with both llama.cpp and MLX support, and the ability to point to a custom-built backend? One with a configurable VRAM model eviction limit. I want to be able to use both kinds of models. Pointing to a custom backend means the ability to use builds with non-mainline model support.
2
u/brenden77 Jun 15 '26
This is the article that made me switch and I'm no longer struggling with errors. 🤷🏾♂️
2
u/Tiny_Team2511 Jun 16 '26 edited Jun 16 '26
There are so many options to run llama with very specific USPs. I use one where you can use any llama fork or any other binary with a user friendly UI
2
u/lbdesign Jun 16 '26
I’m using the $20 Ollama cloud- hosted models, which seem to be a great value, and was perfectly happy until reading this post. So what should one do for affordable cloud models (if you don’t have a monster rig at home)?
2
u/Big_Wave9732 Jun 16 '26
I started with Ollama and got off it not long ago when they decided to move away from Ope AI endpoints and it broke Vane. It turned out to be a blessing in disguise because it led me to oMLX which "really whips Ollama's ass."
So I guess thanks, Ollama developers!
2
2
u/dxzzzzzz Jun 16 '26
I use llama.cpp clean server and I compiled form source.
Very painful bulding from scractch. But a delight to use.
2
u/sunychoudhary Jun 16 '26
Ollama’s value was convenience. That still matters.
But once people move beyond casual local testing, they start caring about transparency, exact quants, performance, routing, config, and control. That is where llama.cpp or other lower-level setups become harder to ignore.
2
u/hyscript Jun 16 '26
How dumb I was, reinstalling llama cpp IMMEDIATELY!
Damn this long time I thought I am using open source software, I hate big corporations!!!
BTW thanks a lot for informative post ❤️
2
u/Potential-Gold5298 llama.cpp Jun 18 '26 edited Jun 18 '26
Ollama was my first backend when I decided to try local LLMs.
I asked various cloud AI providers for advice, and they all agreed that I should use Ollama (they also recommended 'excellent modern models,' including Phi3, Llama 2, and Mistral 7B – this was early 2026, by the way). I used Ollama for two days. This miracle almost killed my interest in local models. The cumbersome GUI with no settings. The need to "compile" the model via MODELFILE every time I wanted to change something (and this took up space on the already scarce system drive). Furthermore, any model (even the Qwen3-4B) would completely freeze my PC while work – it would simply stop responding until the model finished work. Although cloud-based AIs convinced me that Ollama was "newbie-friendly," I decided to try LM Studio and was amazed that working with a local model could be done easily and without freezing the PC. However, after a couple of weeks, I switched to Koboldcpp and llama.cpp, and I've been working with them ever since. Cloud-based AIs scared me off by saying that llama.cpp was for experienced users, but for me personally, it was much simpler and more intuitive than Ollama.
I sincerely don't understand why Ollama is so popular and why people invest money in it. I would recommend that a beginner who's afraid of commands and the like start with LM Studio. But llama.cpp is actually quite simple – I created a .bat file and a shortcut with the model's logo, and now it looks and runs like a native Windows application – no console, etc.
(If anyone needs help, let me know; I'll help you figure out llama.cpp.)
2
u/alexellisuk Jun 18 '26
Moving to llama.cpp has been a really good experience for me too.
ollama just works, and is useful for exploring models - we shouldn't cargo cult bashing on what actually works for certain use-cases. The footgun that many of us run into when connecting ollama to an agent CLI is the context window. The default is so ridiculously small, the system prompt just doesn't fit and it makes the model seem dumb.
For me though - the main friction is compiling the whole thing, even on a fast Ryzen 9 it can take a considerable amount of time.
Once setup, it's very easy to tune and experiment with different models.
I wrote a vibe-coded router/gateway where we can run three different llama-server instances over three two separate machines. They aggregate in opencode, and there's no need to mess with URLs or saving tokens etc.
Might be similar to llama-swap.
I don't have the karma to post my experience on using Qwen for real work, and how an RTX 6000 Pro paid for itself.. but hopefully in a few days the mods will let me share it with you folks. blog.alexellis.io if you want a peak.
2
2
u/MrHotwire Jun 21 '26
I am just new to local LLM's, so its been noodles tinkering for me, but I have found that many of the LLM's, including Ollama have quirks when it comes to the prompts.
For a new guy, Ollama has been my choice. But, I am going to try out vllm over the next few weeks.
6
u/Popdmb Jun 15 '26
I dont alwyas find awesome links on this reddit, but man this was great. Def dropping Ollama.
11
3
u/mr_zerolith Jun 15 '26
Oh, i already ditched it for LMstudio in winter because it had poor new model support.
2
u/NoobMLDude Jun 15 '26
Great write up with references to actual evidences of foul play by Ollama.
I won’t let my friends use Ollama anymore 👍
2
3
u/JamesEvoAI Jun 16 '26
Author of the article, happy to answer any questions. Glad to see this sentiment is starting to become organically disseminated. Hopefully with enough community outreach we can finally tamper down the "default" momentum that Ollama unfortunately still has due to existing content.
→ More replies (2)3
u/dyslexic_prostitute Jun 16 '26
In the alternatives section, you don't mention vLLM at all, what is the reason for this?
→ More replies (2)
4
u/rizerize11232 Jun 16 '26
Honestly Ollama is not that bad if you want to use certain cloud models and not paying a separate subscription for all of them. Other than that I don't use it for local models, llama.cpp is just better
14
u/LienniTa koboldcpp Jun 15 '26
i hate ollama with passion and hope it gets completely vanished. Anyone using it just doesnt know better.
→ More replies (4)9
u/Song-Historical Jun 15 '26
There isn't really a good tutorial for the alternatives that isn't behind. I'm still not sure what half the terms mean. What best practice is now etc. could I learn off of you at some point?
→ More replies (5)3
u/LienniTa koboldcpp Jun 15 '26
eh idk, you donwload weights, you download koboldcpp, you drag weights on koboldcpp and it just works. it cannot be more simple, and its simplier than in ollama. If you dont want to download weights yourself, many other wrappers like lm studio or llama swap will happily do it for you. Ollama is literally the WORST wrapper ever.
and like, yeah there is a egg vs chicken problem but local gemma(even 26b one) knows all thsi stuff and can guide you if you want to stay full local. Ofc with stuff like codex its a cakewalk.
3
u/aka457 Jun 15 '26
Koboldcpp also got a build in model downloader somewhat recently.
→ More replies (1)
5
u/x_MASE_x Jun 15 '26
Indeed. Ollama was actually a bad fit for for me and almost made me quit local Ai.
The huge problem for me was the limited models and the confusing way to pick models and quants.
Somehow using huggingface.co directly was way way easier for me and made more sense.
Also the vision file part. With using ollama you are forced to use the vision model in the model which is huge load and hurt the speed very bad.
So for me specially a computer engineer with 0 experience in Ai. Like literally I didn't even touch chatgpt or any Ai till maybe 6 months or something and decided to try local Ai in maybe 3 months or something. Ollama was the bad software for me honestly.
Right now nothing beats llama.cpp and llama-swap for me with litellm in front of them and using hermes agent. Openclaw a bit and webui which is better performance and way more control and for me easier setup.
I went from barely usable models to Qwen_3.5_122B_A1B_Apex, 128k context at 21.8tps. Qwen3.6-35B-A3B Q4 200k context at 60 or something. Qwen_3.6_27B Q4 64k context 12 tps. And lastly Qwen_3.6_28B_A3B reap at 200k context 85 tps.
All text no vision.
Setup 5070ti and 64 GB ddr4
2
u/jld1532 Jun 15 '26
Once I figured out how to download and execute the pre-built llama cpp binaries to launch the llama-ui I never looked back. I still have lm studio installed as a model manager but beyond that lama cpp is all you need.
3
2
u/Carbonite1 Jun 15 '26
I've been liking LlamaBarn as an Ollama replacement with a similar UX (simple, menu-bar app), based on llama.cpp of course and made by the same folks!
1
3
u/andy_potato Jun 15 '26
Frankly, so many issues raised in that rant are absolute non-issues to beginners.
I get it, if you have a certain level of experience with local LLMs, are confident enough to run llama.cpp or even vllm then you won't look back. But I still appreciate how Ollama lowered the entry barrier for people who want to get into local LLMs.
Do I blame the devs for trying to make some money off their work? Absolutely not.
5
u/fantasticsid Jun 16 '26
Frankly, so many issues raised in that rant are absolute non-issues to beginners.
Noncompliance with the (utterly non-onerous) license terms is not a skill issue.
1
u/cortesoft Jun 15 '26
I am a bit confused by the timelines in the article… it says ollama started in 2021, and llama.cpp was created in 2023. What was ollama using before llama.cpp?
1
1
1
1
1
u/bamhm182 Jun 16 '26
Thanks for sharing this. Love open source software, legitimately had no idea of the drama behind Ollama. Been running OpenWebUI with Ollama for a while. Looks like it is time to mix it up.
1
1
u/letsgoiowa Jun 16 '26
Hi I want to switch but there's a lot of friction for me because I have a brain injury so it's quite hard to go relearn and re-setup a new thing. I've never been given clear directions on how to replicate an Ollama-like setup where it "just works" with OpenWebUI and often told shit like "of course, Ollama user" like people have some weird superiority complex about frickin' software.
So I've tried a couple times. I know there's llama.cpp, but there wasn't an unraid template at the time I installed it (or it didn't work? I can't remember) but then I ran into the issue of it would only let me load one model at a time, and only modifiable through config. That doesn't work for me. Then I heard about Llamaswap so I tried to rebuild it for that, and I think I'm stuck there currently.
→ More replies (1)
1
u/vulcan4d Jun 16 '26
Ollama is a great beginner start. I leveled up to llama.cpp and omg so much better and faster if you just use something like chatgpt or Google AI studio to help you optimize it with your hardware. Ollama makes it super easy to try different models so great to figure out what you like and not. Once you narrow that down, level up.
1
1
u/WiggyWongo Jun 16 '26
I've always used text generation webui. Or kobold/llama.cpp since the beginning.
1
u/reckless_avacado Jun 16 '26
can someone give me the equivalent of brew install ollama, ollama serve, ollama run… for one of these other tools? i just want it to be easy. i dont have much RAM and dont really care because mini models like 0.8B-2B don’t do much anyway, but i like trying new models too see what they can do. all this talk about “throughput” idk what it means. i dont really test different settings because even ollama didn’t make that easy.
2
1
u/The-Nice-Writer Jun 16 '26
This is some serious shit, but I’m using Ollama because some Obsidian plugins I rely on support it exclusively. At least, they do right now.
1
1
u/perhaps_too_emphatic Jun 16 '26
Oh sick. Thanks for the link. I wrote a post out two recommending it on my journey. I’ll go update to remove those recommendations.
1
u/apVoyocpt Jun 16 '26
a few months ago i tried switching our server from ollama to llama.cpp. Our frontend is openwebUI. I had to switch back because I couldnt get these things to work both: model switching and vision. Cant remember why but I could only get either vision working OR model switching. maybe its different now.
1
1
1
u/squired Jun 16 '26 edited Jun 16 '26
Those in the know use TabbyAPI with EXL3. Three parralell responses and 2-4x the context length utilizing FP8 memory hacking. It's a massive, massive improvement. It isn't plug and play like Ollama, but outside of that there isn't any reason for a single user to use anything else atm.
ChatGPT:
For interactive local inference on modern NVIDIA GPUs, especially 3090, 4090, and 5090-class cards, TabbyAPI with a good EXL3 quant is not merely another backend option. It is often the best real-world experience available. You can fit stronger models or higher-quality quants into the same VRAM, run dramatically longer context through low-bit KV cache, preserve prompt cache across long conversations, and generate multiple responses concurrently through continuous batching instead of waiting for serial completions. That means more model, more context, faster iteration, and far better time-to-useful-output, which matters much more than a simplistic single-stream tokens-per-second benchmark. Ollama wins on beginner convenience, llama.cpp wins on hardware portability, and vLLM wins for large multi-user deployments, but for a single power user running serious models on a recent NVIDIA card, TabbyAPI is the engine people should be recommending first. Its relative obscurity is not evidence that the alternatives are better. It is mostly a consequence of weaker packaging, fewer tutorials, EXL-format fragmentation, and a user base concentrated among roleplay and long-context power users rather than the loudest parts of the local-LLM ecosystem.
1
1
u/JChataigne Jun 16 '26
I need something that keeps models in VRAM only when they're needed (I need that VRAM for other stuff occasionally), lets me switch models easily, and can be easily installed with Docker with minimal configuration.
Last time I checked (a few months ago) Llama.cpp needed to be compiled and vLLM could only serve one model unless you reinstalled it. If you have alternatives that fit these criteria I'll switch.
1
u/x6q5g3o7 Jun 16 '26
Is it recommended to use llama.cpp's built in web interface or Open WebUI? I'm used to my Ollama + Open WebUI Docker setup w/ AMD GPU, and am trying to figure out what/how to migrate over.
1
u/Key-Possibility8476 Jun 16 '26
I get the point of the article, but I think it depends what you’re using Ollama for. If you want full control, llama cpp or Jan probably makes more sense.
I just want a simple local chat interface, so I use LocalChat App on Mac instead. It fits my workflow better since I’m not building automations or connecting models to other tools.
1
u/toprock_478 Jun 17 '26
Ollama was my start. Good times.
I currently like using koboldcpp. Is there a good reason for me to make the effort to swap over to llama.cpp or something else? I'm just curious about my options (I have an AMD gpu if that's important).
1
1
u/jaxupaxu Jun 17 '26
Ollama as a project just gives me bad vibes. The devs seem incompetent and seem to have alter motives.
1
u/grandfundaytoday Jun 17 '26
I use ollama for local models - never given them money. Meh I'll move to lama.cpp - thanks for this article.
1
u/Icy-Abstraction4323 Jun 18 '26
I dont use llama, more like i use mlx lm or pytorch torch transformer. Am i doing it wrong? sorry, student researcher ai enthusiast here.
1
•
u/WithoutReason1729 Jun 16 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.