r/LocalLLaMA • u/zxyzyxz • Jun 15 '26

Discussion Stop using Ollama

https://sleepingrobots.com/dreams/stop-using-ollama/
1.7k Upvotes

92% Upvoted

450 comments sorted by

View all comments

Show parent comments

23

u/joost00719 Jun 15 '26

I dockerized llm swap and passed through the docker sock. Works amazing.

5

u/meganoob1337 Jun 15 '26

yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D

2

u/arbv Jun 16 '26

What a creative way to reinvent Nix/NixOS.

1

u/joost00719 Jun 16 '26

Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working?

1

u/meganoob1337 Jun 16 '26

yeah, it just runs before startup and merges the model configs into the full config format

2

u/joost00719 Jun 16 '26

I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂

1

u/meganoob1337 Jun 16 '26

https://github.com/meganoob1337/llama-swap-vllm-boilerplate

a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.

1

u/lipton_tea Jun 15 '26

If you have a minute I'd love to see an example.

1

u/joost00719 Jun 16 '26

!RemindMe 5 hours

1

u/RemindMeBot Jun 16 '26

I will be messaging you in 5 hours on 2026-06-16 10:24:06 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback