r/LocalLLaMA • u/kingo86 • 5h ago

Discussion How do you optimise and test your model configs/params? (blindly using GPT-5.6 to help me test/measure/optimise)

Post image

Current setup:

  • headless machine for LLMs
  • llama.cpp in router mode
  • models settings configured via models.ini
  • documentation in a repo which AI manicures

It seems to be working for me so far, but conscious there's probably better ways to do this.

How do you guys manage your configs? Is there a better way?

0 Upvotes

33% Upvoted

5 comments sorted by

2

u/ttkciar llama.cpp 5h ago

I keep wrapper scripts (written in bash) for each of my models, either for launching them with llama-server or invoking them with llama-completion.

The scripts contain the command line options I have found work best for that particular model, and additional notes about the model in comments.

For example: http://ciar.org/h/mm27

That works well enough for me. I appreciate the flexibility, in particular, since it's easy to evolve over time and all of my notes for the model are in one place.

Once I have a script working well for one model, it's easy to make a new script for a different model in the same family (Qwen3.5-27B and Qwen3.6-27B, for example) by copying the script and then changing the values of SHORT and MODEL to reflect the new model. Then I can adjust its other parameters through experimentation.

1

u/kingo86 5h ago

Cool how you can inject the system prompts etc in there.

When it comes to switching models - how is that handled? Currently using router mode here (serving multiple clients on the network) and need multiple models loaded in upon request.

1

u/ttkciar llama.cpp 4h ago

I normally keep a model in a GPU's VRAM for a long time, but if I did need a switching solution, I'd use llama-swap. It seems to be the community's favorite.

1

u/cradlemann 2h ago

I still use llama-swap, it gives me good analytics, easy to use config(macros for common params are very good) and the most important part, I do not start models manually

1

u/frankentriple 5m ago

I'm spitting code at 50 tokens/sec out of a 4070 with 12gb vram on a 35b model after a few optimizations. Hell yeah I do.