Local LLM

I’ve been using Claude Code for a while for coding related tasks, but I get frustrated by running out of tokens. I have a GPU that should be sufficient for my needs, so I’ve also been tinkering with running a persistent local LLM for personal analysis and code generation.

I used to use Ollama quite a bit to load in various models, but my experience loading arbitrary 3rd party models into memory was frustrating. First, Ollama seems to dump the model out of memory wantonly. If I want to force a model to remain in memory and not dump out after some arbitrary time, it doesn’t work so well. Second, I can’t seem to figure out how to set up a continuously running service on my Windows machine that hosts the llama-server via NSSM.

You might say that the memory saving features that offload models when infrequently used and the GUI-first nature of the LLM service are features, not bugs, but it’s definitely frustrating that it seemst there are performance tradeoffs to using models via Ollama on top of the slightly inconvenient UX.

So, I’ve decided to follow a blog post by Quesma to use llama-server directly.

I had to download and build llama.cpp from source myself to include support for my system (using an AMD GPU via Vulkan on Windows), but I believe that there are pre-built binaries for people who prefer the road more traveled. It’s working, so when our AI overlords decide to start charging retail users at profitable price-per-token, I’ll be ready.