Should I Use Free LLMs?
Summary
Free language models that run from your home can handle routine work, but buying the hardware to run them on can be expensive. A Mac with enough memory to hold one of the better free models starts at around £10K, but even that free model is a year or two behind the best frontier ones. You could try running on cheaper machines, but you’ll probably wait a long time for the LLM to respond to your requests.
If you work alone and use AI every day, you may have wondered about the bloke with his home-grown AI, chugging away in his spare room. He’s on LinkedIn, he’s on YouTube, and he’s posting all the time about his success.
Spare room guy runs free models on hardware he has configured himself. He does not routinely rely on OpenAI, Google or Anthropic. He talks fervently about his army of AI agents the way people who bake talk about sourdough.
The problem is that every time you pay your monthly subscription to Google or Anthropic, you wonder if you’re missing something huge, and if matey over there knows something you don’t.
£18 per month
When I first started experimenting with Claude, I was on the Pro subscription for which I paid £18 per month. But as I got more and more into building things, I soon started to run out of tokens.
Then I moved to the Max 5x plan at £75 per month. That was good for a while, and then as my ambitions grew, I again started to regularly run out of tokens. This meant I often had to stop working until they replenished, which sometimes meant waiting several days.
Today I pay Anthropic about £150 per month for the Claude Max 20x plan, and I use it for most of my working day. I don’t ever run out of tokens.
But I have been asking myself if I too should be looking to run my own models in my home office to cut costs. I was made to think about it again by a recent LinkedIn discussion.
What the tech blokes get right
The guys do have a point. You do not need the best model for much of what we freelancers and solopreneurs do. Summaries, rewrites, first drafts, sorting, classifying … that is the easy stuff. A model that can run on a desktop can do all of that.
A model works by answering one question and then it stops. But placed inside a harness that gives it a memory, a list of instructions, a place to write data, and a second model to check its work, then a weaker model with a good harness may be able to beat a stronger model with a less capable harness. In other words, smaller models can be made to look better than they are.
There’s also the question of privacy and control. Banks, hospitals, government departments and other large organisations worry about their data leaving the building. They worry about what happens if a large US provider like OpenAI vanishes one day. They worry if access might be compromised because, say, the US President is feeling cranky and signs an executive order that screws overseas customers.
Downloading the software, running it in-house and owning the hardware feels like an answer to all of that.
Where they are wrong
Small models checking each other can’t conjure up knowledge or judgement that none of them had to begin with. Where the answer depends on knowing something, five copies of a small model will deliver five similar, poor quality answers. All the re-checking gets you is consistency.
Further, a smaller model may cause you to shrink your own ambition because it never pushes you to the point where you can expand it. Ninety percent of my work could probably be done by any model, but I pay £150 per month for the other ten percent that pushes me and the boundaries in my projects.
Local is probably slow
Maybe the guy in his spare room runs one job at a time, or maybe he sometimes drops to a smaller, less capable model to make the speed bearable.
A frontier model on subscription runs in a data centre serving thousands of people, so it is fast and almost always available. You can run as many jobs as your tokens allow. A single box at home can take two jobs at once, but each runs slower, because they share the same machine.
You can spend a few hundred pounds on second-hand kit, or wire together a stack of Raspberry Pis, to run a small model. People do this. But what you get is a small model running slowly, because a model can only go as fast as its memory can be read.
Cheap boards have slow memory and slow connections between them. Ten Raspberry Pis are ten slow computers taking turns.
To run one of the better free models at a speed you could put up with all day, you would be looking at a box costing £10,000 or more, and the model you are running would still be a year or two behind Anthropic’s best.
If all you wanted was a cheap model for a cheap job, you can buy that by the token, without buying any hardware outright, and you’d save a lot of money (and headache) by doing so.
Cheap cloud models such as OpenAI’s Luna cost very little. At Luna’s rates, ten million tokens a month comes to about £2, and it would take over 100 years at that rate to spend the £2,500 you might drop on a modest home box upfront.
The planet and the hours
But it is often argued that a small home setup is greener. A desktop box drawing 180 watts sounds better environmentally than a data centre GPU drawing 700 watts. That is until you count the energy per answer rather than per machine.
The GPU in the data centre is serving hundreds of people at once, so each answer costs a tiny bit of electricity. The box in the spare room is serving one person and sitting idle most of the day. For one person, the data centre server is the greener choice.
The four ways in, and who they suit
The subscription
You pay a flat monthly fee for the best model a lab will sell you, the frontier models, through a chat window, plus agent tools that write code, browse the web and edit files for you. The usage limit resets on a timer. It suits anyone working alone or in a small team who sells their knowledge : writers, developers, consultants, teachers. It is the cheapest way to a top model there is, because on subscription, the tokens are subsidised.
In return you give up control. The limits change, the model updates under you and your data goes to a US company. Further, it won’t work in a boggy field with no signal. But most of what a solopreneur feeds a model is not top secret. It’s a blog post or a first draft or a plan for the same genre of app thousands of other people are also building.
The API
You pay per token for the same frontier models. There’s no weekly allowance because you pay as you go. But of course you need software of your own, or a tool that takes your key, to use it.
The bill grows with every word in and every word out. This is for anyone building a product with a model inside it, anyone running an automated process that can’t stop when tokens run out, or anyone whose use is so infrequent that a flat rate would be a poor economy.
You lose predictability, and you lose the subsidy. Heavy use through the API costs several times the subscription for the same work, and an agent left running overnight can leave you with a nasty surprise in the morning.
Somebody else’s hardware
You run a free model because you do not want to pay out for a frontier one. But you run it on somebody else’s computer. There are two ways to do this.
Marketplace
You can pay per token through a marketplace such as OpenRouter, or through the labs’ own websites, which host the open models and charge a fraction of what the closed ones cost. Some of the smaller ones are free.
GPU Rental
Or you can rent a GPU by the hour from a cloud company, load whatever free model you like, and switch it off when you’re done. That gives you your own machine without buying one.
This method suits a cheap job done in bulk, like summarising ten thousand documents, or an agent working through the night on something. It suits anyone who wants to try a free model before buying a box, or to swap between models to try them out.
Most of the LinkedIn argument was between people who owned hardware and people who paid Anthropic. For some reason they mostly ignored this option.
What you lose is the best model, again, because the free ones are still a year or two behind, and a little control, because the machine isn’t yours. Renting a GPU also takes some technical confidence, since you are setting the model up yourself rather than opening a chat window.
Your own hardware
You buy a box with a lot of memory, download free models and build whatever harness you need. Nobody meters you, nobody can raise the price, and nothing leaves your premises.
It suits people bound by law or contract to keep data on site, people who have to work without an internet connection, and people who want agents running all night where a smaller model is good enough and the volume would be ruinous at rented prices. It also suits people who enjoy building things.
But you give up the awesomeness of using the best models. Whatever is on your box is a year or two behind the best, and unless you have spent around £10,000 or more, it runs at a speed you will probably have to learn to tolerate. It also sits on hardware that will need replacing as the AI technology improves.
So who’s the mug?
The men with the server racks aren’t mugs, and neither am I. They pay in hours and hardware for control and for the pleasure of the build.
I pay £150 a month for the best model there is, running on someone else’s electricity, and keep my time for productive work. However, with all that chugging, matey’s spare room will be lovely and warm this winter.
