Leah clears space on her laptop, downloads an AI model and watches the cursor blink.
She wanted help with a theatre newsletter. Now she suspects the laptop wants an interval.
Leah and the scenes below are fictional, but you can use her questions to check your own setup.
Keep working when the connection disappears
At rehearsal, the Wi-Fi drops just as Leah sits down to shorten a notice about lost umbrellas from the draft on her laptop.
Her model and the software needed to run it are already downloaded, so the local chat can work without an internet connection.
LM Studio's offline guide explains which tasks work offline and which still need the network.
You can bring your own documents and work through them without sending them to a hosted model.
When your notes contain something you would rather keep at home, check the whole app, including any connected services.
The Ollama FAQ distinguishes local runs from cloud models and explains how to disable cloud features.
Before someone borrows her laptop, Leah locks it. Check who can open your saved chats or backups too.
Keep control without pretending the computer is free
Leah can rewrite the newsletter whenever the director remembers another thank-you, without a cloud provider charging for those local inference tokens.
Inference means using an already trained model to produce an answer. Your computer does that work when you run it locally.
The model still processes tokens, the small pieces of text explained in our guide to tokens and embeddings.
You pay for the machine and its electricity, and you spend time keeping the setup working. A local run is not free just because no API invoice arrives.
We would try the computer you already own before buying hardware for a newsletter.
You also get to choose which supported model to keep, subject to its licence. You can supply your preferred writing examples without changing its trained weights.
With an approved notice beside her prompt, Leah gets a draft that sounds less like a royal announcement, though she still checks the rehearsal time.
Give the model somewhere to work
At home, Leah cannot see why the larger model struggles when its download fitted so comfortably on her drive.
The drive stores the model between uses. RAM is the computer's working memory, where programs can quickly reach data they need while running.
A separate graphics card usually has its own working memory, called VRAM. You cannot assume spare system RAM becomes equally fast extra VRAM.
Some computers share memory between the CPU and GPU. Apple's memory explanation shows how this differs from a separate graphics card.
Whichever design you have, the operating system and other apps need space too.
Think of Leah's rehearsal room. Owning a cupboard full of costumes does not mean she can spread every costume across the floor during a dance.
Your model's files may fit on the drive while the running setup runs short of working memory. Check the available memory, rather than just the drive's free space.
Check how quickly memory can supply the processor
Leah looks up her processor, hoping its impressive numbers will tell her whether the notice will arrive before rehearsal ends.
Computing power describes how much calculation hardware can perform in a given time. A language model uses many calculations to turn your prompt into output.
A GPU can do many suitable calculations in parallel, while a CPU can also run supported models. Having a graphics card is not a requirement for every setup.
Memory bandwidth is different. It describes how much data can move from memory in a given time.
Imagine the theatre's costume team sewing quickly but waiting for someone to bring the fabric. More sewing machines alone will not fix that queue.
Your model can face a similar wait for data. NVIDIA's inference guide explains why processing the prompt and generating the reply can hit different limits.
You cannot read a single processor score and know the chat speed. The model and how it runs matter too.
Keep the weights close without making RAM a hard rule
Leah imagines her computer reading the model from the drive for every new word and wonders whether it could fetch a little piece each time.
The model contains learned numbers called weights, which help shape its answers. A dense model repeatedly uses most of these weights while generating tokens.
Keeping them in fast working memory avoids repeatedly fetching them from the slower drive. You can think of keeping the script open beside the stage manager.
But you do not always have to copy the whole model into system RAM at once.
llama.cpp supports memory mapping, which lets the operating system bring file pages into memory as needed.
Mapping a file does not make all its contents physically present in RAM.
Supported runtimes can also split weights across CPU and GPU memory. Accelerate documents disk offloading when faster memory is insufficient.
These options do not remove the work of moving data. Repeated disk reads or transfers can make your replies painfully slow.
Make the weights smaller and check what changes
Leah finds a smaller download of the same model and worries someone has removed half its vocabulary. The label says it is quantized.
Quantization stores numbers with fewer bits, reducing the memory used for weights. It does not mean deleting a matching share of the model's words.
Here is a simplified calculation you can repeat. Assume exactly eight billion weights, each stored with sixteen bits, and eight bits to a byte.
The raw weights take 16 GB. At four bits each, that theoretical total becomes 4 GB, using decimal gigabytes.
You can multiply the weight count by the bit count and divide by eight to get bytes.
Those totals exclude extra information needed for quantization and other runtime memory. A download labeled four-bit is not a promise that your whole chat fits in 4 GB.
llama.cpp's quantization guide explains the trade-offs, including possible losses in accuracy.
Leah compares the rehearsal dates in both drafts. You should compare answers on your own tasks before deciding the smaller version is good enough.
Leave room for the conversation to grow
The short notice works, so Leah pastes in a long rehearsal history. She hopes the model will sort out who promised to return the umbrellas.
Now the same model needs more room for the conversation. Its weight file has not grown.
Many chat models keep a key-value cache, or KV cache. It stores intermediate attention results from earlier tokens so the model can reuse them.
You can picture Leah leaving rehearsal notes open instead of finding the same page again each time someone asks a question.
The cache also uses memory, and longer context can increase that cost.
The amount depends on the model and cache design. Hugging Face's cache guide explains the choices and their limits.
You also need space for temporary calculations. Increasing the allowed context or running chats in parallel can change the memory your setup needs.
Leah starts a fresh chat with only the relevant notes. You still need to supply enough information for the answer to make sense.
Try your real task before buying more hardware
Leah saves the useful notice from the smaller model. It gives the correct rehearsal time and asks people to collect their umbrellas by the door.
She decides that is enough for this job.
We would keep the draft that passes the task's checks, even if a larger download looks tempting.
Before you spend money, use a short sample you can judge and record what happens.
- Check that your app and chosen model support your hardware.
- See whether the model runs on the CPU, GPU or both.
- Check available memory while your usual apps are open.
- Try a longer prompt that resembles your actual work.
- Disconnect the network and repeat any task you need offline.
Our Ollama setup guide can help you start.
Try one small local model on a task you already understand, then check whether its answer and waiting time suit you.