Own the Experience: When AI Runs Close to You

The location changes the experience

Artificial intelligence feels different when it runs on a computer I can see, measure, and control. When an AI service runs in the cloud, my request travels to someone else’s infrastructure, the model processes it there, and an answer comes back. That is often the right design. Cloud systems can offer enormous models, easy access, and elastic capacity.

Local AI changes that path. The model can run on a laptop, desktop, workstation, or a server inside an organization. After the software and model have been downloaded, many tasks can operate without sending every prompt across the internet.

That control matters when I am exploring sensitive documents, prototyping a new experience, testing many models, or building a workflow that must continue when the network is unavailable. But local does not automatically mean private or secure. Software can still be configured to connect outward, and a poorly protected local service can still expose data. The design must make the promise real.

Start with the job, not the largest model

The first question I ask is not, “What is the biggest model I can run?” I ask, “What experience am I trying to create?” A smaller model may be excellent at summarizing notes, classifying text, extracting fields, or supporting a focused assistant. A larger model may improve difficult reasoning, but it also needs more memory, storage, power, and time.

Parameters are the adjustable values a model learned during training. This is like the number of connections in a very large map: more connections can represent more detail, but the map also becomes heavier to carry. Model size alone does not guarantee the best answer for a specific job.

Quantization reduces how precisely those learned values are stored. This is like packing a winter coat into a vacuum bag: it is still the same coat, but it occupies less space. The tradeoff is that aggressive compression can reduce quality. The practical goal is the smallest package that still delivers an experience people can trust.

Two tools, two different intentions

Ollama and vLLM illustrate why “running locally” is not one single use case.

I see Ollama as a practical personal workbench. It simplifies downloading, managing, and running models on a computer. Its official Windows documentation makes an important reality clear: the application itself needs storage, and the models can require tens or even hundreds of gigabytes. Local AI moves the resource decision to the owner; it does not make that decision disappear.

I see vLLM as an engine room. It is designed for high-throughput model serving and can expose an OpenAI-compatible API. An API, this is like a familiar electrical outlet: applications can connect through a known shape even when the machinery behind the wall changes. vLLM also supports several forms of parallelism, allowing a system to divide work across available hardware.

Local is a design choice, not a badge

It is tempting to present local AI as the opposite of cloud AI. I see a spectrum. Some experiences should remain entirely on the device. Others can use a private organizational server. Some should use the cloud because the task requires a frontier model or rapidly changing capacity. A thoughtful product may use all three and route each task to the right place.

NIST’s AI Risk Management Framework focuses attention on trustworthiness across the design, development, use, and evaluation of AI systems. That framing is useful because the location of a model is only one part of trust. I still need to understand the data, control access, test output quality, update components, and explain limitations.

The same principle applies to speed. Local processing can remove a network round trip, but a model that overwhelms the available memory may respond more slowly than a cloud service. A fast answer that is wrong is not a good experience. A private answer produced by an unmaintained system is not automatically a safe experience.

What I measure before I trust it

I evaluate local AI as an end-to-end experience, not as a successful installation. First, I measure whether the answer is useful for the actual task. Then I measure time to first token, this is like the pause before a person begins speaking. I also measure the full response time, memory use, stability across repeated requests, and what happens when the model reaches its limits.

I test the moments around the model too. Can a regular user understand which model is active? Is it clear whether information stays on the machine? Does the system fail gracefully when memory is insufficient? Can the user remove a model and recover the storage? These details decide whether advanced technology feels empowering or confusing.

The strongest local AI experience is not the one with the most impressive specification. It is the one that makes its boundaries understandable.

Control should feel simple

Local AI gives engineers new freedom to experiment. It gives organizations another way to handle sensitive work. Most importantly, it gives product teams the opportunity to make privacy and resilience part of the experience itself.

My goal is not to move every AI task onto one machine. My goal is to put each task where it creates the most value with the least unnecessary risk. Sometimes that will be a simple local runner. Sometimes it will be a high-throughput server. Sometimes it will be the cloud.

The technology is only successful when that complexity disappears behind a choice people can understand: where their intelligence runs, what it can do, and who remains in control.