Ollama: Local LLM Inference for Personal Computers and Small Services

Ollama packages model acquisition, local inference, a command-line interface, a desktop application, and an HTTP API into one tool. It is one of the easiest ways to download an open-weight model and run it on a developer workstation without first assembling a Python environment, inference engine, tokenizer, prompt template, and GPU runtime.

A minimal session is:

ollama pull qwen3:8b
ollama run qwen3:8b

Ollama selects an appropriate local runner, stores the model, starts it when needed, and applies the model’s chat template. This convenience makes Ollama a good fit for personal computers, application development, demonstrations, and small internal services. It does not automatically make a laptop into a high-throughput or highly available production inference cluster.

What Ollama Can Do #

Ollama can:

• download, list, inspect, run, copy, and delete models;
• provide interactive text chat and text generation;
• run supported vision-language, embedding, tool-calling, and reasoning models;
• constrain supported models to structured JSON output;
• import supported model weights or GGUF files and customize model behavior with a Modelfile;
• serve local models through its native REST API;
• provide partial compatibility with the OpenAI API;
• keep one or more models resident in memory and process concurrent requests, subject to available RAM or VRAM; and
• optionally use Ollama-hosted cloud models instead of local hardware.

These are runtime features, not capabilities that every model possesses. A text-only model cannot understand an image because the request uses a vision endpoint shape, and a model that was not trained for tool use may produce poor tool calls. Check the model card, license, context limit, and tags in the Ollama model library before building around it.

The local server normally listens at http://127.0.0.1:11434. The CLI, desktop app, API clients, editor extensions, and third-party chat interfaces are all clients of that service:

What Ollama Is Not #

Ollama is not:

• an LLM or a source of model weights by itself;
• a model-training or full fine-tuning framework;
• a guarantee that a model is accurate, safe, unbiased, or licensed for a particular use;
• a retrieval-augmented generation system, vector database, agent framework, or tool sandbox;
• a complete implementation of every OpenAI API endpoint and option;
• an authentication, authorization, quota, or billing gateway for the local API;
• a multi-node scheduler with automatic failover; or
• a substitute for capacity planning, monitoring, backups, and controlled upgrades.

Ollama can be one component inside a RAG or agent application, but the application still owns document ingestion, retrieval, conversation state, tool execution, access control, and error handling. Tool calls are especially important: the model proposes a call, while trusted application code must validate and execute it.

Running a model locally also does not mean that every operation is offline. Pulling a model requires network access, and cloud models or web-search features send requests to remote services. Set OLLAMA_NO_CLOUD=1 or disable_ollama_cloud in ~/.ollama/server.json when local-only operation is a requirement.

Ollama vs. vLLM #

vLLM is also an inference engine and server, but it optimizes for a different center of gravity. Its PagedAttention, continuous-batching, prefix-caching, parallelism, and distributed-serving features target efficient shared serving of many requests. Ollama optimizes the path from no runtime to a model running on one machine.

The comparison is not simply “development versus production.” An Ollama service can be suitable in production when it serves a bounded internal workload on one machine and the surrounding system supplies authentication, TLS, monitoring, and recovery. Likewise, vLLM can run on one development server. The deciding factor is the workload:

• Choose Ollama for a personal assistant, private experimentation, local application development, offline-capable workflows, classroom use, or a lightly shared single-host service.
• Choose vLLM when concurrent users, tokens per second per accelerator, predictable queueing, Prometheus metrics, distributed GPUs, or Kubernetes-based scaling justify additional operational complexity.
• Benchmark both when the intended service lies between these cases. Measure time to first token, output tokens per second, tail latency, memory use, and behavior under the expected concurrency and context lengths.

Ollama’s advantage is low operational friction and broad workstation support. Its disadvantage is that it gives up many controls and optimizations expected from a dedicated fleet-serving engine. vLLM’s advantage is accelerator utilization and serving scale; its disadvantages are a steeper setup path, greater hardware sensitivity, and the absence of a consumer desktop experience.

Deployment #

Choosing the Deployment #

The shortest reliable rule is:

Ollama succeeds by making local inference ordinary: install one application, pull one model, and use the same runtime from a chat window, terminal, editor, or program. That simplicity is its main feature. Keep the boundary clear, however. Ollama runs models; a dependable product must still provide the security, state, evaluation, observability, and operational controls around them.

Installing on a Linux Host with Docker #

Docker is a natural deployment method on a Linux inference host because the container can access Linux GPU device interfaces. First install a working GPU driver on the host. For NVIDIA GPUs, also install and configure the NVIDIA Container Toolkit:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Start Ollama with a persistent model volume. Binding the published port to loopback avoids unintentionally exposing an unauthenticated API:

docker run --detach \
  --name ollama \
  --restart unless-stopped \
  --gpus all \
  --publish 127.0.0.1:11434:11434 \
  --volume ollama:/root/.ollama \
  ollama/ollama

Pull a model and verify that inference works:

docker exec ollama ollama pull qwen3:8b
docker exec -it ollama ollama run qwen3:8b
curl http://127.0.0.1:11434/api/version
docker exec ollama ollama ps

ollama ps reports whether a loaded model is on the CPU, GPU, or split between them. Do not assume that the presence of a GPU means the model is using it.

For CPU-only inference, omit --gpus all. This is useful for small models, functional tests, and machines without a supported accelerator:

docker run --detach \
  --name ollama \
  --restart unless-stopped \
  --publish 127.0.0.1:11434:11434 \
  --volume ollama:/root/.ollama \
  ollama/ollama

For a supported AMD GPU, use the ROCm image and pass the GPU devices:

docker run --detach \
  --name ollama \
  --restart unless-stopped \
  --device /dev/kfd \
  --device /dev/dri \
  --publish 127.0.0.1:11434:11434 \
  --volume ollama:/root/.ollama \
  ollama/ollama:rocm

Consult Ollama’s current Docker and GPU compatibility pages before selecting a host. GPU family, driver, container runtime, and image must agree. For a controlled deployment, pin a tested Ollama release tag or image digest instead of implicitly upgrading with latest.

If another machine must reach the service, do not merely change the publish address to 0.0.0.0 and open the firewall. Place Ollama behind an authenticated reverse proxy or API gateway that provides TLS, request-size limits, timeouts, rate limits, and audit logging.

Installing Natively on macOS #

Native installation is the correct path on a Mac:

1. Download ollama.dmg from the official macOS installation page.
2. Drag Ollama into Applications and launch it.
3. Allow the app to place an ollama CLI link in /usr/local/bin when prompted.
4. Pull and run a model:

ollama pull qwen3:8b
ollama run qwen3:8b

Current releases require macOS Sonoma 14 or newer. Apple M-series systems can use both CPU and GPU through the native runtime; Intel Macs use CPU inference. Models and configuration are stored under ~/.ollama, and models can consume tens or hundreds of gigabytes.

The desktop application normally owns the background server. ollama serve can start the server explicitly in a terminal when a foreground service is preferred.

Why macOS and Docker Are the Wrong Combination #

A Linux container on Docker Desktop for macOS does not run directly on the macOS kernel. It runs inside a Linux virtual machine:

Apple’s Metal GPU interface is available to native macOS applications, but Docker Desktop does not pass that GPU through or emulate it as a supported Linux accelerator for the Ollama container. Ollama therefore cannot use GPU acceleration in Docker Desktop on macOS.

The container can technically run CPU-only, so “Ollama cannot run in Docker on a Mac” is too broad. The practical problem is that the Linux VM adds overhead while discarding the Mac’s main inference advantage: its GPU and unified memory. Native Ollama can use Metal, has a simpler filesystem and networking path, and provides the desktop app. Use macOS plus native Ollama for inference; reserve Linux containers for Linux hosts. A separate web UI may still run in a container and call the native Ollama server.

Ollama Administrative CLI #

The administrative CLI manages the Ollama server, models, and account session. The desktop app or system service normally starts the server; use ollama serve only when running it manually.

# Show available commands and global options.
ollama --help

# Show the installed Ollama version.
ollama --version

# Show options for a particular command.
ollama run --help

# Start the Ollama server in the foreground.
ollama serve

# Download a model, or update the local copy of an existing tag.
ollama pull qwen3:8b

# List models stored on this machine.
ollama list

# Use the shorter alias for `ollama list`.
ollama ls

# Show a model's metadata, parameters, template, and capabilities.
ollama show qwen3:8b

# List models currently loaded into memory and their CPU/GPU placement.
ollama ps

# Unload a model and release its memory.
ollama stop qwen3:8b

# Create a custom model from a Modelfile.
ollama create my-model --file ./Modelfile

# Copy a model to another local name or tag.
ollama cp qwen3:8b qwen3:project

# Delete a local model.
ollama rm qwen3:project

# Publish a namespaced model after authenticating with ollama.com.
ollama push account/my-model

# Sign in to ollama.com for cloud models and model publishing.
ollama signin

# End the ollama.com session.
ollama signout

User Interfaces #

Naive CLI Chat #

# Start an interactive chat session.
ollama run qwen3:8b

# Run a single prompt without entering an interactive session.
ollama run qwen3:8b "Explain copy-on-write in one paragraph."

# Send the contents of a text file through standard input.
cat notes.txt | ollama run qwen3:8b

# Send multiline text through standard input.
cat <<'EOF' | ollama run qwen3:8b
Compare optimistic and pessimistic locking.
Include one practical example of each.
EOF

# Ask a vision-capable model about an image.
ollama run gemma4 ./diagram.png "Explain this diagram."

# Send WAV audio to an audio-capable model.
ollama run AUDIO_MODEL ./recording.wav "Transcribe this recording."

# Generate an embedding vector from an argument.
ollama run embeddinggemma "Hello world"

# Generate an embedding vector from standard input.
echo "Hello world" | ollama run nomic-embed-text

# Interactively select and configure a supported external integration.
ollama launch

# Launch a specific integration.
ollama launch claude

# Launch an integration with a particular Ollama model.
ollama launch claude --model qwen3.5

# Configure a supported integration without launching it.
ollama launch droid --config

# Start an interactive chat through a Dockerized Ollama server.
docker exec -it ollama ollama run qwen3:8b

# At the `>>>` prompt, triple quotes delimit multiline interactive input:
# """Explain this program:
# int main(void) { return 0; }
# """

Run ollama <command> --help for command-specific options. Model names include tags: qwen3:8b and qwen3:14b are different artifacts, while an omitted tag usually resolves to latest. Image and WAV input require models with the corresponding multimodal capability. The CLI does not directly ingest video, PDF, or office-document files; transcribe, extract, or convert that content before sending it to a compatible model.

Web/GUI Frontend #

Several independent frontends can connect to an already-running Ollama service. They do not replace Ollama’s model runtime; they add conversation storage, document workflows, user accounts, or a desktop interface.

Choose according to the application boundary:

• Chatbox requires the least additional infrastructure for one user.
• Open WebUI provides a direct path to a shared browser interface centered on local models.
• LibreChat or LobeChat fits a deployment that must combine Ollama with several model providers.
• AnythingLLM centers the interface on document collections and retrieval.

Container networking changes the meaning of localhost: inside a frontend container, it identifies that container rather than the Ollama host. Use a Docker host alias, a Compose service name, or a private server address as appropriate. If a browser connects directly to Ollama, configure only the required origins with OLLAMA_ORIGINS; avoid * because it permits every web origin.

Each frontend stores additional state independently of Ollama. Back up its conversation database and uploaded documents, configure authentication, and review whether optional cloud features send prompts or files to remote services. Do not expose Ollama’s unauthenticated port merely to make a frontend reachable; expose the authenticated frontend and keep Ollama on an internal network.

Using the Native API #

The native API is available at http://localhost:11434/api. A nonstreaming chat request looks like this:

curl http://localhost:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:8b",
    "messages": [
      {"role": "user", "content": "Explain copy-on-write in two paragraphs."}
    ],
    "stream": false
  }'

Streaming is enabled by default for generation and chat, with a sequence of newline-delimited JSON objects returned as tokens are produced. Applications should parse the stream incrementally, handle disconnects and non-2xx responses, and impose their own deadlines.

Important endpoint groups include:

Ollama maintains official Python and JavaScript libraries. The Python client keeps the same basic data model:

from ollama import chat

response = chat(
    model="qwen3:8b",
    messages=[
        {"role": "user", "content": "Why does context length use memory?"}
    ],
)
print(response.message.content)

Using the OpenAI-Compatible API #

Applications already written for an OpenAI client can point supported operations at http://localhost:11434/v1/:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",
)

response = client.chat.completions.create(
    model="qwen3:8b",
    messages=[
        {"role": "user", "content": "Give me three names for a backup tool."}
    ],
)
print(response.choices[0].message.content)

The local server ignores the API key, although the OpenAI SDK requires a nonempty value. Supported compatibility endpoints include Chat Completions and Responses, but compatibility is intentionally partial. An accepted request shape does not imply support for every parameter, event, usage field, model behavior, or future OpenAI feature. Test the exact operations an application uses, and prefer Ollama’s native API when Ollama-specific controls are needed.

Context Management #

Context management spans the client and server because Ollama does not own a durable conversation database. A CLI, GUI, or application assembles each request; Ollama renders that request and enforces the model’s context limit.

Consequently, the same model and Ollama server can produce different results through different frontends because their requests may contain different system prompts, history selections, summaries, retrieved documents, and generation parameters.

Interactive ollama run #

ollama run MODEL maintains two different kinds of history:

1. Prompt-entry history lets the user recall previous input with the arrow keys. /set history and /set nohistory control this readline feature. Prompt-entry history is not the conversation supplied to the model because it contains user input rather than complete user-and-assistant turns.
2. Conversation history is an ordered array of system, user, and assistant messages held by the running CLI process. After each response, the CLI appends the assistant message. On the next turn, it sends the accumulated message array to the chat API.

Conversation history therefore survives between prompts in the same ollama run process, but it is not automatically restored after the process exits. /clear removes the current conversation context while retaining the active system message.

/save <model-name> provides explicit persistence by creating a new Ollama model whose definition includes the current messages, parameters, and system message. A later /load <model-name> loads that saved model and its initial message history. The saved artifact is a derived model definition, not a searchable transcript database. Avoid saving secrets because the embedded messages become part of the model artifact and can be inspected or copied.

Does Every Turn Reach the Model? #

The CLI sends its complete accumulated message array for every new prompt. The inference server then renders the chat template and fits the result into the configured context window. When the rendered history exceeds num_ctx, the server removes older messages from the front while preserving system messages and the latest user message. Thus:

• the API request can contain every turn retained by the CLI;
• the model sees only the system messages and newest suffix that fit;
• an old fact can disappear from effective context without disappearing from the CLI’s in-memory array; and
• increasing num_ctx retains more history but increases KV-cache memory use and can reduce concurrency.

Set and inspect the session context with:

# Change the context window for the current interactive session.
/set parameter num_ctx 16384

# Clear accumulated conversation messages.
/clear

# Persist the conversation as a derived model.
/save project-session

# Load a saved session or another model.
/load project-session

This behavior is truncation, not durable memory or semantic recall. The CLI does not summarize an old conversation before dropping turns. Applications that require durable conversation state should store messages themselves, apply an explicit summarization policy, and send the selected history through the chat API.

Frontend and API Client Strategies #

Ollama’s chat API processes the messages array in the current request; it does not retrieve a previous conversation by conversation ID. An API client must store completed turns and resend the history required for continuity.

A frontend can implement a more advanced policy than ollama run:

• retain complete transcripts in its own database;
• send every message until the request approaches the context limit;
• keep the first system message and the most recent turns;
• replace older turns with a generated summary;
• maintain separate summaries for user preferences, decisions, or tool state;
• retrieve relevant prior messages by embedding similarity; or
• inject document chunks selected by a RAG pipeline.

Frontend settings can also change Ollama’s effective context length by sending num_ctx. A larger value does not force the frontend to send more history; it only gives the assembled request more room. Conversely, a frontend may discard or summarize history well before Ollama reaches that limit.

Inspect a frontend’s documentation or outbound request when exact behavior matters. Conversation history visible in a GUI is not proof that every visible message was sent to the model. Open WebUI, LibreChat, AnythingLLM, LobeChat, and Chatbox own their conversation and retrieval policies independently of Ollama, and those policies can change between releases.

Background Documents and RAG #

The Ollama server does not include a document collection, chunker, vector index, or retrieval policy. It therefore does not provide a complete RAG system. Several integration levels are possible:

1. Direct context: paste document text into a prompt or pipe extracted text into ollama run. The model can use the text only while it remains inside the context window. Image and audio paths require a model with the matching multimodal capability; PDF and office documents must first be converted to text or images.
2. Static model context: place short, stable instructions or facts in a SYSTEM or MESSAGE entry in a Modelfile. These entries are included in prompts rather than indexed, so they consume context and do not scale to a document corpus.
3. Application-managed RAG: use Ollama’s embedding API to embed document chunks, store the vectors and source metadata in a vector database, retrieve relevant chunks for each question, and include those chunks in a chat request. Ollama supplies embedding and generation models; application code owns ingestion, retrieval, citations, access control, and conversation persistence.
4. Third-party interface: use a client such as Open WebUI that implements a knowledge-base or RAG workflow and connects to Ollama as its model runtime. The interface’s retrieval behavior, storage, permissions, and backups are separate from Ollama.

A minimal RAG request path is:

Directly pasting an entire corpus is prompt stuffing rather than RAG because no retrieval step selects relevant evidence. Proper RAG also preserves source identifiers alongside chunks so the application can display citations and enforce document-level authorization before content reaches the model.

Selecting a Model and Sizing the Machine #

Model choice has more effect on the experience than the Ollama installation. Consider:

• Task: general chat, code, embeddings, vision, tools, or domain-specific work;
• License: permission for commercial use, redistribution, and generated output;
• Parameter count and quantization: smaller or more heavily quantized models use less memory but may lose quality;
• Context length: a longer prompt and KV cache consume more memory;
• Language coverage: benchmark the languages and terminology users need; and
• Prompt template and capabilities: use the correct Ollama model tag rather than assuming two variants are interchangeable.

The model’s on-disk size is not a sufficient memory estimate. Inference also needs runtime buffers and a KV cache that grows with context length, concurrency, model architecture, and precision. On Apple Silicon, CPU and GPU share unified memory, but macOS and other applications still need part of it. On a discrete GPU host, weights that spill into system RAM generally run much more slowly.

Ollama’s default context policy has changed between releases and may depend on available VRAM. Set the required context deliberately, then use ollama ps to verify the allocated context and CPU/GPU placement:

OLLAMA_CONTEXT_LENGTH=16384 ollama serve
ollama ps

Larger is not automatically better. It increases memory pressure and can reduce concurrency or cause CPU offloading. Test quality as well: a model’s advertised maximum context does not guarantee equally good attention across that entire window.

Production and Security Checklist #

Before treating Ollama as a shared service:

1. Protect the API. The local API has no authentication. Keep it on loopback or a private network, or put an authenticated TLS proxy in front of it. Never rely on a dummy OpenAI API key for security.
2. Pin the runtime and model. Record the Ollama version, exact model tag or digest, Modelfile, prompt template, context, and generation parameters. A mutable tag or runtime upgrade can change output.
3. Plan storage. Preserve the model volume across container replacement. Use OLLAMA_MODELS when model data belongs on another disk. Back up custom Modelfiles and configuration; large registry models can usually be repulled.
4. Set capacity limits. OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS, OLLAMA_MAX_QUEUE, context length, and keep_alive all affect memory and latency. An overloaded server can reject requests with HTTP 503.
5. Warm important models. A first request may include model-load latency. An empty generate or chat request can preload a model, while keep_alive controls how long it remains resident.
6. Observe the service. Collect container or service logs, request latency, queue failures, disk use, memory, accelerator utilization, and application error rates. Ollama’s basic inspection commands are not a complete monitoring system.
7. Bound requests and outputs. Apply body-size limits, deadlines, output limits, concurrency limits, and cancellation. Treat generated text and model-proposed tool arguments as untrusted input.
8. Test failure and recovery. Verify behavior after an out-of-memory error, corrupt or full disk, interrupted model pull, process restart, and client disconnect. A second host or fallback service is needed if downtime is not acceptable.
9. Control outbound traffic. Decide whether model pulls, cloud models, web search, proxies, and third-party UIs are permitted. Local inference protects prompts only when the entire request path remains local.
10. Review both licenses. Ollama’s license and a model’s license are separate. Deploying the runtime does not grant rights to every model in its library.

References #

• Ollama documentation
• Ollama Docker installation
• Ollama macOS installation
• Ollama API reference
• Ollama OpenAI compatibility
• Ollama CLI reference
• Ollama context length
• Ollama embeddings
• Ollama Modelfile reference
• Ollama FAQ
• Ollama interactive CLI source
• Ollama chat-prompt truncation source
• vLLM documentation
• vLLM project repository