1. Install Ollama#

One line on macOS and Linux:

1
curl -fsSL https://ollama.com/install.sh | sh
2

3
# Validate
4
ollama --version

On Windows, use PowerShell:

1
irm https://ollama.com/install.ps1 | iex

macOS users can also use Homebrew: brew install ollama.

The install script also registers Ollama as a background service (systemd on Linux, launchd on macOS), so after installation the API is already listening on http://localhost:11434. If ollama serve reports address already in use, don’t panic—that just means the service was already running.

1
# Pull and enter the interactive REPL
2
ollama run llama3.1:8b
3

4
# Common management commands
5
ollama list          # Downloaded models
6
ollama ps            # Models currently loaded into VRAM/memory
7
ollama show llama3.1:8b
8
ollama stop llama3.1:8b   # Manually unload to free VRAM
9
ollama rm llama3.1:8b

1
export OLLAMA_KEEP_ALIVE=30m

2. Customize your own model with a Modelfile#

A Modelfile is to a model what a Dockerfile is to a container—it bakes in “the parameters you’d otherwise have to pass repeatedly on every call.” This is Ollama’s most underrated feature.

Create a Modelfile:

1
FROM qwen2.5:7b
2

3
# Sampling parameters
4
PARAMETER temperature 0.3
5
PARAMETER num_ctx 8192
6
PARAMETER top_p 0.9
7

8
# System prompt, determines the model's default persona
9
SYSTEM """You are a rigorous code review assistant.
10
You point out where the problems are and give directly applicable fix suggestions.
11
Do not belabor obviously correct code."""

Then build and run:

1
ollama create code-reviewer -f ./Modelfile
2
ollama run code-reviewer

Want to modify an existing model? ollama show --modelfile <model> can print out its full recipe; change a few lines and then create.

Two parameters are worth expanding on:

• num_ctx: Ollama’s default context window is relatively conservative (commonly 2048 by default). When doing RAG or throwing long documents in, you must increase it, otherwise the excess will be silently truncated—this is the number one reason for “the document clearly contains this passage, but the model says it doesn’t know.” The cost is VRAM and speed.
• Quantization level: The q4_K_M in the model name is the default recommended quantization tier, roughly 4 times smaller than FP16, with very little quality loss on most tasks. When VRAM is tight, drop to q4_0; if you’re chasing quality, you can go up to q8_0.

3. Use it like OpenAI#

Besides the native API (/api/generate, /api/chat), Ollama also exposes an OpenAI-compatible interface http://localhost:11434/v1. This means LangChain, LlamaIndex, various Agent frameworks, and IDE plugins often only need a single base_url changed.

1
from openai import OpenAI
2

3
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
4
# api_key is a required field enforced by the client library, but Ollama ignores its value
5

6
resp = client.chat.completions.create(
7
    model="llama3.1:8b",
8
    messages=[{"role": "user", "content": "Explain what quantization is in one sentence"}],
9
    temperature=0.3,
10
)
11
print(resp.choices[0].message.content)

If you don’t want to install an SDK, the native interface is more direct with curl:

1
curl http://localhost:11434/api/chat -d '{
2
  "model": "llama3.1:8b",
3
  "messages": [{"role": "user", "content": "Hello"}],
4
  "stream": false
5
}'

Setting stream to false returns the complete result in one go; streaming is NDJSON, one JSON object per line. Use non-streaming when debugging, and streaming in production to improve perceived responsiveness.

4. Install the graphical interface Open WebUI#

The command line is enough to get by, but an interface with chat history, multiple sessions, and document upload is a much better experience. Open WebUI is a self-hosted ChatGPT-style frontend with all data kept locally.

1
docker run -d -p 3000:8080 \
2
  --add-host=host.docker.internal:host-gateway \
3
  -v open-webui:/app/backend/data \
4
  --name open-webui --restart always \
5
  ghcr.io/open-webui/open-webui:main

A few parameters explained:

• -p 3000:8080: Listens on 8080 inside the container, mapped to port 3000 on the host, so you access http://localhost:3000.
• -v open-webui:/app/backend/data: Persists chat history, accounts, and settings to a named volume. Without this, all data is gone once the container is deleted.
• --add-host=host.docker.internal:host-gateway: Lets the container access Ollama on the host. Ollama is installed on the host rather than in the container because this way you can reuse the existing GPU drivers and model cache, avoiding re-downloading tens of GB.

If you have an NVIDIA GPU, add docker run to --gpus all, and switch the image name to the :cuda tag.

After startup, open http://localhost:3000. The first registered account automatically becomes the administrator, then:

1. Avatar in the top right → Admin Panel → Settings → External Connections
2. Fill in the Ollama API address as http://host.docker.internal:11434
3. Click Verify Connection; if you see a green checkmark, you’re done

If the connection fails, first confirm that curl http://localhost:11434 returns “Ollama is running.” If the list in the UI is empty, it’s usually because no models have been pulled yet.

5. Turn it into a private knowledge base (RAG)#

Open WebUI has a built-in RAG engine, so no extra code is needed.

First install an embedding model, used to convert document chunks into vectors:

1
ollama pull nomic-embed-text
2
# For better Chinese documentation results, you can choose bge-m3
3
ollama pull bge-m3

Then in Admin Panel → Settings → Documents, set Semantic Vector Model Engine to Ollama, choose bge-m3 as the Embedding model, and save.

Two ways to use it:

• Ask documents temporarily: In the chat input box, click the paperclip icon to upload a PDF/Word/TXT, and subsequent questions will automatically include the context of this document. The first upload will be slower because the embedding model has to be loaded on the spot.
• Long-term knowledge base: Workspace → Knowledge → + to create a new knowledge base, and upload files in batches. After that, typing # in chat will attach the entire knowledge base to the current conversation.

Once the volume of documents grows large, the default vector storage becomes a bottleneck. You can migrate vector retrieval to a dedicated vector database like Qdrant; Open WebUI supports switching via environment variables.

Pitfall checklist#

A few problems I actually ran into during setup:

1. Irrelevant answers / claiming content isn’t in the document. First check num_ctx. The default context often can’t fit the chunks that RAG stuffs in, and the excess is silently discarded; the symptom is “I clearly uploaded it, but it says it doesn’t know.” Set it to 8192 or 16384 and try again.

2. Running out of RAM rather than VRAM. When VRAM is insufficient, Ollama automatically puts some layers on the CPU, and the result isn’t an error but extreme slowness. Check with ollama ps; if the PROCESSOR column shows CPU/GPU mixed, that’s the situation. Use a smaller model or lower the quantization tier.

3. Slow model loading. The few seconds of lag before each conversation is cold start. Set OLLAMA_KEEP_ALIVE=30m to keep commonly used models resident.

4. It can only run GGUF. Ollama is built on llama.cpp and only accepts the GGUF format. safetensors models on HuggingFace need to be converted first, or you should just switch to an inference framework like vLLM.

5. It’s a single-user tool, don’t treat it as a production service. Ollama has no concurrent scheduling or queue management; if several people use it at the same time, they’ll queue behind each other. If you really need a multi-user production-grade deployment, vLLM or TGI is a more suitable choice.

Conclusion#

The value of local large models isn’t in “free replacement for GPT-4”—the reasoning ability of 7B models is indeed still far behind. Their value lies in controllability: data never leaves the machine, no network dependency, no pay-per-use billing, you can freely tweak the system prompt, and you can keep using it on the subway with no signal. For scenarios involving sensitive data, or if you just want to understand what LLMs are really about at low cost, this combo offers great value for money.

From curl one-line installation to chatting your first sentence in the browser, ten minutes is enough if you’re proficient. The rest of the time, spend it on tuning num_ctx and picking models—that’s where the real fun of local deployment lies.

References#

• Ollama official documentation and API reference: https://github.com/ollama/ollama/blob/main/docs/api.md
• Ollama OpenAI-compatible API documentation: https://github.com/ollama/ollama/blob/main/docs/openai.md
• Open WebUI official documentation: https://docs.openwebui.com/
• Ollama model library: https://ollama.com/library
• llama.cpp project homepage: https://github.com/ggml-org/llama.cpp