Ollama installation, running local LLM models (Llama 3, Mistral, Gemma), API usage, model management, GPU acceleration and Docker deployment.
Ollama is an open-source tool that allows you to easily run large language models (LLMs) on your local server. You can download and use Llama 3, Mistral, Gemma, Phi-3 and many more models with a single command. This guide covers Ollama installation, model management, API usage and GPU acceleration in detail.
Ollama is a lightweight runtime designed to run LLM models locally. It allows you to download and manage models with a Docker-like approach. Key features:
| Component | Minimum | Recommended |
|---|---|---|
| RAM | 8 GB | 16 GB+ |
| Disk | 20 GB | 50 GB+ |
| GPU (optional) | NVIDIA 4 GB VRAM | NVIDIA 8 GB+ VRAM |
| OS | Linux, macOS, Windows | Ubuntu 22.04 LTS |
Use the official installation script to install Ollama on Linux:
curl -fsSL https://ollama.com/install.sh | sh
After installation, the Ollama service starts automatically. Check the status:
systemctl status ollama
Output:
● ollama.service - Ollama Service
Loaded: loaded (/etc/systemd/system/ollama.service.d/...)
Active: active (running)
If the script doesn't work, you can install manually:
# Download binary
curl -L https://ollama.com/download/ollama-linux-amd64 -o /usr/local/bin/ollama
chmod +x /usr/local/bin/ollama
# Create ollama user
useradd -r -s /bin/false -m -d /usr/share/ollama ollama
# Systemd service file
cat > /etc/systemd/system/ollama.service << 'EOF'
[Unit]
Description=Ollama Service
After=network-online.target
[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="HOME=/usr/share/ollama"
[Install]
WantedBy=default.target
EOF
systemctl daemon-reload
systemctl enable --now ollama
To run Ollama as a Docker container:
docker run -d \
--name ollama \
-p 11434:11434 \
-v ollama_data:/root/.ollama \
ollama/ollama
docker run -d \
--name ollama \
--gpus all \
-p 11434:11434 \
-v ollama_data:/root/.ollama \
ollama/ollama
version: '3.8'
services:
ollama:
image: ollama/ollama
container_name: ollama
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: unless-stopped
volumes:
ollama_data:
# Llama 3.1 8B model
ollama pull llama3.1
# Mistral 7B
ollama pull mistral
# Google Gemma 2 9B
ollama pull gemma2
# Microsoft Phi-3
ollama pull phi3
# CodeLlama (for code generation)
ollama pull codellama
# Small model (for low RAM)
ollama pull llama3.1:8b-q4_0
# Interactive chat
ollama run llama3.1
# Single query
ollama run llama3.1 "How do I check disk usage on Linux?"
# With system prompt
ollama run llama3.1 --system "You are a Linux system administrator."
# List installed models
ollama list
# Model info
ollama show llama3.1
# Delete model
ollama rm mistral
# Show running models
ollama ps
# Copy model
ollama cp llama3.1 my-llama
Ollama provides a REST API on port 11434:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is Docker?"
}
],
"stream": false
}'
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1",
"prompt": "Write an Nginx reverse proxy configuration",
"stream": false
}'
curl http://localhost:11434/api/embeddings -d '{
"model": "llama3.1",
"prompt": "Server security best practices"
}'
import requests
import json
def chat_with_ollama(prompt, model="llama3.1"):
response = requests.post(
"http://localhost:11434/api/chat",
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"stream": False
}
)
return response.json()["message"]["content"]
# Usage
result = chat_with_ollama("How to sort a list in Python?")
print(result)
Modelfile is a configuration file similar to Dockerfile:
# Modelfile
FROM llama3.1
# System prompt
SYSTEM """You are a REXE Technology technical support assistant.
You help with server management, Docker, Linux and networking topics."""
# Model parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
PARAMETER stop "<|eot_id|>"
Create the custom model:
ollama create rexe-assistant -f Modelfile
ollama run rexe-assistant
Ollama automatically detects NVIDIA GPUs. Make sure CUDA drivers are installed:
# Check NVIDIA driver
nvidia-smi
# CUDA version
nvcc --version
# Verify Ollama GPU usage
ollama run llama3.1 "test" 2>&1 | grep -i gpu
# Ollama with ROCm support
curl -fsSL https://ollama.com/install.sh | OLLAMA_ACCELERATION=rocm sh
# Docker with AMD GPU
docker run -d \
--name ollama \
--device /dev/kfd \
--device /dev/dri \
-p 11434:11434 \
-v ollama_data:/root/.ollama \
ollama/ollama:rocm
# GPU configuration via environment variables
export OLLAMA_NUM_GPU=1 # Number of GPUs to use
export OLLAMA_GPU_MEMORY=6144 # Maximum GPU memory (MB)
export OLLAMA_MAX_LOADED_MODELS=2 # Simultaneously loaded models
# Select specific GPU
export CUDA_VISIBLE_DEVICES=0
# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434" # Access from all interfaces
Environment="OLLAMA_MODELS=/data/ollama" # Model directory
Environment="OLLAMA_NUM_PARALLEL=4" # Parallel request count
Environment="OLLAMA_MAX_LOADED_MODELS=2" # Loaded model limit
Environment="OLLAMA_KEEP_ALIVE=5m" # Model memory retention time
Apply changes:
systemctl daemon-reload
systemctl restart ollama
# Save RAM with quantized models
ollama pull llama3.1:8b-q4_0 # 4-bit quantize (~4.7 GB)
ollama pull llama3.1:8b-q5_1 # 5-bit quantize (~5.7 GB)
ollama pull llama3.1:8b-q8_0 # 8-bit quantize (~8.5 GB)
# Context window setting
ollama run llama3.1 --num-ctx 2048 # Lower = less memory
With REXE GPU servers, you can run large language models with high performance using Ollama. NVIDIA A100/H100 GPUs provide fast responses even with 70B parameter models.
Ollama supports Llama 3.1, Mistral, Gemma 2, Phi-3, CodeLlama, Qwen 2, DeepSeek Coder, Vicuna, Neural Chat and many more open-source models. For the full list, use the 'ollama list' command or visit ollama.com/library.
Yes, Ollama works in CPU mode as well. However, response times will be much slower compared to GPU. In CPU mode, 7B parameter models run at reasonable speeds. GPU is recommended for larger models.
Ollama provides /api/chat and /api/generate endpoints. It also has an OpenAI-compatible /v1/chat/completions endpoint. This allows applications using the OpenAI SDK to be redirected to Ollama with minimal changes.
Yes, you can set the number of models kept in memory simultaneously with the OLLAMA_MAX_LOADED_MODELS environment variable. Multiple models can serve in parallel as long as there is sufficient RAM/VRAM.
On Linux, models are stored by default in /usr/share/ollama/.ollama/models. You can change this directory with the OLLAMA_MODELS environment variable. If using Docker, ensure persistent storage with volume mounts.
By default, Ollama is only accessible from localhost. For remote access, set the OLLAMA_HOST=0.0.0.0:11434 environment variable. For security, place it behind a reverse proxy (Nginx) and add authentication.
Open WebUI installation with Docker, Ollama integration, user management, RAG (document querying), model management, custom prompts and multi-user support.
NVIDIA driver installation, CUDA toolkit, nvidia-container-toolkit, model serving with vLLM, GPU monitoring and AI model deployment with Docker guide.
Stable Diffusion installation, AUTOMATIC1111 and ComfyUI interfaces, model downloading, LoRA training, ControlNet usage and Docker self-hosted image generation guide.