Ollama Installation: Local LLM Running Guide

Ollama is an open-source tool that allows you to easily run large language models (LLMs) on your local server. You can download and use Llama 3, Mistral, Gemma, Phi-3 and many more models with a single command. This guide covers Ollama installation, model management, API usage and GPU acceleration in detail.

What is Ollama?

Ollama is a lightweight runtime designed to run LLM models locally. It allows you to download and manage models with a Docker-like approach. Key features:

Easy Installation: Single command setup and model download
Wide Model Support: Llama 3, Mistral, Gemma, Phi-3, CodeLlama, Qwen and more
REST API: OpenAI-compatible API endpoint
GPU Acceleration: NVIDIA and AMD GPU support
Modelfile: Create custom model configurations
Low Resource Usage: Low memory consumption with quantized models

System Requirements

Component	Minimum	Recommended
RAM	8 GB	16 GB+
Disk	20 GB	50 GB+
GPU (optional)	NVIDIA 4 GB VRAM	NVIDIA 8 GB+ VRAM
OS	Linux, macOS, Windows	Ubuntu 22.04 LTS

Installation on Linux

Use the official installation script to install Ollama on Linux:

hljs bash

curl -fsSL https://ollama.com/install.sh | sh

After installation, the Ollama service starts automatically. Check the status:

hljs bash

systemctl status ollama

Output:

● ollama.service - Ollama Service
     Loaded: loaded (/etc/systemd/system/ollama.service.d/...)
     Active: active (running)

Manual Installation

If the script doesn't work, you can install manually:

hljs bash

# Download binary
curl -L https://ollama.com/download/ollama-linux-amd64 -o /usr/local/bin/ollama
chmod +x /usr/local/bin/ollama

# Create ollama user
useradd -r -s /bin/false -m -d /usr/share/ollama ollama

# Systemd service file
cat > /etc/systemd/system/ollama.service << 'EOF'
[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="HOME=/usr/share/ollama"

[Install]
WantedBy=default.target
EOF

systemctl daemon-reload
systemctl enable --now ollama

Docker Installation

To run Ollama as a Docker container:

CPU Mode

hljs bash

docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

GPU Mode (NVIDIA)

hljs bash

docker run -d \
  --name ollama \
  --gpus all \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

Docker Compose

hljs yaml

version: '3.8'
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama_data:

Downloading and Running Models

Download Models

hljs bash

# Llama 3.1 8B model
ollama pull llama3.1

# Mistral 7B
ollama pull mistral

# Google Gemma 2 9B
ollama pull gemma2

# Microsoft Phi-3
ollama pull phi3

# CodeLlama (for code generation)
ollama pull codellama

# Small model (for low RAM)
ollama pull llama3.1:8b-q4_0

Running Models

hljs bash

# Interactive chat
ollama run llama3.1

# Single query
ollama run llama3.1 "How do I check disk usage on Linux?"

# With system prompt
ollama run llama3.1 --system "You are a Linux system administrator."

Model Management

hljs bash

# List installed models
ollama list

# Model info
ollama show llama3.1

# Delete model
ollama rm mistral

# Show running models
ollama ps

# Copy model
ollama cp llama3.1 my-llama

REST API Usage

Ollama provides a REST API on port 11434:

Chat Completion

hljs bash

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is Docker?"
    }
  ],
  "stream": false
}'

Text Generation

hljs bash

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Write an Nginx reverse proxy configuration",
  "stream": false
}'

Creating Embeddings

hljs bash

curl http://localhost:11434/api/embeddings -d '{
  "model": "llama3.1",
  "prompt": "Server security best practices"
}'

Python API Usage

hljs python

import requests
import json

def chat_with_ollama(prompt, model="llama3.1"):
    response = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False
        }
    )
    return response.json()["message"]["content"]

# Usage
result = chat_with_ollama("How to sort a list in Python?")
print(result)

Custom Models with Modelfile

Modelfile is a configuration file similar to Dockerfile:

hljs dockerfile

# Modelfile
FROM llama3.1

# System prompt
SYSTEM """You are a REXE Technology technical support assistant.
You help with server management, Docker, Linux and networking topics."""

# Model parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
PARAMETER stop "<|eot_id|>"

Create the custom model:

hljs bash

ollama create rexe-assistant -f Modelfile
ollama run rexe-assistant

GPU Acceleration

NVIDIA GPU

Ollama automatically detects NVIDIA GPUs. Make sure CUDA drivers are installed:

hljs bash

# Check NVIDIA driver
nvidia-smi

# CUDA version
nvcc --version

# Verify Ollama GPU usage
ollama run llama3.1 "test" 2>&1 | grep -i gpu

AMD GPU (ROCm)

hljs bash

# Ollama with ROCm support
curl -fsSL https://ollama.com/install.sh | OLLAMA_ACCELERATION=rocm sh

# Docker with AMD GPU
docker run -d \
  --name ollama \
  --device /dev/kfd \
  --device /dev/dri \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama:rocm

GPU Memory Management

hljs bash

# GPU configuration via environment variables
export OLLAMA_NUM_GPU=1          # Number of GPUs to use
export OLLAMA_GPU_MEMORY=6144    # Maximum GPU memory (MB)
export OLLAMA_MAX_LOADED_MODELS=2 # Simultaneously loaded models

# Select specific GPU
export CUDA_VISIBLE_DEVICES=0

Environment Variables

hljs bash

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"    # Access from all interfaces
Environment="OLLAMA_MODELS=/data/ollama"     # Model directory
Environment="OLLAMA_NUM_PARALLEL=4"          # Parallel request count
Environment="OLLAMA_MAX_LOADED_MODELS=2"     # Loaded model limit
Environment="OLLAMA_KEEP_ALIVE=5m"           # Model memory retention time

Apply changes:

hljs bash

systemctl daemon-reload
systemctl restart ollama

Performance Tips

hljs bash

# Save RAM with quantized models
ollama pull llama3.1:8b-q4_0    # 4-bit quantize (~4.7 GB)
ollama pull llama3.1:8b-q5_1    # 5-bit quantize (~5.7 GB)
ollama pull llama3.1:8b-q8_0    # 8-bit quantize (~8.5 GB)

# Context window setting
ollama run llama3.1 --num-ctx 2048  # Lower = less memory

With REXE GPU servers, you can run large language models with high performance using Ollama. NVIDIA A100/H100 GPUs provide fast responses even with 70B parameter models.

Ollama Installation: Local LLM Running Guide

Table of Contents