Container Logging: Docker and Kubernetes Log Management

Logging in container environments differs from traditional server logging. Since containers are ephemeral, collecting logs centrally outside the container is critical. This guide covers Docker and Kubernetes log management, centralized log collection solutions, and production best practices.

Docker Logging Basics

The docker logs Command

hljs bash

# View container logs
docker logs CONTAINER_NAME

# View last N lines
docker logs --tail 100 CONTAINER_NAME

# Live log streaming
docker logs -f CONTAINER_NAME

# View with timestamps
docker logs -t CONTAINER_NAME

# From a specific time
docker logs --since 2024-01-01T00:00:00 CONTAINER_NAME
docker logs --since 1h CONTAINER_NAME  # Last 1 hour

# Specific time range
docker logs --since 2h --until 1h CONTAINER_NAME

# Live + timestamps
docker logs -f -t CONTAINER_NAME

Docker Log Drivers

Docker supports different log drivers:

hljs bash

# Check current log driver
docker info | grep 'Logging Driver'

# View container log driver
docker inspect CONTAINER_NAME | grep LogConfig -A 5

Log Driver Configuration

hljs json

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

hljs bash

# Restart Docker daemon
systemctl restart docker

Log Driver at Container Start

hljs bash

# json-file with log rotation
docker run -d \
  --name myapp \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  myapp:latest

# Forward to syslog
docker run -d \
  --name myapp \
  --log-driver syslog \
  --log-opt syslog-address=udp://localhost:514 \
  myapp:latest

# Disable logging (performance critical)
docker run -d \
  --name myapp \
  --log-driver none \
  myapp:latest

Log Configuration with Docker Compose

hljs yaml

# docker-compose.yml
version: '3.8'

services:
  app:
    image: myapp:latest
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"
        compress: "true"
        labels: "app,environment"
    labels:
      - "app=myapp"
      - "environment=production"

  nginx:
    image: nginx:latest
    logging:
      driver: json-file
      options:
        max-size: "5m"
        max-file: "3"

Kubernetes Pod Logs

hljs bash

# View pod logs
kubectl logs POD_NAME

# Specific container logs (multi-container pod)
kubectl logs POD_NAME -c CONTAINER_NAME

# Live log streaming
kubectl logs -f POD_NAME

# Last N lines
kubectl logs --tail=100 POD_NAME

# From a specific time
kubectl logs --since=1h POD_NAME

# Previous container logs (after crash)
kubectl logs --previous POD_NAME

# Logs from all pods in a deployment
kubectl logs -l app=myapp --all-containers

# With namespace
kubectl logs -n production POD_NAME

Centralized Log Collection: Loki + Grafana

Loki Stack with Docker Compose

hljs yaml

# loki-stack.yml
version: '3.8'

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=StrongPass123!
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  loki-data:
  grafana-data:

Promtail Configuration

hljs yaml

# promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*?)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'

Loki with Docker Log Driver

hljs bash

# Install Loki Docker plugin
docker plugin install grafana/loki-docker-driver:latest \
  --alias loki \
  --grant-all-permissions

# Send logs to Loki at container start
docker run -d \
  --name myapp \
  --log-driver loki \
  --log-opt loki-url="http://localhost:3100/loki/api/v1/push" \
  --log-opt loki-batch-size=400 \
  myapp:latest

hljs json

// /etc/docker/daemon.json — Loki for all containers
{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push",
    "loki-batch-size": "400",
    "loki-retries": "3",
    "loki-max-backoff": "800ms",
    "loki-timeout": "1s"
  }
}

Log Rotation and Disk Management

hljs bash

# Check Docker log file sizes
du -sh /var/lib/docker/containers/*/*-json.log

# Clear all container logs (careful!)
truncate -s 0 /var/lib/docker/containers/*/*-json.log

# Clear specific container log
truncate -s 0 $(docker inspect --format='{{.LogPath}}' CONTAINER_NAME)

# Automatic rotation with logrotate
cat > /etc/logrotate.d/docker-containers << 'EOF'
/var/lib/docker/containers/*/*.log {
  rotate 7
  daily
  compress
  size=10M
  missingok
  delaycompress
  copytruncate
}
EOF

Structured Logging (JSON)

hljs javascript

// Node.js — JSON logging with Winston
const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  transports: [
    new winston.transports.Console()
  ]
});

// Usage
logger.info('User login', {
  userId: 123,
  action: 'login',
  ip: '192.168.1.1'
});

logger.error('Database error', {
  error: err.message,
  query: 'SELECT * FROM users'
});

Conclusion

Effective container logging is essential for troubleshooting and monitoring system health in production environments. Prevent disk overflow with Docker log rotation, set up centralized log collection with Loki or ELK Stack, and simplify log analysis with structured JSON logs. With these configurations on REXE servers, you can manage your container infrastructure with full visibility.

Container Logging: Docker and Kubernetes Log Management

Table of Contents