Skip to main content
Back to Category

Whisper Setup: Self-Hosted Speech Recognition and Transcription

OpenAI Whisper installation, fast transcription with faster-whisper, multi-language support, API server, Docker deployment and speech recognition optimization guide.

Read time: 13 min AI & ML Self-Hosting
whisperspeech recognitiontranscriptionartificial intelligenceself-hostingdocker
Author
REXE Teknoloji Network & Security Team
Editor
REXE Teknoloji Technical Editorial
First published
Last updated

Whisper Setup: Self-Hosted Speech Recognition and Transcription

OpenAI Whisper is a powerful AI model capable of speech recognition and transcription in 99 languages. By running it on your own server, you can perform unlimited transcription, protect your privacy, and eliminate API costs. This guide covers Whisper and faster-whisper installation, creating an API server, and Docker deployment.

What is Whisper?

Whisper is an automatic speech recognition (ASR) model developed by OpenAI. It was trained on 680,000 hours of multilingual data. Key features:

  • 99 Language Support: Transcription in 99 languages including Turkish
  • Translation: Automatic translation from any language to English
  • Timestamps: Word and sentence-level timestamps
  • Noise Resilience: High accuracy against background noise
  • Multiple Formats: MP3, WAV, M4A, FLAC, OGG and more
  • Open Source: Free usage under MIT license

Model Sizes

ModelParametersVRAMSpeedAccuracy
tiny39 M~1 GBVery fastLow
base74 M~1 GBFastMedium
small244 M~2 GBMediumGood
medium769 M~5 GBSlowVery good
large-v31.55 B~10 GBSlowestBest

Whisper Installation (Python)

Basic Installation

hljs bash
# Python virtual environment
python3 -m venv whisper-env
source whisper-env/bin/activate

# Whisper installation
pip install openai-whisper

# FFmpeg installation (for audio processing)
sudo apt-get install -y ffmpeg

Command Line Usage

hljs bash
# Basic transcription
whisper audio.mp3 --model medium --language English

# Translation to English
whisper audio.mp3 --model medium --task translate

# SRT subtitle format output
whisper audio.mp3 --model medium --language English --output_format srt

# Word-level timestamps
whisper audio.mp3 --model medium --language English --word_timestamps True

Python Usage

hljs python
import whisper

# Load model
model = whisper.load_model("medium")

# Transcription
result = model.transcribe(
    "audio.mp3",
    language="en",
    task="transcribe",
    word_timestamps=True
)

print(result["text"])

# Segment-based output
for segment in result["segments"]:
    start = segment["start"]
    end = segment["end"]
    text = segment["text"]
    print(f"[{start:.2f} - {end:.2f}] {text}")

Faster-Whisper Installation

Faster-whisper is a CTranslate2-based Whisper implementation. It runs up to 4 times faster than original Whisper and uses less memory.

hljs bash
# Faster-whisper installation
pip install faster-whisper

Python with Faster-Whisper

hljs python
from faster_whisper import WhisperModel

# Load model (GPU)
model = WhisperModel(
    "large-v3",
    device="cuda",
    compute_type="float16"
)

# For CPU mode
# model = WhisperModel("large-v3", device="cpu", compute_type="int8")

# Transcription
segments, info = model.transcribe(
    "audio.mp3",
    language="en",
    beam_size=5,
    word_timestamps=True,
    vad_filter=True  # Skip silent sections
)

print(f"Detected language: {info.language} (probability: {info.language_probability:.2f})")

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

Creating an API Server

Whisper API with FastAPI

hljs python
from fastapi import FastAPI, UploadFile, File
from faster_whisper import WhisperModel
import tempfile
import os

app = FastAPI(title="Whisper API")
model = WhisperModel("large-v3", device="cuda", compute_type="float16")

@app.post("/v1/audio/transcriptions")
async def transcribe(
    file: UploadFile = File(...),
    language: str = None,
    response_format: str = "json"
):
    # Save to temporary file
    with tempfile.NamedTemporaryFile(delete=False, suffix=os.path.splitext(file.filename)[1]) as tmp:
        content = await file.read()
        tmp.write(content)
        tmp_path = tmp.name
    
    try:
        segments, info = model.transcribe(
            tmp_path,
            language=language,
            beam_size=5,
            vad_filter=True
        )
        
        segments_list = list(segments)
        full_text = " ".join([s.text.strip() for s in segments_list])
        
        if response_format == "text":
            return full_text
        
        return {
            "text": full_text,
            "language": info.language,
            "duration": info.duration,
            "segments": [
                {
                    "start": s.start,
                    "end": s.end,
                    "text": s.text.strip()
                } for s in segments_list
            ]
        }
    finally:
        os.unlink(tmp_path)

@app.get("/health")
async def health():
    return {"status": "ok", "model": "large-v3"}

Starting the Server

hljs bash
pip install fastapi uvicorn python-multipart
uvicorn whisper_api:app --host 0.0.0.0 --port 9000

Docker Deployment

Dockerfile

hljs dockerfile
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y \
    python3 python3-pip ffmpeg \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

COPY whisper_api.py .

EXPOSE 9000

CMD ["uvicorn", "whisper_api:app", "--host", "0.0.0.0", "--port", "9000"]

Docker Compose

hljs yaml
version: '3.8'
services:
  whisper:
    build: .
    container_name: whisper-api
    ports:
      - "9000:9000"
    volumes:
      - whisper_cache:/root/.cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  whisper_cache:

API Usage Examples

Transcription with cURL

hljs bash
# Transcription with file upload
curl -X POST http://localhost:9000/v1/audio/transcriptions \
  -F "file=@meeting.mp3" \
  -F "language=en"

# Automatic language detection
curl -X POST http://localhost:9000/v1/audio/transcriptions \
  -F "file=@audio.wav"

Python API Usage

hljs python
import requests

def transcribe_audio(file_path, language=None):
    url = "http://localhost:9000/v1/audio/transcriptions"
    
    with open(file_path, "rb") as f:
        files = {"file": f}
        data = {}
        if language:
            data["language"] = language
        
        response = requests.post(url, files=files, data=data)
    
    return response.json()

# Usage
result = transcribe_audio("meeting.mp3", language="en")
print(f"Text: {result['text']}")
print(f"Duration: {result['duration']:.1f} seconds")

for seg in result["segments"]:
    print(f"  [{seg['start']:.1f}s] {seg['text']}")

Batch Transcription

hljs python
import os
import glob
from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

# Transcribe all audio files in directory
audio_files = glob.glob("audio_files/*.mp3") + glob.glob("audio_files/*.wav")

for audio_file in audio_files:
    print(f"Processing: {audio_file}")
    segments, info = model.transcribe(audio_file, language="en", vad_filter=True)
    
    output_file = os.path.splitext(audio_file)[0] + ".txt"
    with open(output_file, "w", encoding="utf-8") as f:
        for segment in segments:
            f.write(f"[{segment.start:.2f} - {segment.end:.2f}] {segment.text}\n")
    
    print(f"  Saved: {output_file}")

SRT Subtitle Generation

hljs python
def generate_srt(segments, output_path):
    with open(output_path, "w", encoding="utf-8") as f:
        for i, segment in enumerate(segments, 1):
            start = format_timestamp(segment.start)
            end = format_timestamp(segment.end)
            f.write(f"{i}\n{start} --> {end}\n{segment.text.strip()}\n\n")

def format_timestamp(seconds):
    hours = int(seconds // 3600)
    minutes = int((seconds % 3600) // 60)
    secs = int(seconds % 60)
    millis = int((seconds % 1) * 1000)
    return f"{hours:02d}:{minutes:02d}:{secs:02d},{millis:03d}"

# Usage
segments, info = model.transcribe("video.mp4", language="en")
generate_srt(list(segments), "subtitles.srt")

Perform high-speed speech recognition and transcription with Whisper on REXE GPU servers. Get high-accuracy results in 99 languages including English with the large-v3 model.

Frequently Asked Questions

What audio formats does Whisper support?

Whisper supports MP3, WAV, M4A, FLAC, OGG, WMA and many more audio formats. As long as FFmpeg is installed, it can extract audio from and transcribe almost all audio and video formats.

What is the difference between faster-whisper and original Whisper?

Faster-whisper is a CTranslate2-based implementation that runs up to 4 times faster than original Whisper. It also uses less memory. With INT8 quantization support, it works efficiently in CPU mode as well.

Does Whisper work without a GPU?

Yes, Whisper works in CPU mode as well. However, GPU provides much faster results. In CPU mode, you can improve performance using faster-whisper with INT8 quantization. Small or base models are suitable for CPU.

How accurate is English transcription?

Whisper large-v3 model provides high accuracy in English. Over 95% accuracy can be achieved with clean audio recordings. Accuracy may decrease in noisy environments or with accented speech. Using VAD filter improves accuracy.

How are long audio files processed?

Whisper automatically splits audio files into 30-second segments. Files lasting hours can also be processed. Faster-whisper's VAD filter shortens processing time by skipping silent sections.

Can live audio streams be transcribed?

Whisper fundamentally works file-based, but near-real-time transcription can be achieved by splitting the audio stream into small chunks. Projects like whisper-streaming can be used for this purpose.

Related Articles

Ollama Installation: Local LLM Running Guide

Ollama installation, running local LLM models (Llama 3, Mistral, Gemma), API usage, model management, GPU acceleration and Docker deployment.

15 min
ollamallmartificial intelligence
Network TrafficInbound— GbpsOutbound— Gbps