whisper

SKILL.md

Whisper - Robust Speech Recognition

OpenAI's multilingual speech recognition model.

When to use Whisper

Use when:

Speech-to-text transcription (99 languages)

Podcast/video transcription

Meeting notes automation

Translation to English

Noisy audio transcription

Multilingual audio processing

Metrics:

72,900+ GitHub stars

99 languages supported

Trained on 680,000 hours of audio

MIT License

Use alternatives instead:

AssemblyAI: Managed API, speaker diarization

Deepgram: Real-time streaming ASR

Google Speech-to-Text: Cloud-based

Quick start

Installation

# Requires Python 3.8-3.11
pip install -U openai-whisper

# Requires ffmpeg
# macOS: brew install ffmpeg
# Ubuntu: sudo apt install ffmpeg
# Windows: choco install ffmpeg

Basic transcription

import whisper

# Load model
model = whisper.load_model("base")

# Transcribe
result = model.transcribe("audio.mp3")

# Print text
print(result["text"])

# Access segments
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")

Model sizes

# Available models
models = ["tiny", "base", "small", "medium", "large", "turbo"]

# Load specific model
model = whisper.load_model("turbo")  # Fastest, good quality

Model
Parameters
English-only
Multilingual
Speed
VRAM

tiny
39M
✓
✓
~32x
~1 GB

base
74M
✓
✓
~16x
~1 GB

small
244M
✓
✓
~6x
~2 GB

medium
769M
✓
✓
~2x
~5 GB

large
1550M
✗
✓
1x
~10 GB

turbo
809M
✗
✓
~8x
~6 GB

Recommendation: Use turbo for best speed/quality, base for prototyping

Transcription options

Language specification

# Auto-detect language
result = model.transcribe("audio.mp3")

# Specify language (faster)
result = model.transcribe("audio.mp3", language="en")

# Supported: en, es, fr, de, it, pt, ru, ja, ko, zh, and 89 more

Task selection

# Transcription (default)
result = model.transcribe("audio.mp3", task="transcribe")

# Translation to English
result = model.transcribe("spanish.mp3", task="translate")
# Input: Spanish audio → Output: English text

Initial prompt

# Improve accuracy with context
result = model.transcribe(
    "audio.mp3",
    initial_prompt="This is a technical podcast about machine learning and AI."
)

# Helps with:
# - Technical terms
# - Proper nouns
# - Domain-specific vocabulary

Timestamps

# Word-level timestamps
result = model.transcribe("audio.mp3", word_timestamps=True)

for segment in result["segments"]:
    for word in segment["words"]:
        print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")

Temperature fallback

# Retry with different temperatures if confidence low
result = model.transcribe(
    "audio.mp3",
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
)

Command line usage

# Basic transcription
whisper audio.mp3

# Specify model
whisper audio.mp3 --model turbo

# Output formats
whisper audio.mp3 --output_format txt     # Plain text
whisper audio.mp3 --output_format srt     # Subtitles
whisper audio.mp3 --output_format vtt     # WebVTT
whisper audio.mp3 --output_format json    # JSON with timestamps

# Language
whisper audio.mp3 --language Spanish

# Translation
whisper spanish.mp3 --task translate

Batch processing

import os

audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]

for audio_file in audio_files:
    print(f"Transcribing {audio_file}...")
    result = model.transcribe(audio_file)

    # Save to file
    output_file = audio_file.replace(".mp3", ".txt")
    with open(output_file, "w") as f:
        f.write(result["text"])

Real-time transcription

# For streaming audio, use faster-whisper
# pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="float16")

# Transcribe with streaming
segments, info = model.transcribe("audio.mp3", beam_size=5)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

GPU acceleration

import whisper

# Automatically uses GPU if available
model = whisper.load_model("turbo")

# Force CPU
model = whisper.load_model("turbo", device="cpu")

# Force GPU
model = whisper.load_model("turbo", device="cuda")

# 10-20× faster on GPU

Integration with other tools

Subtitle generation

# Generate SRT subtitles
whisper video.mp4 --output_format srt --language English

# Output: video.srt

With LangChain

from langchain.document_loaders import WhisperTranscriptionLoader

loader = WhisperTranscriptionLoader(file_path="audio.mp3")
docs = loader.load()

# Use transcription in RAG
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

Extract audio from video

# Use ffmpeg to extract audio
ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav

# Then transcribe
whisper audio.wav

Best practices

Use turbo model - Best speed/quality for English

Specify language - Faster than auto-detect

Add initial prompt - Improves technical terms

Use GPU - 10-20× faster

Batch process - More efficient

Convert to WAV - Better compatibility

Split long audio - <30 min chunks

Check language support - Quality varies by language

Use faster-whisper - 4× faster than openai-whisper

Monitor VRAM - Scale model size to hardware

Performance

Model
Real-time factor (CPU)
Real-time factor (GPU)

tiny
~0.32
~0.01

base
~0.16
~0.01

turbo
~0.08
~0.01

large
~1.0
~0.05

Real-time factor: 0.1 = 10× faster than real-time

Language support

Top-supported languages:

English (en)

Spanish (es)

French (fr)

German (de)

Italian (it)

Portuguese (pt)

Russian (ru)

Japanese (ja)

Korean (ko)

Chinese (zh)

Full list: 99 languages total

Limitations

Hallucinations - May repeat or invent text

Long-form accuracy - Degrades on >30 min audio

Speaker identification - No diarization

Accents - Quality varies

Background noise - Can affect accuracy

Real-time latency - Not suitable for live captioning

Resources

GitHub: https://github.com/openai/whisper ⭐ 72,900+

Paper: https://arxiv.org/abs/2212.04356

Model Card: https://github.com/openai/whisper/blob/main/model-card.md

Colab: Available in repo

License: MIT