VideoCaptioner: AI-Powered Video Subtitling Assistant

Key Features:
- Speech-to-text transcription via multiple ASR engines
- Semantic-based subtitle segmentation using LLM
- Context-aware translation with reflection mechanism
- Voice dubbing (TTS) with Edge, SiliconFlow, Gemini
- Video synthesis with customizable subtitle styling
- Free cloud services for ASR and translation (no API key needed)
- Cross-platform: Windows, macOS, Linux

# Check Python version (3.10 or higher required)
python --version
python3 --version

# Check FFmpeg installation (required for video processing)
ffmpeg -version
ffprobe -version

# If FFmpeg is not installed:
# Ubuntu/Debian:
sudo apt update && sudo apt install ffmpeg

# macOS (with Homebrew):
brew install ffmpeg

# Windows (with Chocolatey):
choco install ffmpeg

# Windows (with Scoop):
scoop install ffmpeg

# Verify installation
ffmpeg -version | head -1
ffprobe -version | head -1

# Install VideoCaptioner (CLI + GUI)
pip install videocaptioner

# Verify installation
videocaptioner --version
videocaptioner --help

# Launch GUI (three equivalent ways)
videocaptioner
videocaptioner gui
videocaptioner-gui

# Run CLI help to see all commands
videocaptioner --help

1. Visit: https://github.com/WEIFENG2333/VideoCaptioner/releases
2. Download the latest Windows installer (e.g., videocaptioner-{version}-windows-x64.exe)
3. Run the installer and follow the setup wizard
4. Launch from Start Menu or desktop shortcut
5. The application includes Python runtime and all dependencies

# One-line macOS installation
curl -fsSL https://raw.githubusercontent.com/WEIFENG2333/VideoCaptioner/master/scripts/run.sh | bash

# Alternative: Install via pip with platform-specific optimizations
pip install videocaptioner

# macOS security: Remove quarantine attribute if app is blocked
xattr -d com.apple.quarantine $(which videocaptioner-gui) 2>/dev/null || true

# Launch the GUI
videocaptioner-gui

# Clone the repository
git clone https://github.com/WEIFENG2333/VideoCaptioner.git
cd VideoCaptioner

# Install uv if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Sync dependencies and create virtual environment
uv sync

# Run GUI
uv run videocaptioner

# Run CLI help
uv run videocaptioner --help

# Type checking
uv run pyright

# Run tests
uv run pytest tests/test_cli/ -q

# Lint with ruff
uv run ruff check videocaptioner

# Interactive configuration setup
videocaptioner config init

# Non-interactive setup with profile (for CI/automated environments)
videocaptioner config init --non-interactive --profile dubbing

# View current configuration
videocaptioner config show

# Get specific config value
videocaptioner config get llm.api_key

# Set configuration values
videocaptioner config set llm.api_key sk-your-api-key-here
videocaptioner config set llm.api_base https://api.openai.com/v1
videocaptioner config set llm.model gpt-4o-mini

# Find config file location
videocaptioner config path

# Diagnose environment and dependencies
videocaptioner doctor
videocaptioner doctor --json  # JSON output for scripting

# Config file: ~/.config/videocaptioner/config.toml

# LLM settings (for subtitle optimization and translation)
[llm]
api_key = "sk-your-api-key"
api_base = "https://api.openai.com/v1"
model = "gpt-4o-mini"

# Transcription settings
[transcribe]
asr = "bijian"  # Options: bijian, jianying, whisper-api, whisper-cpp, faster-whisper
language = "auto"  # Auto-detect or specify: zh, en, ja, etc.

# Subtitle processing
[subtitle]
optimize = true  # LLM-based text correction
split = true     # Semantic-based segmentation

# Translation settings
[translate]
service = "bing"  # Options: bing (free), google (free), llm (paid)

# Dubbing/TTS settings
[dubbing]
preset = "edge-cn-female"  # Edge TTS preset
api_key = ""  # For SiliconFlow/Gemini
voice = "xiaoxiao"
timing = "balanced"  # Options: strict, balanced, natural, none
audio_mode = "replace"  # Options: replace, mix, duck
tts_workers = 5  # Concurrent TTS workers

# Transcribe video using free Bijian ASR (Chinese/English)
videocaptioner transcribe video.mp4 --asr bijian

# Transcribe with explicit language
videocaptioner transcribe video.mp4 --asr bijian --language zh
videocaptioner transcribe video.mp4 --asr bijian --language en

# Transcribe audio files directly
videocaptioner transcribe audio.mp3 --asr bijian

# Output to custom path
videocaptioner transcribe video.mp4 --asr bijian -o output_dir/

# Generate word-level timestamps (for advanced processing)
videocaptioner transcribe video.mp4 --asr bijian --word-timestamps

# Output in different formats
videocaptioner transcribe video.mp4 --asr bijian --format json
videocaptioner transcribe video.mp4 --asr bijian --format ass  # Advanced Subtitles

# Translate subtitle to English using free Bing translation
videocaptioner subtitle input.srt --translator bing --target-language en

# Translate to Japanese
videocaptioner subtitle input.srt --translator bing --target-language ja

# Translate to Korean
videocaptioner subtitle input.srt --translator bing --target-language ko

# Translate to Spanish
videocaptioner subtitle input.srt --translator bing --target-language es

# Use Google translation (also free)
videocaptioner subtitle input.srt --translator google --target-language fr

# Create bilingual subtitles (Chinese source + English target)
videocaptioner subtitle input.srt --translator bing --target-language en --layout target-above

# Optimize without translation (LLM-based correction)
videocaptioner subtitle input.srt --no-translate --api-key $OPENAI_API_KEY

# Skip optimization, only translate
videocaptioner subtitle input.srt --translator bing --target-language en --no-optimize

# List of common language codes:
# zh-Hans (Simplified Chinese), zh-Hant (Traditional Chinese)
# en (English), ja (Japanese), ko (Korean)
# fr (French), de (German), es (Spanish)
# ru (Russian), ar (Arabic), pt (Portuguese)

# Configure LLM API (OpenAI or compatible)
videocaptioner config set llm.api_key $OPENAI_API_KEY
videocaptioner config set llm.api_base https://api.openai.com/v1
videocaptioner config set llm.model gpt-4o-mini

# Or use environment variables
export OPENAI_API_KEY="sk-your-key"
export OPENAI_BASE_URL="https://api.openai.com/v1"
export OPENAI_MODEL="gpt-4o-mini"

# Optimize subtitles with LLM (correct ASR errors)
videocaptioner subtitle input.srt --api-key $OPENAI_API_KEY

# Translate with LLM (higher quality, context-aware)
videocaptioner subtitle input.srt --translator llm --target-language en --api-key $OPENAI_API_KEY

# Enable reflection-based translation (higher quality, slower)
videocaptioner subtitle input.srt --translator llm --target-language en --reflect --api-key $OPENAI_API_KEY

# Custom prompt for optimization
videocaptioner subtitle input.srt --prompt "Make subtitles more formal and professional" --api-key $OPENAI_API_KEY

# Alternative LLM providers (OpenAI-compatible):
# - SiliconCloud: https://cloud.siliconflow.cn
# - DeepSeek: https://platform.deepseek.com
# - Local: Ollama with OpenAI compatibility layer

# Burn subtitles as hard-coded (visible in video)
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard

# Add removable soft subtitles (track embedded)
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode soft

# High quality output
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard --quality ultra

# Use preset style (anime style)
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard --style anime

# Custom subtitle styling via JSON override
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard \
  --style-override '{"outline_color": "#ff0000", "font_size": 48}'

# Rounded background mode (modern look)
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard --render-mode rounded

# Custom rounded style with white text and semi-transparent background
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard \
  --render-mode rounded \
  --style-override '{"text_color": "#ffffff", "bg_color": "#ff000099", "corner_radius": 12}'

# View all available styles
videocaptioner style

# Output to custom path
videocaptioner synthesize video.mp4 -s subtitle.srt --subtitle-mode hard -o output.mp4

# Generate audio using Edge TTS (free, no API key)
videocaptioner dub subtitle.srt --preset edge-cn-female -o dub.wav

# Chinese female voice (Edge TTS)
videocaptioner dub input.srt --preset edge-cn-female -o output.wav

# English friendly voice (Gemini, requires API key)
videocaptioner dub input.srt --preset gemini-en-friendly \
  --tts-api-key $VIDEOCAPTIONER_TTS_API_KEY -o output.wav

# SiliconFlow CosyVoice2
videocaptioner dub input.srt --preset siliconflow-cn-female \
  --tts-api-key $VIDEOCAPTIONER_TTS_API_KEY -o output.wav

# Multi-speaker with voice mapping
videocaptioner dub input.srt --video video.mp4 \
  --speaker-voice Alice=anna \
  --speaker-voice Bob=benjamin \
  -o video_dubbed.mp4

# Speaker syntax in subtitle file:
# [Alice] Hello, this is Alice speaking.
# Bob: This line uses another voice.

# Voice cloning with SiliconFlow
videocaptioner dub input.srt \
  --speaker-clone Alice=reference_audio.mp3|This is the reference text \
  --tts-api-key $VIDEOCAPTIONER_TTS_API_KEY \
  -o output.mp4

# Timing strategies
videocaptioner dub input.srt --preset edge-cn-female --timing strict -o output.wav  # Match subtitle timing
videocaptioner dub input.srt --preset edge-cn-female --timing natural -o output.wav  # Natural speech speed

# Audio output modes when embedding in video
videocaptioner dub input.srt --video video.mp4 --audio-mode replace -o output.mp4  # Replace original audio
videocaptioner dub input.srt --video video.mp4 --audio-mode mix -o output.mp4      # Mix with original
videocaptioner dub input.srt --video video.mp4 --audio-mode duck -o output.mp4      # Lower original as background

# Full pipeline with free services (ASR + Translation)
videocaptioner process video.mp4 --asr bijian --translator bing --target-language en

# Full pipeline including dubbing (generate dubbed video)
videocaptioner process video.mp4 --asr bijian --translator bing --target-language ja --dub-only

# Full pipeline with Edge TTS dubbing
videocaptioner process video.mp4 \
  --asr bijian \
  --translator bing \
  --target-language zh-Hans \
  --dub-only \
  --timing strict

# Chinese video → English dubbed video with Gemini TTS
videocaptioner process video.mp4 \
  --translator bing \
  --target-language en \
  --dub-only \
  --preset gemini-en-friendly \
  --tts-api-key $VIDEOCAPTIONER_TTS_API_KEY

# Full pipeline with LLM optimization
videocaptioner process video.mp4 \
  --asr whisper-api \
  --whisper-api-key $WHISPER_API_KEY \
  --translator llm \
  --target-language fr \
  --api-key $OPENAI_API_KEY \
  -v  # Verbose output

# Output to custom directory
videocaptioner process video.mp4 --asr bijian --translator bing --to en -o ./output/

# Quiet mode for scripting
videocaptioner process video.mp4 --asr bijian --translator bing --to en -q

# Download from YouTube
videocaptioner download "https://youtube.com/watch?v=xxx"

# Download to specific directory
videocaptioner download "https://youtube.com/watch?v=xxx" -o ./downloads/

# Download from Bilibili
videocaptioner download "https://bilibili.com/video/BVxxxxx"

# Then process the downloaded video
videocaptioner process downloaded_video.mp4 --asr bijian --translator bing --to en

Launch Methods:
- videocaptioner (opens GUI by default)
- videocaptioner gui (explicit GUI launch)
- videocaptioner-gui (GUI-only command)

GUI Features:
- Drag-and-drop video files
- Visual progress bars for each processing stage
- Configurable ASR/translation/TTS settings
- Built-in subtitle editor
- Preview rendered subtitles
- Batch processing queue
- Project management

Settings Location in GUI:
- File: Settings → API Configuration
- Or use CLI: videocaptioner config init

Supported File Formats:
- Video: MP4, MKV, AVI, MOV, WEBM
- Audio: MP3, WAV, FLAC, M4A
- Subtitle: SRT, VTT, ASS, SUB

# Run diagnostics
videocaptioner doctor
videocaptioner doctor --json  # For scripting/CI

# Check if FFmpeg is properly installed
which ffmpeg
ffmpeg -version

# Test API connectivity (OpenAI)
curl -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/models

# View verbose logs
videocaptioner transcribe video.mp4 --asr bijian -v

# macOS security issues
xattr -d com.apple.quarantine /path/to/videocaptioner

# Port already in use (GUI)
videocaptioner config set server.port 8889

# Disk space for Whisper models
df -h  # Check available space

# Clear cache if models are corrupted
rm -rf ~/.cache/whisper

# Python version issues
python --version  # Must be 3.10-3.12
python3.11 --version  # Try specific version

# Virtual environment conflicts
which python
which pip
python -m pip list | grep videocaptioner

# ASR speed comparison (fastest to slowest):
# Bijian (cloud) - Fast, free, Chinese/English only
# Jianying (cloud) - Fast, free, Chinese/English only  
# Whisper API - Requires payment, all languages
# Faster-Whisper (local) - GPU accelerated, requires VRAM
# Whisper-CPP (local) - CPU friendly, slower

# GPU acceleration for local Whisper
# NVIDIA GPU users:
nvidia-smi  # Verify GPU detection
# Install CUDA-enabled pytorch for faster processing

# Adjust TTS concurrency for faster dubbing
videocaptioner config set dubbing.tts_workers 10  # Default is 5

# Use cost-effective LLM models
videocaptioner config set llm.model gpt-4o-mini  # Cheaper than gpt-4

# Batch processing (process multiple videos)
for video in *.mp4; do
  videocaptioner process "$video" --asr bijian --translator bing --to en -q
done

# Parallel processing with xargs
ls *.mp4 | xargs -P 4 -I {} videocaptioner process {} --asr bijian --translator bing --to en -q

# LLM Configuration
export OPENAI_API_KEY="sk-your-key"
export OPENAI_BASE_URL="https://api.openai.com/v1"
export OPENAI_MODEL="gpt-4o-mini"

# TTS/Dubbing Configuration  
export VIDEOCAPTIONER_DUB_PRESET="edge-cn-female"
export VIDEOCAPTIONER_TTS_API_KEY="your-tts-api-key"
export VIDEOCAPTIONER_TTS_API_BASE="https://api.example.com"
export VIDEOCAPTIONER_TTS_MODEL="cosyvoice2"
export VIDEOCAPTIONER_TTS_VOICE="xiaoxiao"
export VIDEOCAPTIONER_TTS_WORKERS=5
export VIDEOCAPTIONER_DUB_TIMING="balanced"
export VIDEOCAPTIONER_DUB_AUDIO_MODE="replace"
export VIDEOCAPTIONER_TTS_MAX_SPEED=1.5
export VIDEOCAPTIONER_TTS_REWRITE_TOO_LONG=true

# Example: Run with all env vars
export OPENAI_API_KEY="sk-xxx"
export VIDEOCAPTIONER_TTS_API_KEY="your-key"
videocaptioner process video.mp4 --asr bijian --translator bing --to en

ASR (Speech Recognition) Providers:
- Bijian (必剪): FREE, cloud-based, Chinese/English, no API key
- Jianying (剪映): FREE, cloud-based, Chinese/English, no API key
- Whisper API: Paid (OpenAI), all languages, highest accuracy
- Faster-Whisper: Local models, offline, requires GPU for speed
- Whisper-CPP: Local models, CPU-friendly, slower

Translation Providers:
- Bing: FREE, good quality, all languages
- Google: FREE, good quality, all languages
- LLM: Paid (OpenAI-compatible), contextual, highest quality

TTS (Text-to-Speech) Providers:
- Edge TTS: FREE, Microsoft Azure, good quality voices
- SiliconFlow CosyVoice2: Paid, Chinese-focused, voice cloning
- Gemini TTS: Paid, Google, natural sounding
- Coqui TTS: Self-hosted, open-source alternatives

LLM Providers (for optimization/translation):
- OpenAI: gpt-4, gpt-4o-mini, gpt-3.5-turbo
- DeepSeek: deepseek-chat (cost-effective)
- SiliconCloud: Multiple model options
- Local: Ollama, LM Studio with OpenAI compatibility

Tech Stack:
- Core: Python 3.10+ with type hints
- CLI: Click-based command-line interface
- GUI: PyQt5 + PyQt-Fluent-Widgets
- Audio Processing: pydub (FFmpeg wrapper)
- Video Processing: FFmpeg/FFprobe
- TTS: edge-tts, SiliconFlow API, Gemini API
- LLM: openai SDK (100% compatible with OpenAI API)
- Download: yt-dlp (supports 1000+ sites)
- Config: TOML files with platformdirs
- Package Manager: uv (modern, fast)
- Build: hatchling with hatch-vcs
- Testing: pytest with markers for integration/LLM tests
- Linting: Ruff + Pyright

Project Structure:
videocaptioner/
  ├── cli/          # Command-line interface
  ├── ui/           # PyQt5 GUI
  ├── core/         # Processing logic
  ├── asr/          # Speech recognition
  ├── subtitle/     # Subtitle processing
  ├── translate/    # Translation services
  ├── dubbing/      # TTS and voice generation
  ├── resource/     # Styles, translations, assets
  └── tests/        # Unit and integration tests

Key Dependencies:
- requests: HTTP client
- openai: LLM integration
- yt-dlp: Video downloading
- pydub: Audio manipulation
- PyQt5: GUI framework
- platformdirs: Cross-platform paths
- tenacity: Retry logic
- pillow: Image processing
- fonttools: Font manipulation

Official Resources:
- GitHub: https://github.com/WEIFENG2333/VideoCaptioner
- Documentation: https://weifeng2333.github.io/VideoCaptioner/
- Online Demo: https://www.videocaptioner.cn
- Releases: https://github.com/WEIFENG2333/VideoCaptioner/releases

Related Technologies:
- Whisper (OpenAI): https://github.com/openai/whisper
- Faster-Whisper: https://github.com/guillaumekln/faster-whisper
- Edge TTS: Microsoft Azure TTS
- yt-dlp: https://github.com/yt-dlp/yt-dlp
- PyQt5: https://www.riverbankcomputing.com/software/pyqt
- FFmpeg: https://ffmpeg.org

Community:
- Issues: https://github.com/WEIFENG2333/VideoCaptioner/issues
- Chinese Community: QQ Group (see GitHub README)
- License: GPL-3.0