cto-legends · module specification · studio · v0.1.1
legends-captions: Contextual ASR Repair, Forced-Alignment Timing & Active-Word ASS Rendering
Deterministic video caption QA and ASS rendering CLI with contextual ASR repair, NeMo Forced Aligner word timing, spoken acronym locks, and proof frames.
GitHub Repository (https://github.com/avalonreset/legends-captions) · Handoff: cto-legends handoff legends-captions
legends-captions (catalog alias legends-ultimate-captions) is an agentic video caption QA, policy normalization, and ASS subtitle rendering system built in pure Python 3.11+ standard library with zero runtime package dependencies. Generic auto-caption tools treat raw ASR output as final copy—routinely mangling AI model names, technical acronyms, hardware identifiers, and rapid idiomatic speech—while leaving manual fixes trapped inside a single timeline edit. legends-captions treats raw ASR transcripts as fallible evidence, routing word tokens through a deterministic correction policy, forced-alignment timing authority, active-word ASS styling, and structured proof-frame QA.
Architecturally, the package separates speech capture/transcription (handled upstream by legends-ambient-intelligence, legends-yt-dlp, or hyperyap) from deterministic caption engineering. In src/legends_captions/policy.py, longest-match phrase rules and domain glossaries repair homophones (such as 'low raw' to 'LoRA' or 'exacting 2.0' to 'exciting as Seedance 2.0'), redistribute word timestamps proportionally across virtual replacement tokens, and detect spelled-out acronyms ('L O R A') so each letter retains independent active-word highlight timing. The ASS builder (src/legends_captions/ass_builder.py) and vertical 4K60 profile (configs/default_vertical_4k60.json) enforce PlayResX/Y 2160x3840 geometry, 0.09s highlight lead, hard-cut reset rules, and machine-readable policy manifests (legends_captions.policy_manifest.v1) so every correction becomes a regression test.
Deterministic Execution Boundary
Why Raw Frontier LLM Prompts Cannot Replace Deterministic Execution
Asking a raw LLM chat window to 'fix my SRT file' frequently breaks timestamp monotonicity, silently drops or shifts word-level timing spans, hallucinates words not spoken in the audio, and forgets domain corrections on the very next video. A production active-word caption pipeline requires deterministic token-span math, acoustic forced alignment against the final video clock, and verifiable frame-level geometry.
- Token-Span Timing Redistribution & Virtual Token Flags: When a 2-word ASR error ('exacting 2.0') expands into a 4-word repair ('exciting as Seedance 2.0'), distribute_timing() in policy.py deterministically subdivides the original [start, end] acoustic window across replacement tokens and tags them with 'virtual' or 'corrected' provenance flags.
- Per-Letter Spoken Acronym Highlight Preservation: Naive text cleanup collapses spelled-out acronyms ('L O R A') into a single word ('LoRA'), freezing the active-word karaoke highlight for four syllables. mark_spoken_acronyms() detects single-letter alpha runs against SPOKEN_ACRONYMS and preserves per-letter highlightable tokens with spoken_acronym_letter flags.
- Two-Tier Timing Authority (Parakeet ASR + NeMo Forced Aligner): Raw ASR word timestamps are treated strictly as planning data. When exact active-word highlight transitions and hard-cut boundaries matter, the architecture designates NVIDIA NeMo Forced Aligner (NFA) CTM word output as final timing authority with a 0.09s lead at 60 fps.
- Audit Manifests & Regression-Locked Revision Learning: Every normalization run emits a legends_captions.policy_manifest.v1 JSON receipt recording category, original span, replacement, reason, confidence, and token indices—turning operator revisions into unit-tested PHRASE_RULES and correction_glossary.json entries.
CLI Subcommands & Capability Surface
legends-captions doctor [--config <path>] — Environment & Media Toolchain Readiness Doctor: Inspects the local Python 3.11+ execution environment, resolves the active caption profile (defaulting to vertical_4k_60fps_active_word in configs/default_vertical_4k60.json), counts preserved .raw/sources lineage files, and verifies ffmpeg and ffprobe binary availability on PATH.legends-captions normalize [--uppercase] <text...> — Contextual ASR Phrase & Domain Glossary Normalization: Applies longest-first token matching across PHRASE_RULES and configs/correction_glossary.json to fix domain homophones ('low raw' -> 'LoRA', 'C dance 2.0' -> 'Seedance 2.0', 'RTX4090' -> 'RTX 4090'), grammar ('what is this unlock' -> 'what does this unlock'), and joined tokens ('fuckdown' -> 'fuck down') while preserving trailing punctuation.legends-captions normalize --manifest <text...> — Structured Correction Manifest Generation: Emits a deterministic legends_captions.policy_manifest.v1 JSON document containing UTC creation timestamps, normalized mixed-case text, uppercase display text, and an array of CorrectionDecision records with exact span_start/span_end token indices and confidence scores.mark_spoken_acronyms() via legends-captions normalize — Spoken Acronym Letter-Level Timing Engine: Scans token streams for consecutive single-letter tokens matching known technical acronyms (api, css, gpu, html, lora, nfa, seo, ui, ux, vram). Normalizes letter casing and attaches spoken_acronym_timing decisions so active-word renderers highlight each letter as the speaker spells it.legends-captions policy-report — Policy Rule Inventory & Confidence Inspection: Exports the full active catalog of contextual correction rules as structured JSON, detailing each rule's match tuple, replacement tuple, classification category (contextual_model_name, domain_term, grammar_repair, idiom_repair, token_split), confidence threshold, and documented rationale.legends_captions.ass_builder.build_ass_document() — Vertical 4K60 Active-Word ASS Document Builder: Generates v4.00+ SubStation Alpha (.ass) subtitle scripts configured for 2160x3840 vertical 4K60 canvases using the LegendsActive style (Montserrat Black 156pt, primary/outline color codes, centisecond ass_time formatting, and escaped dialogue events).
Raw Frontier LLM vs. legends-captions Execution Matrix
- Technical Homophones & Model Names: Raw LLM (Either leaves raw ASR errors ('low raw', 'exacting 2.0') untouched or rewrites sentences with unrequested paraphrasing.) vs. Module (Executes deterministic longest-match PHRASE_RULES and context-tagged glossary repairs with explicit confidence thresholds.) — legends-captions normalize --manifest outputs exact CorrectionDecision records verified by unit tests in tests/test_policy.py.
- Spoken Acronym Highlighting: Raw LLM (Collapses 'L O R A' into 'LoRA', destroying per-letter timing synchronization during spelled-out speech.) vs. Module (Preserves separate letter tokens ('L O R A') and tags each token with spoken_acronym_letter and acronym identity.) — Manifest inspection confirms category='spoken_acronym_timing' and confidence=0.99 across exact token spans.
- Word-Boundary Timing Authority: Raw LLM (Fabricates or corrupts SRT/ASS timestamps when inserting or splitting corrected words.) vs. Module (Proportionally redistributes acoustic intervals across virtual tokens and uses NeMo Forced Aligner CTM as final timing authority.) — Timing Authority Policy gates (0.09s highlight lead, 0.85s hold gap, no cut-crossing speech events, 3-frame Shorts tail).
- Cross-Run Revision Memory: Raw LLM (Fixes a typo in one chat session and repeats the exact same mistake on the next video.) vs. Module (Classifies every user fix into 10 buckets (lexical, casing, grammar, timing, grouping, style, qa, etc.) under Revision Learning Policy.) — Every automatable fix is committed to configs/correction_glossary.json, policy.py, and python -m unittest discover -s tests.
Explicit Safety Boundaries & Exclusions
Operator FAQ
- Does legends-captions bundle FFmpeg, PyTorch, or ASR model weights?
- No. The core Python package has zero runtime dependencies and uses only the Python 3.11+ standard library for policy normalization, ASS script generation, and manifest building. External tools such as ffmpeg, ffprobe, faster-whisper, or WSL NVIDIA NeMo/Parakeet (configured via LEGENDS_CAPTIONS_WSL_PYTHON) are orchestrated on your machine without bundling heavy weights.
- How does legends-captions handle spelled-out acronyms like 'L O R A' versus spoken words like 'LoRA'?
- When ASR hears the spoken word as 'low raw', 'low ra', or 'lora', phrase rules normalize it to the single token 'LoRA'. When the speaker spells 'L O R A' letter by letter, mark_spoken_acronyms() preserves four separate letter tokens and flags them with spoken_acronym_letter so active-word ASS rendering highlights each letter individually in sync with the voice.
- Can I generate styled subtitle files without burning them into a new MP4 video?
- Yes. You can run policy normalization, emit correction manifests, and build standalone .ass, .srt, or .vtt subtitle artifacts directly, invoking the FFmpeg burn-in and proof-frame extraction stage only when you want a rendered MP4 delivery file.
All 19 CTO Legends Packages (1 Router + 18 Modules)
cto-legends (v0.2.5) — Single-skill Python CLI router and isolated environment manager that resolves 18 SHA-256-pinned modules on demand without MCP startup bloat.legends-empire (v0.3.2) — SHA-256 Markdown knowledge foundation with recoverable NTFS and POSIX transactions, .raw/ evidence quarantine, and read-only vault stewardship.legends-hetzner (v0.1.3) — Hetzner Cloud CLI toolkit with Cost-Optimized CX/CAX stock radar, Fractional CTO sizing, SHA-256 server plans, 10-check preflight, and VPS sniping.legends-coolify (v0.2.1) — Read-only Coolify PaaS CLI for self-hosted app inventory, secret-free env key inspection, backup schedule audits, deployment history, and runbooks.legends-cloudflare (v0.1.1) — Zero-dependency Cloudflare DNS CLI enforcing SHA-256 preview-first single-record upserts, exact-ID deletes, setting preservation, and API readback.legends-astro (v0.2.1) — Astro v7 migration CLI that extracts HTML and WordPress source contracts, scaffolds Content Layer collections, and verifies dist/ build parity.legends-gohighlevel (v0.3.0) — Python 3.11+ GoHighLevel CLI for API v2 CRM, Workflows v3, Agent Studio AI, and 5-layer agency RBAC/1.05x rebilling with SHA-256 preview and readback.legends-geogrid (v0.2.2) — Circular-lattice Google Maps rank tracker, GBP name-stuffing audit workbench, and 18-page vector PDF strategy studio for coding agents.legends-dataforseo (v0.1.6) — Zero-dependency Python API and CLI for DataForSEO v3 SERP, Maps, GBP, and keyword research with offline previews and evidence banking.legends-firecrawl (v0.3.2) — Agent-first CLI wrapper (lfc and lax) for official Firecrawl web extraction and Alexandria queries with full local JSON evidence banking.legends-github (v0.2.1) — Deterministic Python CLI for GitHub repository audits, DataForSEO keyword and SERP demand, 20-topic synthesis, non-destructive READMEs, and OG checks.legends-grant (v0.3.0) — Standard-library Python CLI for 56-jurisdiction US grant discovery, SQLite evidence banking, qualification checks, and proposal verification.legends-jev (v0.2.1) — Zero-dependency Python CLI for TypeSafe Jev 1.14 optical recognition (OCR), storefront signage checks, pixel bounding boxes, and typed UI decisions.legends-obs (v0.1.2) — Guarded Windows OBS Studio CLI over WebSocket v5 for hardware encoder planning, pre-mutation rollback snapshots, and FFprobe-verified recording canaries.legends-captions (v0.1.1) — Deterministic video caption QA and ASS rendering CLI with contextual ASR repair, NeMo Forced Aligner word timing, spoken acronym locks, and proof frames.legends-yt-dlp (v0.1.2) — Paced yt-dlp orchestration CLI with SHA-256 binary verification, channel item ledgers, opt-in Mullvad Lockdown guards, and Parakeet/NFA word-level search.legends-stable-audio-3 (v0.1.3) — Hardware-aware Stable Audio 3 CLI for VRAM-calibrated local Medium batches, guarded Large REST jobs, cue-trimmed streaming mixes, and float WAV mastering.legends-hyperyap (v0.1.1) — Offline desktop dictation runtime running INT8 ONNX NVIDIA Parakeet TDT 0.6B v2 locally with CUDA-first inference, active-window paste, and terminal hooks.legends-ambient-intelligence (v0.1.1) — Passive audio pipeline CLI for SHA-256 deduplicated USB recorder ingest, offline Whisper/NeMo Parakeet ASR, rollover recovery, and vault distillation.