Skip to content

CHANGELOG.md

Latest commit

 

History

History
441 lines (312 loc) · 61.4 KB

CHANGELOG.md

File metadata and controls

441 lines (312 loc) · 61.4 KB
Raw

Changelog

All notable changes to franken_whisper are documented in this file.

franken_whisper is an agent-first Rust ASR orchestration stack with adaptive Bayesian backend routing, real-time NDJSON streaming, speculative cancel-correct transcription, and SQLite-backed persistence. It wraps whisper.cpp, insanely-fast-whisper, and whisper-diarization behind a unified 10-stage composable pipeline, then adds in-process Rust native engines under a staged conformance-gated rollout.

Format follows Keep a Changelog. Commit links point to the canonical GitHub repository.

[0.5.0] - 2026-07-11

CPU int8 encoder + SDPA poly-exp — a measured, quality-gated CPU speedup on x86-64

This is a performance release for the CPU native engine (x86-64 AVX2/FMA — AMD Zen/Threadripper, Intel Haswell and newer). Where v0.4.0 moved the encoder onto the GPU for Apple Silicon, v0.5.0 makes the CPU encoder substantially faster while holding output quality to a measured zero-WER-Δ budget. Every lever below was kept only on a measured win — verified byte-identical where it is byte-exact, and WER-neutral where it changes numerics — and gated on the candidate median against a paired null (A/A) control, not a single before/after pair. Rejected levers are recorded with their null-control and a retry-condition in docs/NEGATIVE_EVIDENCE.md.

Quality-safe int8 encoder (calibrated, policy-gated)

  • The encoder attention-output GEMM now runs int8×int32 under a calibrated per-model quality policy (035e83b, a997f37) — engaged when the model's calibration certifies it within a WER-Δ budget of 0.0 and a quantization rel-RMSE budget of 0.09; FW_ENC_ATTN_OUT_I8I32=0 is the kill switch, =1 forces it. Measured 1.47× encoder vs the f32 path, WER-neutral.
  • Fusing the SDPA gather into the int8 QKV GEMM (3293b47) writes q/k/v head-major straight out of the maddubs GEMM, skipping the standalone gather → 1.47→1.67× vs f32, byte-exact.
  • Register-blocked int8 microkernels: M4×N2 attn.out i8 GEMM (b6c3028, 165→129 ms), the M2×N4 maddubs tile (40fc09d), a 2-token column tile dot_i8_2col (85776f4, 1.15–1.19× @tq=64), AVX2 round-half-away activation-quantize (0ce9f64), and eliding the i7 output zero-fill (db3272f).
  • More aggressive full-int8 paths stay opt-in: q/k/v+fc1 int8 (FW_ENC_INT8_ATTN_IN, ~1.23× encoder, proper-noun-safe, e36fec2) and fc1-only int8 (FW_ENC_INT8_FC1, ~1.10×, byte-identical, 5481d46).
  • The quality-safe policy is WER-gated by a conformance test (9fcedac).

SDPA softmax poly-exp (large-v3-turbo)

  • A degree-5 AVX2/FMA polynomial exp replaces libm in the fused SDPA softmax for large-v3-turbo (94714c1, 5935d68) — default-on for turbo only (tiny.en is uncertified and stays off; FW_SDPA_POLY_EXP=0 kills it, FT_SDPA_POLY_EXP=1 forces it for a certified fine-tune). Measured 1.0722× e2e (cv 0.8%, 5/5 paired), transcript byte-identical on jfk ×1/×3/×8, WER Δ 0.000 vs whisper.cpp. Evidence: docs/PROPOSAL_ft_sdpa_poly_exp_default_on.md.

Byte-exact encoder fusions

  • Fuse the f32 mlp_fc bias into the GELU pass (f06543d, 1.04–1.06× encoder), fold the f32 mlp_proj bias into the residual add (5cb3cac, ~1.01×), fuse MLP GELU into the fc2 int8 activation-quantize (ede9f15), an M2 activation-column tile for the row-morsel f16 batch GEMV (ce43019, byte-exact 1.26×), and head-major i7 QKV row scheduling (50cbd65). All byte-identical to the prior output.

Honesty & correctness

  • Native engines now report probed capabilities, not declared ones (e782733) — reported feature availability reflects what the running binary actually supports.
  • The HuggingFace-token requirement now fires only when the insanely-fast bridge actually diarizes (84afe64, bd-0522) — the native path no longer demands a token it never uses.

Measurement methodology (why these numbers are trustworthy)

  • Median-vs-paired-null gate — every ratio is the candidate median compared against a same-binary A/A null control (ABBA-interleaved), so host contention and order bias cannot masquerade as a win.
  • Byte/ULP-exact verification — byte-exact levers are asserted bit-identical against the reference path; the one numerics-affecting lever (poly-exp) is held to a measured WER-Δ of 0.000 vs whisper.cpp.
  • Negative-evidence ledger — rejected levers are logged in docs/NEGATIVE_EVIDENCE.md with a reject-id, the null-control, and a retry-condition, so a dead end stays dead.

Evidence sources: docs/PERF_LEDGER.md, docs/NEGATIVE_EVIDENCE.md, docs/PROPOSAL_ft_sdpa_poly_exp_default_on.md, docs/cc_lane_finalization.md, and the per-lever commit messages linked above. CPU measurements on an AMD Threadripper PRO 5975WX (32 physical cores, release-perf, target-cpu=x86-64-v3) unless noted; Apple-Silicon GPU figures are in the v0.4.0 entry.

Compare: v0.4.0...v0.5.0

[0.4.0] - 2026-07-03

Fused GPU encoder — the whole transformer stack on the GPU (Apple Silicon)

  • On Apple Silicon with a large model the entire encoder now runs on the GPU, replacing v0.3.0's per-matmul offload. A new ft-kernel-metal::fused module keeps activations resident on the GPU (GpuTensor) and encodes each layer's ops — layernorm, q/k/v projections, multi-head attention, output projection, residual, MLP (fc → GELU → proj), residual — into one command buffer with a single CPU↔GPU sync per layer, instead of the CPU running layernorm/attention/GELU and blocking on the GPU for every matmul. Weights are uploaded once and cached per model. Enabled by default for n_state ≥ 1024 (medium/large; tiny/base/small keep the CPU path); FRANKEN_WHISPER_GPU=0 forces CPU, FRANKEN_WHISPER_FUSED_ENC=0 falls back to the v0.3.0 GEMM-only offload.
  • Measured (M4 Pro, 120s clip, large-v3-turbo): fused encoder 29.5s vs 35.7s GEMM-only vs 57.0s CPU — 48% faster than CPU and 17% faster than the v0.3.0 GEMM-only path. Killing the per-op ping-pong is the win.
  • Correctness: the GPU encoder tracks the CPU encoder closely (jfk transcript identical; on long audio the transcript is valid-but-not-bit-identical, like any GPU backend). A subtle bug was fixed along the way: MSL tanh overflows to NaN for large arguments, so the GPU GELU now uses ggml's GGML_GELU_FP16 clamp (x≥10→x, x≤−10→0), matching the CPU exactly. All Metal unsafe stays in ft-kernel-metal, so franken_whisper keeps #![deny(unsafe_code)]; non-macOS builds are unaffected.
  • Follow-up: the attention kernels are still naive (un-tiled); a tiled/flash-attention kernel would widen the win further.

[0.3.0] - 2026-07-02

GPU acceleration — automatic, hardware-selected (Apple Silicon)

  • The native engine now auto-offloads its large matmuls to the Metal GPU on Apple Silicon, enabled by default. A new ft-kernel-metal crate (sibling to ft-kernel-cpu) provides a shared-memory + register-blocked tiled f32 Metal GEMM (~1.5–1.9 TFLOP/s on an M4 Pro, ~5–8× the naive kernel); all unsafe Metal FFI is contained there so franken_whisper keeps #![deny(unsafe_code)]. At runtime nn::matmul_bias routes a GEMM to the GPU when (1) the target is macOS, (2) a Metal device is present, and (3) the matmul is large enough (m·k·n ≥ 2e9) that the compute win beats the launch/sync overhead — otherwise it stays on the already-fast multi-threaded CPU kernel. Auto-selection by hardware + workload: Apple Silicon large models → Metal GPU; x86-64-v3 (incl. AMD Threadripper) → the optimized AVX2/FMA CPU path; everything else → portable CPU. No config required. Override with FRANKEN_WHISPER_GPU=0 to force CPU.
  • Measured (M4 Pro): large-v3-turbo transcribe ~34% faster wall-clock (offloading ~3× the CPU compute), output byte-identical to the CPU path (transcription-level conformance holds). Small models (e.g. tiny.en) are unchanged — their GEMMs stay on the CPU, so there is no regression, and non-macOS builds are entirely unaffected (the Metal dep is target-gated out). This is a GEMM-only first cut; a batched/overlapped GPU pipeline (and an f16 path) would widen the win.

Everything in v0.2.1 (native-default engine, NaN-hardening, the aarch64 target-cpu codegen fix, the routing fallback fix, and the CPU int8/GELU perf work) is included.

Unreleased (since v0.1.0)

152 commits since v0.1.0

Compare: v0.1.0...main

Backend routing reliability (post-2026-07-01)

  • Fix: a degenerate native-only decision audit no longer panics transcription (7ade440) — on an install whose only usable backend is the in-process native engine (no external whisper.cpp / insanely-fast / whisper-diarization bridge), the adaptive router computes observed_state == 2 and franken-decision returns an internally-inconsistent audit whose to_evidence_ledger() then .expect()-panicked (PosteriorNotNormalized), aborting every transcription on the common native-only path (repro: youtube/transcribe with backend=auto). The routing decision now guards the evidence-ledger conversion — the ledger is a diagnostic, not the result: it skips a degenerate/unnormalized-posterior audit, with a catch_unwind safety net, degrading to no evidence entry and recording an evidence_ledger_conversion_failed router event, mirroring the existing "diagnostics never kill transcription" guards. Verified: native-only transcribe returns real timestamped segments with zero panics on Linux, macOS, and Windows. The root cause is also fixed upstream in franken-decision (its to_evidence_ledger now sanitizes to the ledger invariants instead of .expect()-ing; asupersync 43d46846).

YouTube Ingestion (franken_whisper youtube) (post-2026-06-06)

  • New youtube subcommand: download YouTube audio and transcribe it into deep-linked markdown + JSON (bd-s63n; epic bd-27v1) — a single command takes video URLs, playlist URLs (auto-expanded), and/or a --batch-file, downloads each via yt-dlp, transcribes it through the same engine as transcribe, and writes a {upload_date} - {title} [{id}] markdown/JSON pair (plus kept audio and a .fw_youtube_manifest.json state file) into youtube_transcripts/. The run is idempotent and resumable — re-runs skip videos already done, retry failed ones unless --no-retry, and Ctrl+C cancels cleanly (kills yt-dlp, aborts transcription, leaves the manifest honest). Pairs naturally with the in-process native engine: no cloud, no API keys. Verified end-to-end against real YouTube (watch?v=jNQXAC9IVRw --model tiny.en → 2005-04-24 - Me at the zoo [jNQXAC9IVRw].md in ~0.5 s).
    • b232a52 — wave A: yt-dlp orchestration (resolution/version-probe with a 90-day staleness warning and a FRANKEN_WHISPER_YTDLP_BIN override), collision-proof filename sanitizer (UTF-8-preserving, path-hostile-char folding, id suffix never truncated), and the markdown + JSON renderers (H1 title, metadata line, source link, honesty note, pause/speaker-grouped paragraphs with youtu.be deep-linked timestamps; structured video/run/utterances JSON).
    • d965a6f — wave B/1: cancel-correct ingestion pipeline + manifest state machine (per-video discovered/done/failed states, concurrency-bounded downloads, resumable re-runs, cooperative cancellation).
    • 85b14f9 — wave B/2: wired the franken_whisper youtube CLI subcommand (--url/positional URLs, --batch-file, --output-dir, --model, --language, --backend, --diarize, --concurrency, --no-keep-audio, --no-retry, --abort-on-error, --json-summary) plus a gated e2e integration test.
    • ed4f554 — live-test fix: switched the download format selector to the forgiving bestaudio/best (a strict selector + a stale yt-dlp failed during real-YouTube testing); YouTube breakage is a yt-dlp concern, kept current with yt-dlp -U.

Native Engine Performance Wave — Round 2: f16 decoder compute (post-2026-06-06)

  • f16-resident decoder compute is now the default production path (bd-2th6 Round 2) — a second profile-driven optimization round reopened the three frontiers left at Round-1 convergence (f16 weight traffic, the encoder sgemm, the ft-side microkernel) and harvested the one that paid off. The decoder now keeps its matmul weights f16-resident and dequantizes them with a vectorized NEON-fp16 slice path (convert_to_f32_slice → 8-lane dot8) instead of f32-everywhere, shaving −11.5 % off the large-v3-turbo e2e wall (min, interleaved A/B) with byte-exact golden transcripts on both models. Decoder floors drop to 10.3 ms/token (large, f16 ON) vs 38.9 ms/token (f16 OFF) and 5.2 ms/token (tiny). The encoder stays f32 — it is compute-bound GEMM where halving resident weight bytes buys no compute time (f16 panels measured pure overhead). Full pass-by-pass record, attribution table, and convergence statement: tests/artifacts/perf/20260605T0218Z-native-engine-baseline/RESULTS.md §6.
    • a236433 — criterion bench substrate (benches/native_engine_bench.rs: mel, encoder window, decoder token step, logits GEMV, e2e tiny) with saved baselines — the outstanding bd-2th6 deliverable.
    • 8abea12 — f16-resident decoder compute path with fused dequant-in-GEMV, shipped opt-in / default off: micro-bench win but an e2e regression (per-element widen serialized the FMA), kept as an env switch pending vectorization.
    • c703035 — vectorized f16 dequant (dequant 13.9 → 56.2 GB/s, isolated f16 GEMV 5.3×) → decoder f16 flipped to default ON; the env var FRANKEN_WHISPER_NATIVE_F16_COMPUTE becomes an opt-out kill switch. Encoder f16 panels prototyped then skipped with measured proof.
    • 0a5c939 — decoder token-step per-sub-part attribution (zero-overhead unless FRANKEN_WHISPER_PERF_SPANS=1) + size-gated logits GEMV widening (8→12 workers for the vocab product only; logits_gemv_large −1.9…−3.9 %, bit-identical). cross_attn f16 K/V and wider head-workers rejected with measured floor diagnostics.
    • franken-decision 0.3.2 migration + Round-2 landing (this commit) — migrated the Bayesian backend router to the franken-decision 0.3.2 API: evaluate() now returns Result<DecisionOutcome, ValidationError>, and a validation failure engages the existing deterministic static-priority fallback (evidence-ledgered with the error string + tracing::warn); DecisionContract::update_posterior now returns Result<(), UpdatePosteriorError>, and a failed update is logged + skipped so it can never kill a transcription (the uniform prior is the safe default). Adds tests that a ValidationError engages the static fallback and that posterior-update failures fail-closed without mutating. Also fixed stage_budget_timeout_maps_to_timeout_error_code to use a multi-worker runtime (asupersync 0.3.2 cannot advance the budget timer concurrently with a blocking spawn_blocking body on a current_thread runtime; the production run_stage_with_budget always runs on a multi-worker runtime). f16 default-ON golden gate byte-exact on both models; full lib suite 3086/3086 green.

Native Engine Performance Wave (post-2026-06-05)

  • Profile-driven optimization pass on the in-process Rust whisper engine (bd-2th6) — six measured levers land a 3.3× speedup on tiny.en (1.57 s → 0.475 s, RTF 0.043) and 4.6× on large-v3-turbo (44.6 s → 9.73 s, RTF 0.88) on this host (Apple M4 Pro, 14 cores, release-perf). Interleaved same-host hyperfine: native tiny.en is 2.33× faster than whisper-cli CPU (475 ms ± 13 vs 1105 ms ± 21); native large-v3-turbo reaches parity with whisper-cli CPU (9.731 s ± 0.272 vs 9.585 s ± 0.224) at lower user CPU (53.8 s vs 65.4 s). Both models are now faster than realtime. Each lever kept only on a measured win; bit-identical except DISC-004's scoped tail truncation.
    • 2ed6471 — version-tag SHA-256 hashed on a background thread at load, moving ~3.0 s off the run's critical path (tag value unchanged).
    • 0361bb2 — parallel f16→f32 dequant + tiled parallel weight transpose at load (model_weights span 2.4 s → 0.47 s).
    • bdbdd21 — language auto-detect reuses window-0's encode instead of a hidden duplicate encoder pass (−8.8 s on large).
    • ee36fd4 — parallelized the serial glue ops between fork-join matmuls (layer_norm / softmax / gelu / im2col / attention-head loops / logits GEMV / cross-K-V): encoder 8.3 → 5.8 s, tiny wall 0.92 → 0.56 s.
    • 0989a5a — tail-window encoder-context truncation (whisper.cpp audio_ctx-style; default-on, kill switch FRANKEN_WHISPER_NATIVE_TAIL_TRUNCATE=0): tail-window encode 4.2 s → 0.24 s (~94%). Main transcript byte-identical; divergence confined to spurious trailing-silence hallucinations on tail windows. See DISCREPANCIES.md DISC-004.
    • 5bb778b — decoder per-token path: KV buffer reuse, in-place logits band reads, hoisted window-constant cross K^T/V transforms, threaded QKV, head-parallel cross-attention (large 62.6 → 39.9 ms/tok, tiny 11.9 → 6.99 ms/tok; bit-identical, 4 new bitwise tests; ~490 MB cross-buffers dropped on large).
    • Evidence-backed abandons — fused QKV (bit-identical but ~16% slower), bmm-batched heads (~4% slower), parallel residual adds (slower), encoder scratch arena (wall/RSS-neutral; sgemm-compute-bound, ft calloc pages are lazy), and an ft-side matmul arena (~9% theoretical/large-window, a frankentorch change, below bar). Full arc + interleaved numbers: tests/artifacts/perf/20260605T0218Z-native-engine-baseline/RESULTS.md.
    • Follow-up (not applied): the release profile's opt-level = "z" costs ~26% on large vs release-perf; evaluate a dist profile change separately.

Real Native Whisper Engine on FrankenTorch Kernels (post-2026-06-04)

  • All three native "pilot" mocks replaced by a genuine in-process Rust ASR engine (bd-frp7 epic; engines bd-jryr / bd-s8w8 / bd-cidv; harness bd-4slu) — the former canned-phrase WhisperCppPilot / InsanelyFastPilot / DiarizationPilot are deleted. The native engine now parses whisper.cpp ggml .bin model files, computes the log-mel frontend, and runs the encoder/decoder transformer forward passes on ft-kernel-cpu (FrankenTorch) compute kernels, decoding tokens with whisper's timestamp rules — real speech recognition, no FFI, no network.
    • Honest availability (bd-jryr) — native_engine::native_model_available() header-sniffs a resolvable, well-formed ggml model (magic + supported dense ftype, no tensor load) instead of the previous dishonest always true constant. With no model present, the router stays bridge-only rather than advertising a fake native recovery path.
    • DTW word-level timestamps (bd-rjsx) — cross-attention weights of the model's alignment heads are recorded during decode and aligned via dynamic time warping to produce real per-word timings (DecodeOutput::word_timings), paid for only when word_timestamps is set.
    • Honest diarizer tagging (bd-cidv) — the native whisper-diarization path runs real ASR then the orchestrator's text/temporal heuristic diarizer, labeling segments SPEAKER_NN. Its raw_output declares "diarizer": "text-temporal-heuristic" (plus a diarizer_note) so no consumer mistakes it for neural diarization; the ECAPA upgrade remains tracked separately.
    • Rollout-machinery e2e proof (bd-4slu) — tests/native_engine_e2e.rs drives the full library dispatch (ingest → normalize → backend) through the CLI with FRANKEN_WHISPER_NATIVE_EXECUTION=1 and FRANKEN_WHISPER_NATIVE_ROLLOUT_STAGE=sole|primary, pointing every bridge binary at /nonexistent so a produced transcript can only have come from the native engine. Covers whisper-cpp, insanely-fast, and diarization native paths, plus the honest bridge-only-unavailable error path. All gated on the real tiny.en model (skip-not-fail when absent).
    • Bridge-vs-native conformance gate (bd-4slu) — a doubly-gated comparison (tiny.en model AND whisper-cli present) runs the reference bridge and the native engine on jfk.wav, asserting WER ≤ 0.10 and per-segment timestamps within a documented native-rollout tolerance profile (0.3 s, not the canonical 50 ms) to absorb greedy-vs-beam divergence. See DISCREPANCIES.md DISC-003.
    • Gated model fetch — scripts/fetch_test_models.sh provisions ggml-tiny.en.bin (sha256-pinned, idempotent, --force) into ${FRANKEN_WHISPER_TEST_MODEL_DIR:-~/.cache/franken_whisper/test-models} and verifies the in-repo jfk.wav fixture, turning the otherwise-skipped native conformance/e2e suites into real runs.

Speculative Streaming: CLI Integration (post-2026-03-21)

  • --speculative flag wired through the CLI orchestrator (4f54fbd) — previously TranscribeArgs::to_request() rejected every speculative request with FW-INVALID-REQUEST and the documentation-vs-behavior gap was wide. The integration adds:
    • SpeculativeRequest to BackendParams (a serde-friendly mirror of the CLI knobs that the orchestrator converts to streaming::SpeculativeConfig at dispatch time, keeping model.rs free of any dependency on streaming.rs).
    • audio::slice_pcm_wav_to_temp_path() — in-memory hound slicer producing per-window WAVs from a normalized 16 kHz mono PCM source. Preserves source spec, clamps OOB ranges, rejects non-PCM input.
    • execute_backend_speculative() in the orchestrator, replacing the regular Backend stage when request.backend_params.speculative.is_some(). Drives SpeculativeStreamingPipeline::process_file_with_models() with a per-window runner that calls backend::execute() for the fast and quality lanes; forwards every transcript.partial / confirm / retract / correct / speculation_stats event into the run-wide NDJSON log.
  • Speculative dispatch hardening (2dbab6f) — three follow-up correctness fixes against the first cut:
    1. Sticky backend: the new backend::resolve_static_backend() walks the static priority list once and returns a single concrete BackendKind; the dispatch path forces that kind onto every per-window invocation so --backend auto runs are engine-consistent end to end instead of letting per-window Bayesian routing produce a mixed-engine transcript.
    2. Partial event preservation on failure: the inner Result is now returned wrapped in an always-Ok outer carrier so the outer match can forward emitted_events whether the speculation pipeline succeeded or failed mid-window. The backend.error event also gains a windows_processed_before_failure field.
    3. Populated ReplayEnvelope for speculative runs: inter.backend_runtime = Some(backend::runtime_metadata(resolved)) and inter.backend_output_sha256 = sha256_segments(&merged), so speculative runs land replay envelopes that downstream drift detection / conformance tooling can meaningfully diff.

Storage: Concurrent-Persist Durability Contract (post-2026-03-21)

  • storage::tests::concurrent_persist_10_threads_with_segments_and_events tightened to require 10/10 successful persists (eede00a) — the test was previously written defensively against an older fsqlite MVCC limitation that silently dropped some concurrent writes; its assertions accepted "majority survives" (>= 5 of 10) as success. The underlying MVCC concurrent-persistence gap has since been resolved upstream in fsqlite-mvcc, so the test now enforces full durability instead of legitimizing data loss. Per-thread false returns from the retry-exhaustion paths are now described as diagnostics (so the aggregate assertion can name which thread didn't land) rather than as accepted dropped writes.

Documentation Refresh (post-2026-03-21)

  • README comprehensive rewrite (9ddb9db) — full top-to-bottom rewrite (~4,300 lines) reflecting the current architecture and capabilities, with worked examples (Bayesian routing arithmetic, mu-law encoding, Brier-score decomposition), anatomy walkthroughs (routing decision, TTY audio session, conformance check, speculative window), operational topics (state directory layout, CLI exit codes, threat model, performance tuning, disk usage growth, extension guides, debugging recipes), a use-cases gallery, JSON schema reference, and embedding-in-other-languages patterns.
  • Removed stale "frankensqlite MVCC limitation" Limitations entry from the README, since the underlying bug has been resolved upstream in the current /dp/frankensqlite checkout that the path dependency picks up.
  • README test count corrected from the marker-derived 3,860 to the runner-derived 3,668 (3,030 library + 638 integration across 26 integration test files).

Workspace Hygiene (post-2026-03-21)

  • Pre-existing clippy gate failures fixed (47fe23b) — src/backend/whisper_diarization.rs:311 and tests/conformance_harness.rs:696 ?-operator rewrites; tests/metamorphic_audio_tests.rs:35 unused generate_silence annotated #[allow(dead_code)]; tests/metamorphic_audio_tests.rs:73 repeat().take() modernized to repeat_n() per clippy::manual_repeat_n on Rust 1.82+.
  • Cargo.lock refresh (7855910) tracking upstream workspace bumps from sibling-agent commits that touched workspace crates while this branch was off the build path.

Release Pipeline and Distribution

  • dist.yml + release-automation.yml release pipeline — multi-platform binary builds (Linux x86_64/ARM64, macOS Intel/ARM64, Windows x86_64), tag-driven release automation that watches Cargo.toml and creates v{VERSION} tags when the version changes (d819076)
  • Dist asset names aligned with install.sh expectations (d70d7c3)
  • Test job's cargo fmt --check made advisory; toolchain channel matched against rust-toolchain.toml (4a406d0, 7cf1bfc)
  • curl|bash installer with SHA-256 checksum verification for one-line installs (e06d87f)
  • Installer strips v prefix from version in asset URLs (4cc9b66)
  • Installer handles versioned binary names in release archives (6a48022)

Dependency Migration to crates.io

  • asupersync, franken-kernel, franken-evidence, franken-decision migrated from local path dependencies to crates.io v0.3.0 — removes the requirement to clone sibling FrankenSuite repositories alongside franken_whisper for normal builds (d75c33f)
  • asupersync bumped to 0.3.1 (e8c1508)
  • Cargo.lock refreshed against upstream dependency bumps (da4125b)

Conformance: Native Engine Rollout Governance

  • Backend version drift detection, native-pilot fixture generator, and expanded invariant coverage — NativeEngineRolloutStage enum (Shadow → Validated → Fallback → Primary → Sole), shadow run comparator validating segment parity + replay envelope between bridge and native, scripts/gen_native_fixtures.py corpus generator, tests/COVERAGE.md mapping spec clauses to tests, and 30+ new conformance fixtures covering long segments, word-level boundaries, replay drift, timestamp edge cases, speaker label patterns, multilingual / code-switching / noisy-overlap corpora (c72eed3)
  • tests/COVERAGE.md tracking spec clause coverage (5a1ec0d)
  • Conformance fixtures expanded with very long segments + word-level boundary cases (1f16dc5) and timestamp / long segment edge cases (48078fe)

Pipeline: Word-Level Timestamps and Cancellation Threading

  • Word-level timestamp pipeline support (WordTimestampParams: enabled, max_len, token_threshold, token_sum_threshold) plus end-to-end cancellation token threading through pipeline stages so stage budgets are honored by the cancellation tokens, mid-pipeline cancellation no longer leaks partial subprocesses or partial transactions, and conformance assertions tighten around word-level segment invariants (f1cbd31)
  • Honor stage budgets in cancellation tokens (bd-xunn) (9e40cda)
  • Block speculative CLI flag where unsupported; stream pipe output reliably; bump deps (25ff052)

Adaptive Router: Calibration Observations

  • record_adaptive_router_prediction captures predicted probability at routing decision time so calibration observations (predicted vs. actual outcome) feed the Brier-score sliding window even when the router falls back to static priority (ef80eb8)
  • Calibration fallback path now records the adaptive prediction it would have made (bd-k87e), preventing fallback runs from silently corrupting calibration data (ed91ddd)
  • Non-finite calibration predictions sanitized (bd-bu04) (1c77e43)

Metamorphic Test Suites

  • tests/metamorphic_accelerate_tests.rs (+475 lines) — softmax / normalization / attention scoring invariants under permutation, scaling, and zero-padding transformations (47583d5)
  • tests/metamorphic_speculation_tests.rs (+481 lines) — string distance and window-invariant metamorphic properties (3c0c972)
  • tests/metamorphic_audio_tests.rs (+515 lines) — audio processing invariants (resampling commutativity, mono mixing properties, silence-padding equivalence) (78b9db6)

Audio Pipeline Hardening

  • Native audio backend expanded with format detection and sync improvements (1ad594e)
  • Audio frame mono averaging uses actual frame length rather than nominal channel count (ad9c51b)
  • normalize_cpu empty-input edge case guarded; two-lane streaming executor propagates thread panics rather than silently hanging (2d8457a)
  • Audio normalization + streaming I/O hardened with cancellation checks (59a33df, 13c8c80)

Robot Mode: Structured Error Envelopes and Diagnostics

  • Structured run_error envelope for invalid robot requests — malformed CLI args now produce a well-formed NDJSON error event instead of a human-readable stderr message (1e576a9)
  • Robot mode avoids human-decorated stderr (2a5982e)
  • Robot error emitted on worker thread panic (8ab2ef8)
  • Database health check hardened (robot health integrity probes resilient to migration / corruption edge cases) (83b23be, fe058e2)
  • ffmpeg health probe improved (092bc1e)

Sync Lock Robustness

  • Sync lock kept alive while owning PID runs (bd-qxf6); stale locks age-gated when PID unknown (bd-3jfj/bd-pvol); EPERM on kill -0 treated as alive (bd-aijz); Windows tasklist errors treated as unknown (bd-7xh8); Windows PID liveness check improved (bd-ht0i) (396c652, 1aa7cf4, eb0b9b8, 80d74db, afa5c16, 4e1ee33)
  • Sync lock parsing hardened; SQLite table identifiers quoted (da9a844)
  • Sync lock release path hardened alongside ffmpeg bundle work (bd-g7b2) (b36ef02)

ffmpeg Bundle and Stdin Hygiene

  • ffmpeg bundle scan hardened (bd-wk8p); stdin extension sanitization + ffmpeg download test stabilization (bd-o8fo) (79f3492, eb220c8)
  • tty-audio retransmit-plan protocol version corrected (6076902)
  • Legacy tty-audio version consistency enforced (bd-ithy) (036fbcf)

Secrets Redaction

  • HuggingFace token redacted in command logs via render_command_for_log() — diarization invocations no longer leak --hf-token values into tracing output (60bf952, 8b3249a)

Storage and Diarization Fixes

  • Storage concurrency tests stabilized (bd-pdyy) (f82031d)
  • Diarize clustering hardened (bd-0ydr) (3adb608)
  • Orchestrator UBS critical findings resolved (bd-9ftg) (ad4d98b)
  • SRT parsing fallback fixed (d409056)
  • Diarization transcript fallback and command pipe output loss fixed (a8328bc)
  • Empty parent components ignored when ensuring directories exist (0a21253)
  • GPU cancellation evidence ledger tests: missing expected_loss entries supplied (a20b828)
  • Speculation partials handled safely when missing (44729dc)

CLI and Robot Surface

  • Expanded CLI with structured subcommands and output modes (+122 lines) (ef221f7)
  • Expanded robot output with structured analysis views (+90 lines) (32ba54f)
  • routing_decision event builder extracted in robot module; backends output aligned with NDJSON schema (6beedc3)
  • CLI argument parsing edge case corrected (8a8a58e)
  • CLI integration tests and orchestrator refinement (a7cca9e)

Speculative Streaming

  • Adaptive window controller for speculative streaming pipeline — dynamically adjusts decode windows based on correction rates (61b7d40)
  • None-timestamp segment handling corrected in speculation and TUI; run detail load errors surfaced (dd04e9c)
  • Speculation test error assertions simplified with expect_err (33423d5)

Audio Processing

  • XDG_STATE_HOME support for tool state storage; ffmpeg extracted to tmpdir instead of cwd (10be321)
  • Active region boundaries clamped to actual audio duration, preventing out-of-bounds processing (2bca2f9)
  • Audio processing pipeline refinement (162da2a)

TTY Audio Transport

  • TTY audio demo script for interactive testing (6a5234d)
  • Extended TTY audio pipeline with telemetry tests (b8edfaf)
  • Frame validation extracted; sequence gaps rejected on finalize (f6f5f41)
  • TTY telemetry test assertions refined (ab2cf8e)

Storage and Persistence

  • Migration-safe column detection and defensive query wrappers (+35 lines) (23b5062)
  • Replay pack validation hardened alongside storage queries (5248ad5)
  • Silent unwrap_or_default replaced with explicit JSON parse error propagation (c2f5474)
  • SqliteValue::Text(String) and Blob(Vec<u8>) migrated to Arc-based types for reduced cloning overhead (4066d5d)
  • Storage query patterns simplified; dead code paths removed (64d340e, 42f718a)
  • Storage queries improved alongside TTY audio demo extension (527f443)

Sync Pipeline

  • Parallel sync pipeline with progress tracking (+44 lines) (1999137)
  • Sync timeout overflow fixed with saturating_mul; backend.ok preferred as rollout stage; atomic rename made portable (9f94946)
  • Off-by-one corrected in sync progress tracking (0105d9a)
  • Sync pipeline refined (6b5e29b)

Pipeline Orchestration

  • Backend module expansion with improved pipeline orchestration (d9166c3)
  • stage_start cleared on checkpoint failure to prevent elapsed time leaking across pipeline stages (b15578e)
  • Pipeline stage transitions refined (5f3f187)
  • 8 pipeline stage .expect() panics replaced with proper FwError returns (0c4d557)
  • Backend/storage pipeline streamlined for robustness (b19eec9)
  • Module streamlining (ca4f587)

Acceleration

  • Division by zero in layer_norm_cpu prevented by clamping epsilon floor (b3c8b56)

CI and Quality Gates

  • Persistent worker quarantine with TTL for quality-gate routing (9ecf72d)
  • flock guard and preblock for unreachable workers in quality-gate routing (77900fb)
  • Quality-gate retry routing hardened with worker quarantine and expanded retryable patterns (1ff4910)
  • Stale-sibling validation rejected when dependency sync is degraded (670a758)

Documentation

  • Major README refresh with updated architecture and feature documentation (+1,867 lines across three commits) (2a2bb9f, 7d207b8, ba227a6)
  • Robot event catalog updated with routing_decision, backends.discovery, and streaming events (ecf702b)
  • CHANGELOG.md rebuilt from git history with live commit links (d678a9e, 0059e01)
  • DISCREPANCIES.md tracking known native-vs-bridge divergences (introduced alongside conformance work in c72eed3)
  • Implementation tracker and execution packet documentation updated (b34893e)

v0.1.0 — 2026-03-04

Initial release. GitHub Release: v0.1.0 | Published: 2026-03-04

Lightweight tag on 6a51618. 59 commits from initial commit (2026-02-22) to tag. 81 files changed, +33,184 / -1,321 lines.

Compare: 2e9f2e9...v0.1.0

Release Artifacts

Platform Archive Size
Linux x86_64 franken_whisper-0.1.0-linux_amd64.tar.gz 2.3 MB
macOS arm64 (Apple Silicon) franken_whisper-0.1.0-darwin_arm64.tar.gz 2.1 MB
Windows x64 franken_whisper-0.1.0-windows_amd64.zip 2.2 MB

SHA-256 checksums: checksums-sha256.txt

Core Architecture

  • 10-stage composable pipeline: Ingest, Normalize, VAD, Source Separate, Backend, Accelerate, Align, Punctuate, Diarize, Persist — stages composed dynamically per-request, skipped when unnecessary, budgeted independently, profiled automatically (2e9f2e9)
  • Adaptive Bayesian backend routing with explicit loss matrix, posterior calibration, and deterministic fallback across whisper.cpp, insanely-fast-whisper, and whisper-diarization
  • Real-time NDJSON streaming with stable schema (v1.0.0) — every pipeline stage emits sequenced, timestamped events for agent consumption
  • Cooperative cancellation via CancellationToken-based graceful shutdown with resource cleanup
  • Per-stage timeout budgets with automatic latency profiling and tuning recommendations
  • 12 structured FW-* error codes (FW-IO, FW-CMD-TIMEOUT, FW-BACKEND-UNAVAILABLE, etc.) for machine-readable error handling
  • #![forbid(unsafe_code)] enforced — zero unsafe blocks across entire codebase
  • Rust 2024 edition, nightly toolchain, cargo clippy --all-targets -- -D warnings enforced

Audio Processing

  • Built-in Rust audio decoder via symphonia: MP3, AAC, FLAC, WAV, OGG, ALAC decoded natively with zero ffmpeg dependency for common formats (934a8f6)
  • Built-in Rust decoder made the primary normalization path; ffmpeg demoted to fallback (934a8f6)
  • Automatic ffmpeg provisioning with bridge-native recovery fallback and sync validation tests (ca25b05)
  • ffmpeg provisioning subsystem unit tests (e794156)
  • VAD redesign and audio pipeline hardening with streaming engine trait (80969cb)
  • Audio duration probing hardened for edge cases (109ebc1)
  • Input validation: reject directory paths with is_file guard (830a443)

Speculative Streaming

  • Speculative cancel-correct streaming pipeline with evidence ledger and file processing — dual-model fast+quality architecture (ad4b54a)
  • TUI speculation display and TTY transcript control frame support (847658d)
  • Conformance validation extended for speculation events (30044ab)
  • Comprehensive integration test suites for speculative pipeline (bc2a526)
  • Massive edge-case test expansion for speculative pipeline (a2d5234)

TTY Audio Transport

  • mulaw+zlib+base64 NDJSON protocol for low-bandwidth audio relay over PTY links with handshake, integrity checks, and deterministic retransmission
  • VAD/separate/punctuate/diarize pipeline stages wired into TTY session protocol with deep audit tests (f183d08)
  • ControlFrameType references updated to TtyControlFrame enum (d625089)

Storage and Persistence

  • SQLite-backed run history with JSONL export/import and SHA-256 replay envelopes
  • Rollback-safe legacy v1 → v2 runs schema migration (1424a8d)
  • acceleration_json column propagated through sync pipeline with fail-closed overwrite semantics (9c7df9d)
  • Cascade-aware overwrite tracking with overwrite-strict conflict policy for verified child-row replacement (4c7cc45, e4db512)
  • Migration error messages enriched with contextual details (07f24f6)
  • MVCC visibility retry in concurrent write test (4e713a2)
  • Incremental export corrected to use finished_at instead of started_at (8974203)
  • Massive test expansion covering storage, protocol, and streaming improvements (2418ea7)

Diarization

  • TinyDiarize support: whisper.cpp's built-in speaker-turn detection without HF token (7abe271)
  • dead_code annotations removed from pipeline stages alongside TinyDiarize addition (7abe271)
  • Speaker constraints integration tests (3412914)

Orchestration and Backend Routing

  • Parallel processing and retry logic for transcription orchestration (5c70d3d)
  • whisper_cpp backend hardening with expanded orchestrator capabilities and pipeline robustness (1f6f694)
  • Routing mode helper extracted; shutdown and storage resilience hardened; sync cursor upgraded to composite key (7cad3b5)
  • Segment timestamps sanitized at extraction boundary (bd99c6c)
  • Mutex poisoning handled gracefully in two-lane streaming executor (b1682e8)
  • Stdout/stderr pipe deadlock prevented by reading on dedicated threads (dd6cbb2)
  • Comprehensive test coverage for all backend engines and routing layer (2640d14)
  • Duration estimation and native engine edge-case tests (22c3a03)

Robot Mode

  • robot health subcommand for database diagnostics (aff4551)
  • Golden test output fixtures added (146d225)
  • Integration tests for robot subcommands (0433627)

Event and Observability

  • Event fingerprinting enriched with seq+stage tuples; ingest failure determinism test added (acbb240)
  • BetaPosterior::new input validation with assertions and tests (e4a898f)
  • Conformance validation and robot event contract extended for speculation events (30044ab)

Sync and Replay

  • Sync and replay_pack edge-case tests with schema validation (8d787f2)

Testing

  • 2,939 tests passing at release (now ~3,668 unit + integration tests at HEAD; see [Unreleased] → Documentation Refresh for the marker-vs-runner count correction)
  • Unit test coverage expanded across orchestrator, error, model, accelerate, robot, and process modules (1ed8429)
  • Conformance harness with 50ms cross-engine timestamp tolerance and drift detection

Infrastructure and Tooling

  • Boilerplate and dead_code annotations eliminated across cli, conformance, and orchestrator (e70b860)
  • rch quality-gate wrapper added (6f5f45e)
  • MCP agent mail configuration for Codex, Cursor, and Gemini (27b1a98)
  • GitHub social preview image (1280×640, 24px border) (098da6c)

Initial Commit

  • 2e9f2e9 — 2026-02-22 — franken_whisper ASR orchestration stack foundation