From 82596187a42c157ba903f5784e1c43fe3ded2349 Mon Sep 17 00:00:00 2001 From: Dmitry Dementev Date: Sat, 25 Apr 2026 21:09:43 +0300 Subject: [PATCH] =?UTF-8?q?docs(superpowers):=20=D0=B4=D0=BE=D0=B1=D0=B0?= =?UTF-8?q?=D0=B2=D0=BB=D0=B5=D0=BD=D1=8B=20spec=20=D0=B8=20=D0=BF=D0=BB?= =?UTF-8?q?=D0=B0=D0=BD=20=D0=B4=D0=BB=D1=8F=20onnx-asr=20=D0=B1=D1=8D?= =?UTF-8?q?=D0=BA=D0=B5=D0=BD=D0=B4=D0=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Зачем: - задизайнить эксперимент с Parakeet/GigaAM через onnx-asr как третий бэкенд. - Что: - spec: архитектура, модель-алиасы (gigaam-v3, parakeet-v3), API, регистрация. - plan: 8 задач TDD: зависимость → скелет → ensure → create → transcribe → регистрация. - Проверка: - gh pr diff, ревью spec/plan. --- ...6-04-25-parakeet-backend-implementation.md | 824 ++++++++++++++++++ .../2026-04-25-parakeet-backend-design.md | 160 ++++ 2 files changed, 984 insertions(+) create mode 100644 docs/superpowers/plans/2026-04-25-parakeet-backend-implementation.md create mode 100644 docs/superpowers/specs/2026-04-25-parakeet-backend-design.md diff --git a/docs/superpowers/plans/2026-04-25-parakeet-backend-implementation.md b/docs/superpowers/plans/2026-04-25-parakeet-backend-implementation.md new file mode 100644 index 0000000..46932e0 --- /dev/null +++ b/docs/superpowers/plans/2026-04-25-parakeet-backend-implementation.md @@ -0,0 +1,824 @@ +# onnx-asr Backend Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add `onnx-asr` as a third pluggable backend (device `"onnx"`) supporting GigaAM v3 (Russian, 4.7% WER, 59x RTF) and Parakeet v3 (multilingual, 11% WER, 34x RTF). + +**Architecture:** Follow `OpenVINOBackend` pattern in `backends/openvino.py`. Implement the structural Backend protocol (`ensure_model_available`, `create_model`, `transcribe`). Register via `get_backend("onnx")` in `__init__.py`. Audio loading reuses `faster_whisper.decode_audio()`. VAD via Silero built into onnx-asr. + +**Tech Stack:** `onnx-asr>=0.11.0`, `onnxruntime` (transitive), `huggingface-hub` (transitive), existing `faster-whisper` for audio decode. + +**Refs:** Spec at `docs/superpowers/specs/2026-04-25-parakeet-backend-design.md`, Backend protocol at `src/local_transcriber/backends/base.py`, Pattern to follow at `src/local_transcriber/backends/openvino.py`, Types at `src/local_transcriber/types.py`. + +--- + +### Task 1: Add onnx-asr dependency + +**Files:** +- Modify: `pyproject.toml:7-15` + +- [ ] **Step 1: Add `onnx-asr[cpu,hub]` to dependencies** + +Open `pyproject.toml`. In the `dependencies` list, add `"onnx-asr[cpu,hub]>=0.11.0"`: + +```toml +dependencies = [ + "typer", + "rich", + "faster-whisper>=1.2.1", + "socksio>=1.0.0", + "nvidia-cublas-cu12>=12.4; sys_platform == 'linux' and platform_machine == 'x86_64'", + "openvino-genai>=2025.0; sys_platform != 'darwin' and (platform_machine == 'x86_64' or platform_machine == 'AMD64')", + "tomli>=2.0; python_version < '3.11'", + "onnx-asr[cpu,hub]>=0.11.0", +] +``` + +- [ ] **Step 2: Install the dependency** + +Run: `uv sync` +Expected: onnx-asr and onnxruntime installed without errors. + +- [ ] **Step 3: Verify import works** + +Run: `uv run python -c "import onnx_asr; print(onnx_asr.__version__)"` +Expected: prints version (e.g. `0.11.0`), no errors. + +- [ ] **Step 4: Commit** + +```bash +git add pyproject.toml uv.lock +git commit -m "build: add onnx-asr[cpu,hub]>=0.11.0 dependency" +``` + +--- + +### Task 2: Backend skeleton — model aliases and class stub + +**Files:** +- Create: `src/local_transcriber/backends/onnx_asr.py` +- Create: `tests/test_onnx_asr.py` +- Read: `src/local_transcriber/backends/openvino.py` (for pattern) +- Read: `src/local_transcriber/backends/base.py` (for protocol) +- Read: `src/local_transcriber/types.py` (for Segment, TranscribeResult) + +- [ ] **Step 1: Write the failing test for model alias resolution** + +Create `tests/test_onnx_asr.py`: + +```python +"""Tests for onnx-asr backend.""" + +import pytest +from local_transcriber.backends.onnx_asr import OnnxAsrBackend, MODEL_ALIASES + + +class TestModelAliases: + def test_gigaam_v3_resolves(self): + backend = OnnxAsrBackend() + result = backend._resolve_model("gigaam-v3") + assert result == "gigaam-v3-ctc" + + def test_parakeet_v3_resolves(self): + backend = OnnxAsrBackend() + result = backend._resolve_model("parakeet-v3") + assert result == "nemo-parakeet-tdt-0.6b-v3" + + def test_raw_name_passes_through(self): + backend = OnnxAsrBackend() + result = backend._resolve_model("nemo-canary-1b-v2") + assert result == "nemo-canary-1b-v2" + + def test_unknown_alias_raises(self): + backend = OnnxAsrBackend() + with pytest.raises(ValueError, match="Неподдерживаемая модель"): + backend._resolve_model("nonexistent-model") +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/test_onnx_asr.py -v` +Expected: FAIL — `OnnxAsrBackend` not defined. + +- [ ] **Step 3: Write minimal implementation** + +Create `src/local_transcriber/backends/onnx_asr.py`: + +```python +"""Бэкенд транскрипции на основе onnx-asr (GigaAM, Parakeet, FastConformer).""" + +from __future__ import annotations + +MODEL_ALIASES: dict[str, str] = { + "gigaam-v3": "gigaam-v3-ctc", + "parakeet-v3": "nemo-parakeet-tdt-0.6b-v3", +} + +SUPPORTED_ALIASES = ", ".join(MODEL_ALIASES) + + +class OnnxAsrBackend: + """Бэкенд транскрипции через onnx-asr (ONNX Runtime).""" + + def _resolve_model(self, model_name: str) -> str: + """Resolve alias to onnx-asr model name. Raw names pass through.""" + if model_name in MODEL_ALIASES: + return MODEL_ALIASES[model_name] + if "/" in model_name or "-" in model_name: + # Looks like a raw onnx-asr name — allow passthrough + return model_name + raise ValueError( + f"Неподдерживаемая модель '{model_name}'. " + f"Доступные алиасы: {SUPPORTED_ALIASES}. " + f"Либо укажите полное имя модели onnx-asr." + ) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest tests/test_onnx_asr.py::TestModelAliases -v` +Expected: 4 PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/local_transcriber/backends/onnx_asr.py tests/test_onnx_asr.py +git commit -m "feat: add onnx-asr backend skeleton with model alias resolution" +``` + +--- + +### Task 3: Implement ensure_model_available + +**Files:** +- Modify: `src/local_transcriber/backends/onnx_asr.py` +- Modify: `tests/test_onnx_asr.py` + +- [ ] **Step 1: Write failing tests** + +Append to `tests/test_onnx_asr.py`: + +```python +class TestEnsureModelAvailable: + def test_returns_model_id_for_gigaam(self): + backend = OnnxAsrBackend() + result = backend.ensure_model_available("gigaam-v3", "int8") + assert result == "gigaam-v3-ctc" + + def test_returns_model_id_for_parakeet(self): + backend = OnnxAsrBackend() + result = backend.ensure_model_available("parakeet-v3", "fp16") + assert result == "nemo-parakeet-tdt-0.6b-v3" + + def test_stores_compute_type(self): + backend = OnnxAsrBackend() + backend.ensure_model_available("gigaam-v3", "float32") + assert backend._resolved_model_id == "gigaam-v3-ctc" + assert backend.actual_compute_type == "float32" +``` + +- [ ] **Step 2: Run tests — expect FAIL** + +Run: `uv run pytest tests/test_onnx_asr.py::TestEnsureModelAvailable -v` +Expected: FAIL — `ensure_model_available` not defined. + +- [ ] **Step 3: Implement ensure_model_available** + +Append to `OnnxAsrBackend` class in `src/local_transcriber/backends/onnx_asr.py`: + +```python + def __init__(self): + self.actual_compute_type: str | None = None + self._resolved_model_id: str | None = None + self._vad: Any = None + + def ensure_model_available( + self, + model_name: str, + compute_type: str, + on_status: Callable[[str], None] | None = None, + ) -> str: + """Resolves model alias and returns the onnx-asr model identifier. + + onnx-asr downloads models automatically via load_model(), + so this just validates the alias and returns the identifier string. + """ + self.actual_compute_type = compute_type + self._resolved_model_id = self._resolve_model(model_name) + return self._resolved_model_id +``` + +Add the import at the top of the file: + +```python +from collections.abc import Callable +from typing import Any +``` + +- [ ] **Step 4: Run tests — expect PASS** + +Run: `uv run pytest tests/test_onnx_asr.py::TestEnsureModelAvailable -v` +Expected: 3 PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/local_transcriber/backends/onnx_asr.py tests/test_onnx_asr.py +git commit -m "feat: implement OnnxAsrBackend.ensure_model_available" +``` + +--- + +### Task 4: Implement create_model + +**Files:** +- Modify: `src/local_transcriber/backends/onnx_asr.py` +- Modify: `tests/test_onnx_asr.py` + +- [ ] **Step 1: Write failing tests** + +Append to `tests/test_onnx_asr.py`: + +```python +class TestCreateModel: + def test_calls_load_model_with_correct_args(self, monkeypatch): + """Verify create_model passes correct args to onnx_asr.load_model.""" + calls = [] + + def fake_load_model(model=None, path=None, quantization=None, + cpu_preprocessing=None, **kwargs): + calls.append({ + "model": model, "path": path, "quantization": quantization, + "cpu_preprocessing": cpu_preprocessing, + }) + return FakeAsrAdapter() + + class FakeAsrAdapter: + def with_vad(self, vad): + return self + + monkeypatch.setattr("onnx_asr.load_model", fake_load_model) + + backend = OnnxAsrBackend() + backend.actual_compute_type = "int8" + model = backend.create_model("gigaam-v3-ctc", "onnx", "int8") + + assert len(calls) == 1 + assert calls[0]["quantization"] == "int8" + assert calls[0]["cpu_preprocessing"] is True + assert model is not None + + def test_loads_silero_vad(self, monkeypatch): + """Verify Silero VAD is loaded and attached to model.""" + vad_calls = [] + + def fake_load_vad(model, **kwargs): + vad_calls.append(model) + return "fake_vad" + + def fake_load_model(**kwargs): + return FakeAsrAdapter() + + class FakeAsrAdapter: + def with_vad(self, vad): + self._vad = vad + return self + + monkeypatch.setattr("onnx_asr.load_model", fake_load_model) + monkeypatch.setattr("onnx_asr.load_vad", fake_load_vad) + + backend = OnnxAsrBackend() + model = backend.create_model("gigaam-v3-ctc", "onnx", "int8") + + assert vad_calls == ["silero"] + + def test_fp16_compute_type(self, monkeypatch): + """Verify fp16 compute_type is passed through.""" + calls = [] + + def fake_load_model(model=None, quantization=None, **kwargs): + calls.append(quantization) + return FakeAsrAdapter() + + class FakeAsrAdapter: + def with_vad(self, vad): + return self + + monkeypatch.setattr("onnx_asr.load_model", fake_load_model) + monkeypatch.setattr("onnx_asr.load_vad", lambda **kw: None) + + backend = OnnxAsrBackend() + backend.create_model("parakeet-v3", "onnx", "fp16") + + assert calls == ["fp16"] +``` + +- [ ] **Step 2: Run tests — expect FAIL** + +Run: `uv run pytest tests/test_onnx_asr.py::TestCreateModel -v` +Expected: FAIL — `create_model` not defined. + +- [ ] **Step 3: Implement create_model** + +Append to `OnnxAsrBackend` class: + +```python + def create_model( + self, + model_path: str, + device: str, + compute_type: str, + cpu_threads: int = 0, + ) -> Any: + """Creates onnx-asr model with VAD. + + model_path: onnx-asr model identifier (e.g. "gigaam-v3-ctc"). + compute_type: "int8", "fp16", or "float32" — passed as quantization. + cpu_threads: not used by onnx-asr (onnxruntime manages threads internally). + """ + import onnx_asr + + ct = compute_type if compute_type in ("int8", "fp16", "float32") else "int8" + + model = onnx_asr.load_model( + model=model_path, + quantization=ct, + cpu_preprocessing=True, + ) + vad = onnx_asr.load_vad("silero") + self._vad = vad + return model.with_vad(vad) +``` + +- [ ] **Step 4: Run tests — expect PASS** + +Run: `uv run pytest tests/test_onnx_asr.py::TestCreateModel -v` +Expected: 3 PASS. + +Note: These tests mock `onnx_asr.load_model` and `onnx_asr.load_vad`, so no real model download happens. + +- [ ] **Step 5: Commit** + +```bash +git add src/local_transcriber/backends/onnx_asr.py tests/test_onnx_asr.py +git commit -m "feat: implement OnnxAsrBackend.create_model with VAD" +``` + +--- + +### Task 5: Implement transcribe + +**Files:** +- Modify: `src/local_transcriber/backends/onnx_asr.py` +- Modify: `tests/test_onnx_asr.py` + +- [ ] **Step 1: Write failing tests** + +Append to `tests/test_onnx_asr.py`: + +```python +from pathlib import Path +from local_transcriber.types import Segment, TranscribeResult + + +class FakeVadSegment: + """Mimics onnx-asr SegmentResult.""" + def __init__(self, start_ts, end_ts, text): + self.start_ts = start_ts + self.end_ts = end_ts + self.text = text + + +class TestTranscribe: + def test_transcribe_collects_segments(self, monkeypatch, tmp_path): + """Verify transcribe maps VAD segments to project Segments.""" + wav_file = tmp_path / "test.wav" + wav_file.write_bytes(b"fake audio") + + audio_samples = [0.0] * 16000 # 1 second of silence + + def fake_decode_audio(path, sampling_rate=16000): + import numpy as np + return np.array(audio_samples, dtype=np.float32) + + class FakeModel: + def recognize(self, waveform, sample_rate, language=None): + yield FakeVadSegment(0.0, 1.0, "hello") + yield FakeVadSegment(1.0, 2.5, "world") + + monkeypatch.setattr("faster_whisper.decode_audio", fake_decode_audio) + + backend = OnnxAsrBackend() + backend.actual_compute_type = "int8" + result = backend.transcribe( + FakeModel(), wav_file, language=None, + ) + + assert isinstance(result, TranscribeResult) + assert len(result.segments) == 2 + assert result.segments[0] == Segment(start=0.0, end=1.0, text="hello") + assert result.segments[1] == Segment(start=1.0, end=2.5, text="world") + assert result.duration == 1.0 # 16000 samples / 16000 Hz + + def test_transcribe_calls_on_segment(self, monkeypatch, tmp_path): + """Verify on_segment callback is invoked per segment.""" + wav_file = tmp_path / "test.wav" + wav_file.write_bytes(b"fake audio") + + def fake_decode_audio(path, sampling_rate=16000): + import numpy as np + return np.array([0.0] * 16000, dtype=np.float32) + + segments_captured = [] + + class FakeModel: + def recognize(self, waveform, sample_rate, language=None): + yield FakeVadSegment(0.0, 2.0, "one") + yield FakeVadSegment(2.0, 4.0, "two") + + monkeypatch.setattr("faster_whisper.decode_audio", fake_decode_audio) + + backend = OnnxAsrBackend() + result = backend.transcribe( + FakeModel(), wav_file, language=None, + on_segment=lambda s: segments_captured.append(s), + ) + + assert len(segments_captured) == 2 + assert segments_captured[0].text == "one" + assert segments_captured[1].text == "two" + + def test_transcribe_passes_language(self, monkeypatch, tmp_path): + """Verify language is passed to recognize().""" + wav_file = tmp_path / "test.wav" + wav_file.write_bytes(b"fake audio") + + def fake_decode_audio(path, sampling_rate=16000): + import numpy as np + return np.array([0.0] * 16000, dtype=np.float32) + + lang_received = [] + + class FakeModel: + def recognize(self, waveform, sample_rate, language=None): + lang_received.append(language) + yield FakeVadSegment(0.0, 1.0, "text") + + monkeypatch.setattr("faster_whisper.decode_audio", fake_decode_audio) + + backend = OnnxAsrBackend() + backend.transcribe(FakeModel(), wav_file, language="ru") + + assert lang_received == ["ru"] + + def test_transcribe_empty_audio(self, monkeypatch, tmp_path): + """Verify zero segments for silent audio.""" + wav_file = tmp_path / "test.wav" + wav_file.write_bytes(b"fake audio") + + def fake_decode_audio(path, sampling_rate=16000): + import numpy as np + return np.array([0.0] * 16000, dtype=np.float32) + + class FakeModel: + def recognize(self, waveform, sample_rate, language=None): + # No segments yielded + if False: + yield + + monkeypatch.setattr("faster_whisper.decode_audio", fake_decode_audio) + + backend = OnnxAsrBackend() + result = backend.transcribe(FakeModel(), wav_file) + + assert len(result.segments) == 0 + assert result.language == "unknown" + assert result.duration == 1.0 +``` + +- [ ] **Step 2: Run tests — expect FAIL** + +Run: `uv run pytest tests/test_onnx_asr.py::TestTranscribe -v` +Expected: FAIL — `transcribe` not defined. + +- [ ] **Step 3: Implement transcribe** + +Append to `OnnxAsrBackend` class: + +```python + def transcribe( + self, + model: Any, + file_path: Path, + language: str | None, + on_segment: Callable[[Segment], None] | None = None, + on_status: Callable[[str], None] | None = None, + ) -> TranscribeResult: + """Transcribes audio file using onnx-asr model with VAD. + + model: result of create_model() — a SegmentResultsAsrAdapter. + file_path: path to audio/video file (any format supported by faster-whisper decode). + language: language code (e.g. "ru", "en") — only meaningful for multilingual models. + """ + from faster_whisper import decode_audio + + _notify(on_status, "Загружаю аудио...") + audio_array = decode_audio(str(file_path), sampling_rate=16000) + duration = len(audio_array) / 16000.0 + + _notify(on_status, "Транскрибирую (onnx-asr)...") + segments: list[Segment] = [] + detected_language = language or "unknown" + + for vad_seg in model.recognize(audio_array, 16000, language=language): + seg = Segment( + start=max(0.0, vad_seg.start_ts), + end=max(0.0, vad_seg.end_ts), + text=vad_seg.text, + ) + if on_segment is not None: + on_segment(seg) + segments.append(seg) + _notify( + on_status, + f"Транскрибирую (onnx-asr)... [{len(segments)} сегм.]", + ) + + return TranscribeResult( + segments=segments, + language=detected_language, + language_probability=1.0 if language else 0.0, + duration=duration, + device_used="", # оркестратор проставит + ) +``` + +Add the helper function at the bottom of the file (before class): + +```python +def _notify(on_status: Callable[[str], None] | None, message: str) -> None: + if on_status is not None: + on_status(message) +``` + +Update imports at the top of `onnx_asr.py` — the full import block should be: + +```python +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path +from typing import Any + +from local_transcriber.types import Segment, TranscribeResult +``` + +- [ ] **Step 4: Run tests — verify all pass** + +Run: `uv run pytest tests/test_onnx_asr.py -v` +Expected: ALL tests pass (4 alias + 3 ensure + 3 create + 4 transcribe = 14 PASS). + +- [ ] **Step 5: Commit** + +```bash +git add src/local_transcriber/backends/onnx_asr.py tests/test_onnx_asr.py +git commit -m "feat: implement OnnxAsrBackend.transcribe with VAD segments" +``` + +--- + +### Task 6: Register backend in __init__.py + +**Files:** +- Modify: `src/local_transcriber/backends/__init__.py` +- Modify: `tests/test_onnx_asr.py` + +- [ ] **Step 1: Write failing test for get_backend("onnx")** + +Append to `tests/test_onnx_asr.py`: + +```python +class TestBackendRegistration: + def test_get_backend_returns_onnx_backend(self): + from local_transcriber.backends import get_backend + backend = get_backend("onnx") + assert isinstance(backend, OnnxAsrBackend) +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `uv run pytest tests/test_onnx_asr.py::TestBackendRegistration -v` +Expected: FAIL — `ValueError` or some error from `get_backend("onnx")`. + +- [ ] **Step 3: Register "onnx" device** + +Open `src/local_transcriber/backends/__init__.py` and add the `"onnx"` case before the fallback. The function should look like: + +```python +def get_backend(device: str, *, compute_type_explicit: bool = True) -> Backend: + """Возвращает экземпляр бэкенда для указанного устройства. + + Импорты ленивые — бэкенд загружается только при запросе. + compute_type_explicit: False если compute_type пришёл из дефолтов (влияет на fallback). + """ + if device in ("openvino", "openvino-gpu", "openvino-cpu"): + try: + from .openvino import OpenVINOBackend + except ImportError: + raise ValueError( + "OpenVINO бэкенд недоступен. Установите: pip install openvino-genai" + ) from None + return OpenVINOBackend( + ov_device=device, compute_type_explicit=compute_type_explicit + ) + + if device == "onnx": + try: + from .onnx_asr import OnnxAsrBackend + except ImportError: + raise ValueError( + "onnx-asr бэкенд недоступен. Установите: pip install onnx-asr[cpu,hub]" + ) from None + return OnnxAsrBackend() + + # cuda, cpu и всё остальное → faster-whisper + from .faster_whisper import FasterWhisperBackend + + return FasterWhisperBackend() +``` + +- [ ] **Step 4: Run test — expect PASS** + +Run: `uv run pytest tests/test_onnx_asr.py::TestBackendRegistration -v` +Expected: 1 PASS. + +- [ ] **Step 5: Run ALL tests to ensure nothing broken** + +Run: `uv run pytest -v` +Expected: all existing tests + new onnx-asr tests pass. No regressions in faster-whisper or OpenVINO. + +- [ ] **Step 6: Commit** + +```bash +git add src/local_transcriber/backends/__init__.py tests/test_onnx_asr.py +git commit -m "feat: register onnx-asr backend for --device onnx" +``` + +--- + +### Task 7: Manual smoke test with real model + +**Files:** +- No code changes. Manual verification only. + +- [ ] **Step 1: Verify CLI shows onnx device option** + +Run: `uv run transcribe --help` +Expected: shows `--device` option, verify `onnx` is listed. + +- [ ] **Step 2: Smoke test with GigaAM v3 on a short audio file** + +Requires a real audio file (WAV or MP3). Use a small test file if available. + +Run: `uv run transcribe /path/to/test.wav --device onnx --model gigaam-v3 --verbose` + +Expected: +- Model downloads from HuggingFace (first run, ~600 MB) +- Transcription runs +- Output `.md` file created with segments and timestamps +- No errors + +- [ ] **Step 3: Smoke test with Parakeet v3** + +Run: `uv run transcribe /path/to/test.wav --device onnx --model parakeet-v3 --language ru --verbose` + +Expected: +- Model downloads (first run) +- Transcription with Russian language +- Output `.md` created + +- [ ] **Step 4: Verify unknown alias error** + +Run: `uv run transcribe /path/to/test.wav --device onnx --model nonexisent` + +Expected: error message with available aliases (`gigaam-v3`, `parakeet-v3`). + +- [ ] **Step 5: Verify compute-type flag works** + +Run: `uv run transcribe /path/to/test.wav --device onnx --model gigaam-v3 --compute-type fp16` + +Expected: model loads with fp16 quantization (slightly larger download), transcription works. + +--- + +### Task 8: Write comparison script (optional, for experiment) + +**Files:** +- Create: `scripts/compare_backends.py` + +- [ ] **Step 1: Create comparison script** + +Create `scripts/compare_backends.py`: + +```python +"""Compare onnx-asr vs OpenVINO backends on real audio files. + +Usage: python scripts/compare_backends.py /path/to/audio.mp3 +""" +import sys +import time +from pathlib import Path +from local_transcriber.transcriber import load_model, _transcribe_file +from local_transcriber.backends import get_backend +from local_transcriber.types import Segment + + +def transcribe_with_backend(file_path: Path, device: str, model_name: str, + compute_type: str, language: str | None) -> tuple[float, int, str]: + """Run transcription and return (elapsed_sec, segment_count, transcript_text).""" + start = time.monotonic() + model_obj, actual_device, backend, model_path = load_model( + model_name, device, compute_type, + strict_device=True, + ) + tfr = _transcribe_file( + model_obj, actual_device, backend, model_path, + file_path, model_name, compute_type, + language=language, + ) + elapsed = time.monotonic() - start + text = " ".join(s.text for s in tfr.result.segments) + return elapsed, len(tfr.result.segments), text + + +def main(): + if len(sys.argv) < 2: + print("Usage: python scripts/compare_backends.py ") + sys.exit(1) + + file_path = Path(sys.argv[1]) + if not file_path.exists(): + print(f"File not found: {file_path}") + sys.exit(1) + + models_to_test = [ + ("gigaam-v3", "onnx", "int8"), + ("parakeet-v3", "onnx", "int8"), + ("medium", "openvino-cpu", "int8"), + ] + + print(f"File: {file_path.name} ({file_path.stat().st_size / 1e6:.1f} MB)") + print() + + for model_name, device, ct in models_to_test: + print(f"--- {model_name} on {device} (compute={ct}) ---") + try: + elapsed, seg_count, text = transcribe_with_backend( + file_path, device, model_name, ct, language="ru" if "gigaam" in model_name else None, + ) + print(f" Time: {elapsed:.1f}s") + print(f" Segments: {seg_count}") + print(f" Text preview: {text[:200]}...") + print() + except Exception as e: + print(f" ERROR: {e}") + print() + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 2: Run the comparison** + +Run: `uv run python scripts/compare_backends.py /path/to/real/audio.mp3` +Expected: timing and transcript preview for each model. + +- [ ] **Step 3: Commit** + +```bash +git add scripts/compare_backends.py +git commit -m "test: add onnx-asr vs OpenVINO comparison script" +``` + +--- + +## Self-Review + +1. **Spec coverage:** Each spec requirement maps to a task: + - Model aliases (gigaam-v3, parakeet-v3) → Task 2 + - ensure_model_available → Task 3 + - create_model with VAD → Task 4 + - transcribe with decode_audio → Task 5 + - Registration via get_backend("onnx") → Task 6 + - Dependencies → Task 1 + - Comparison → Task 8 + - Error handling (ValueError for bad alias) → Task 2 tests + +2. **Placeholder scan:** No TBD, TODO, or vague descriptions. All code is concrete. + +3. **Type consistency:** + - `_resolved_model_id` set in Task 3, used in Task 4 (consistent) + - `actual_compute_type` set in Task 3, checked in Task 4 tests + - `Segment` imported from types.py — matches project type + - `TranscribeResult` fields match types.py definition diff --git a/docs/superpowers/specs/2026-04-25-parakeet-backend-design.md b/docs/superpowers/specs/2026-04-25-parakeet-backend-design.md new file mode 100644 index 0000000..4e14f49 --- /dev/null +++ b/docs/superpowers/specs/2026-04-25-parakeet-backend-design.md @@ -0,0 +1,160 @@ +# Design: onnx-asr Backend (Parakeet / GigaAM) + +**Status**: Draft +**Date**: 2026-04-25 +**Branch**: experiment/onnx-asr-backend + +## Motivation + +Current CPU backends (faster-whisper, OpenVINO on CPU) run at 0.5-1.5x realtime for large models. Target audience has Intel integrated GPU or CPU only — no discrete NVIDIA GPU. + +[onnx-asr](https://github.com/istupakov/onnx-asr) is a lightweight ONNX Runtime wrapper supporting Parakeet, GigaAM, and FastConformer models. Key advantages: + +- **30-90x realtime on CPU** (vs 0.5-1.5x for faster-whisper) +- **Lightweight**: `numpy` + `onnxruntime` + `huggingface-hub` (~no added weight) +- **Python >= 3.10** — compatible with project +- **Russian-optimized models** with WER 4-5% (vs 10%+ for Whisper) + +## Goals + +1. Add `onnx-asr` as a third pluggable backend (alongside FasterWhisper and OpenVINO) +2. Compare speed and quality against OpenVINO on real Russian audio +3. Experiment is on a separate branch — merge only if results are compelling + +## Models + +Two model aliases exposed via CLI `--model`: + +| Alias | onnx-asr name | Language | WER (ru) | RTFx CPU | +|-------|--------------|----------|----------|----------| +| `gigaam-v3` | `gigaam-v3-ctc` | ru only | 4.72% | 59x | +| `parakeet-v3` | `nemo-parakeet-tdt-0.6b-v3` | 25 lang (auto-detect) | 10.95% | 34x | + +`gigaam-v3` is the default for `--device onnx` (best Russian quality + speed). +`parakeet-v3` is the multilingual fallback. + +User can also pass any valid onnx-asr model name directly (e.g. `nemo-canary-1b-v2`). + +## Architecture + +### New file + +`src/local_transcriber/backends/onnx_asr.py` — mirrors `openvino.py` structure. + +Implements the [Backend protocol](../adr/003-pluggable-backends.md) (structural typing): + +``` +class OnnxAsrBackend: + def ensure_model_available(model_name, compute_type, on_status) -> str + def create_model(model_path, device, compute_type, cpu_threads) -> Any + def transcribe(model, file_path, language, on_segment, on_status) -> TranscribeResult +``` + +### Model resolution (`ensure_model_available`) + +1. Resolve alias → onnx-asr model name via `MODEL_ALIASES` dict +2. Allow raw onnx-asr names (e.g. `nemo-parakeet-tdt-0.6b-v3`) to pass through +3. Return resolved model identifier string (onnx-asr handles download internally via `load_model`) + +### Model creation (`create_model`) + +```python +import onnx_asr +model = onnx_asr.load_model( + model_id, quantization=compute_type, cpu_preprocessing=True, +) +vad = onnx_asr.load_vad("silero") +model = model.with_vad(vad) +``` + +- `compute_type`: `int8` (default, quantized, fast), `fp16`, `float32` +- `cpu_preprocessing=True`: keeps mel-spectrogram computation on CPU (faster for CPU-only inference) +- VAD (Silero): always enabled — splits audio by voice activity, handles any length +- VAD segments naturally carry `start_ts`/`end_ts` — no separate `.with_timestamps()` needed + +### Transcription (`transcribe`) + +1. `faster_whisper.decode_audio(file, 16000)` → numpy float32 array (reuses same audio loader as OpenVINO backend, supports all media formats) +2. `model.recognize(audio_array, 16000, language=lang)` → iterator of VAD segments with `start_ts`, `end_ts`, `text` +3. Map each segment → project's `Segment(start, end, text)` dataclass +4. Return `TranscribeResult` with segments, language, duration + +Language handling: +- `gigaam-v3`: Russian only, `language` parameter ignored +- `parakeet-v3`: auto-detect (when `--language auto` / `None`) or explicit `--language ru/en/...` + +### Quantization support + +onnx-asr supports quantized ONNX models via `quantization` parameter: + +| compute_type | Description | RAM | Quality impact | +|-------------|-------------|-----|---------------| +| `int8` | 8-bit quantized (default) | ~300 MB | Minimal | +| `fp16` | Half precision | ~600 MB | None | +| `float32` | Full precision | ~1.2 GB | None | + +### Error handling + +- `RuntimeError` (OOM, ONNX session failure) → warning + fallback to CPU via `CPUExecutionProvider` +- Invalid model name → `ValueError` with list of supported aliases +- Corrupt audio / unsupported format → propagated from `faster_whisper.decode_audio` + +### Registration + +In `backends/__init__.py`: + +```python +if device == "onnx": + from .onnx_asr import OnnxAsrBackend + return OnnxAsrBackend() +``` + +Device `"onnx"` is NOT in auto-detect chain. Only explicit `--device onnx`. +Rationale: experimental backend, don't surprise existing users. + +### Dependencies + +Add to `pyproject.toml`: + +```toml +dependencies = [ + # ... existing ... + "onnx-asr[cpu,hub]>=0.11.0", +] +``` + +`onnxruntime` pulled transitively by `onnx-asr`. + +### CLI integration + +No CLI changes needed — existing `--device`, `--model`, `--compute-type`, `--language` flags work: + +```bash +# Russian, best quality/speed +uv run transcribe meeting.mp4 --device onnx --model gigaam-v3 + +# Multilingual +uv run transcribe podcast.mp3 --device onnx --model parakeet-v3 --language auto + +# With int8 quantization +uv run transcribe lecture.mp4 --device onnx --compute-type int8 +``` + +## Comparison approach + +Experiment compares onnx-asr against OpenVINO backend on real audio files: + +1. Pick 2-3 Russian audio files of varying length (1 min, 5 min, 30 min) +2. Run both backends, measure: elapsed time, segment count, dump transcripts +3. Qualitative: can a readable summary/conspect be made from the transcript? +4. Decision criteria: + - Noticeably faster at comparable quality → keep + - Noticeably better quality at comparable speed → keep + - Neither → discard + +## Non-goals + +- No auto-detect (device `"onnx"` must be explicit) +- No OpenVINO execution provider for onnx-asr (onnx-asr can use it, but out of scope) +- No German/French/etc language optimization — only Russian is benchmarked +- No replacing existing backends