Add word-level alignment for VoxCPM word-click playback
Session from read-along-ai-agent-traces on Hugging Face by kingkw1 · MIT · condensed by Coders Talk
Click a segment to jump to that moment
6msession
0interventions
GOAL
Replace proportional slicing of generated sentence audio with forced-alignment-style word timestamps (via faster-whisper) for more accurate click-to-hear-word clips, while keeping proportional slicing as a fallback.
- +0Prompt · +0Implement forced-alignment-style word timestamps for VoxCPM sentence audio using faster-whisper, falling back to proportional slicing when alignment fails; keep browser TTS fallback and add focused tests.
- +0mAgent did · +0mTraced existing proportional slicer and status plumbing, added alignment-vs-proportional visibility to prewarm status and logs, updated cache-reset test fixtures, and ran the pytest suite locally (24 passed).
- +5mFail · +5mDeveloper ran the app and reported clips bleeding into the next word (clicking 'cat' played 'cat sa'), plus one alignment failure logged for missing timestamp on 'sat'.
- +5mAgent did · +5mDiagnosed that aligned clips reused the proportional slicer's generous padding and didn't clamp word end to the next word's start; added tighter alignment-specific padding, boundary clamping, better fallback diagnostics logging recognized words, and a regression test for the bleed case, then reran the test suite (25 passed).
- +6mOutcome · +6mFix verified: aligned clips now clamp to next word's start with 0.015s padding vs 0.06s for proportional fallback; test suite passes (25 tests); one sentence still fell back to proportional due to a missing 'sat' timestamp, now logged with recognized words for debugging.
VERDICT
Next time I'd specify upfront that word clips must not bleed into adjacent words, since that was the real bug hiding behind 'alignment isn't better' — the fix was clamping word end to the next word's start plus much smaller padding for aligned clips versus the proportional fallback. Reporting the actual runtime log output was what let the agent find the real cause instead of just re-tuning fallback thresholds blindly.