Fix word-click TTS playback in a Gradio reading app (VoxCPM + browser TTS)
Session from read-along-ai-agent-traces on Hugging Face by kingkw1 · MIT · condensed by Coders Talk
Get realistic voice audio to play when a child clicks a word or 'Listen to Sentence' in a Gradio-based reading app, after playback silently stopped working following a curriculum-mode change.
- +0Prompt · +0I don't hear anything when I click individual words or 'Listen to Sentence'. Should I be getting browser TTS for this?
- +0mAgent did · +0mTraced the Gradio audio wiring in app.py, found the speech output component was mounted with visible=False (so nothing was playable) and that Gradio 6 ignores head/css JS passed to Blocks() instead of launch(). Fixed both and added a regression test.
- +7mPrompt · +7mImplement asynchronous background TTS pre-warming: memory cache words per sentence in a background thread, JIT-generate on cache miss for word clicks, so clicks feel zero-latency without static files.
- +7mAgent did · +7mAdded TTS_MEMORY_CACHE, prewarm_level_words(), threaded prewarming on next_sentence, and a JIT fallback in update_audio_help(). Discovered Gradio's Audio component wanted a filepath not raw bytes and added a bytes-to-wav conversion at the UI boundary. Tests passing.
- +17mIntervention · +17mDev heard the wrong audio moment: 'Amazing reading' praise TTS played after a successful recording, not the word-help audio. Wanted word clicks to use instant browser speechSynthesis like before curriculum mode, keeping backend TTS only for 'Listen to Sentence'.Why: That isn't where I want audio applied - word clicking is really the point of this, and it wasn't working.
- +27mFail · +27mBenchmarked backend TTS: Modal VoxCPM calls for a single word didn't return within 20-30s, and even local VoxCPM took ~5s per short word after warmup with frequent 'badcase' retries - unusable for live word-click latency.
- +34mAgent did · +34mMade Off the Grid (local) mode the default, dropped Modal for word audio, and built a hybrid approach: background pre-generate cached local VoxCPM word audio per sentence with a progress indicator, expose it to the browser as base64 data URLs for direct playback, and fall back to browser speechSynthesis when audio isn't ready yet.
- +59mFail · +59mDev reported chaotic playback: clicking 'the' played 4+ overlapping voices, 'cat' produced just a tap noise, and 'sat' said 'at UH!' loudly - isolated short-word VoxCPM generation was producing artifacts and overlapping clips, not a wiring bug.
- +59mAgent did · +59mAdded a quality gate to skip local VoxCPM for short/ambiguous words, then implemented text padding (e.g. 'cat' -> 'Cat.') to reduce autoregressive clipping, and finally switched to generating one full-sentence VoxCPM WAV and proportionally slicing it into per-word clips by character length to avoid isolated-word hallucinations.
- Break · 6h 32m away, not counted
- +1h 16mOutcome · +1h 16m18 tests passing, but proportional sentence slicing still gave imprecise word boundaries. Dev drafted two follow-up prompts (silence-based slicing and an alternative approach) to run in parallel via Codex web while merging the current incremental branch.
Next time I'd start the first prompt by stating explicitly that this is a kids' reading app needing realistic per-word audio with strict latency and reliability constraints, so we don't cycle through browser-TTS-vs-VoxCPM-vs-Modal three separate times. The correction that actually mattered was pointing out the praise audio was firing instead of word-help audio - that's what redirected the whole approach toward hybrid caching instead of chasing a phantom wiring bug. Isolated short-word VoxCPM generation was never going to be reliable; slicing full-sentence audio was the right direction but still needs real alignment, not proportional character-length guessing.