Wire up word-click TTS in Gradio read-along app
Session from read-along-ai-agent-traces on Hugging Face by kingkw1 · MIT · condensed by Coders Talk
Connect word buttons in a Gradio reading app to a TTS wrapper so clicking a word plays it aloud, using either a Modal or local inference backend.
- +0Prompt · +0Wire up the TTS helper so clicking a word button in the Gradio UI plays the word out loud via the existing synthesize_speech wrapper.
- +0mAgent did · +0mTraced the word-click bridge through app.py, local_inference.py and modal_inference.py; found a race in the JS click bridge and an XSS-unsafe word serialization; patched app.py and requirements.txt to fix both and declare missing local TTS deps.
- +1mAgent did · +1mVerified the fix using the project's .venv (system python lacked gradio), ran py_compile and the reading-evaluation pytest suite, and added a regression test for update_audio_help; all green.
- +2mFail · +2mStarted the Gradio app locally; Gradio 6's Blocks() no longer honors css/head kwargs, meaning the click-bridge JavaScript likely never reached the browser despite Python-side code looking correct.
- +3mAgent did · +3mMoved CSS/JS injection to launch() to match Gradio 6.16.0's API, confirmed via the installed package's launch signature, reran tests, and relaunched the server on port 8060/8061.
- +9mPrompt · +9mI don't hear anything for either turbo or local mode -- my PC audio is fine since sentence recording playback works.
- +9mAgent did · +9mConfirmed via server logs and a direct gradio_client call to /update_audio_help that the click endpoint fires but hangs inside the TTS backend (Modal call never returns an audio path); checked for local espeak/pico2wave fallback (none installed).
- +12mAgent did · +12mAdded a browser-side speechSynthesis fallback so word clicks speak instantly without waiting on Modal/VoxCPM cold starts, kept the Gradio backend as a secondary path, made the helper audio visible, and reran compile/tests before restarting the server.
- +22mPrompt · +22mWhere is the word speech actually synthesized, and how can I make the voice sound less robotic across almost any browser for the hackathon?
- +22mOutcome · +22mExplained the word audio is generated client-side via the browser's speechSynthesis API (no wav, no Modal/VoxCPM call) and proposed a hybrid voice-selection strategy (prefer named natural/neural voices, fall back to any English voice) for cross-browser reliability at the hackathon.
Next time I'd open with the fact that TTS needs to work instantly for a kid clicking words, not just eventually — that would have surfaced the Modal cold-start problem before I burned time on Gradio 6 launch-arg compatibility. The correction that actually fixed things was going straight to server logs and a direct gradio_client call instead of trusting 'my PC audio is fine' — that's what showed the endpoint was firing but the backend TTS call was hanging. Switching word-helper audio to the browser's own speechSynthesis was the real unblock, not any of the backend plumbing fixes.