Get Off the Grid local inference badge-ready on a HF Space
Session from read-along-ai-agent-traces on Hugging Face by kingkw1 · MIT · condensed by Coders Talk
Make the Read-Along AI Gradio app's local (Off the Grid) inference mode actually deployable and verifiable on a Hugging Face Space, while keeping the existing Modal/local toggle intact, in order to qualify for the Off the Grid and Llama Champion hackathon badges.
- +0Prompt · +0Review our docs and outline a plan to get the Off the Grid and Llama Champion badges working within our HF Space.
- +0mAgent did · +0mRead all hackathon docs, app.py, local_inference.py, requirements, and model assets; identified that requirements.txt was Modal-only while local mode needed faster-whisper/voxcpm/llama_cpp, and proposed a dual-mode plan using upgraded Space hardware.
- +19mPrompt · +19mPlease implement the dual-mode deployment plan: add local deps, package the Q4 GGUF, harden the toggle, add a smoke check, update README.
- +20mIntervention · +20mDeveloper stopped the agent mid-edit and pushed back: we already have the Space set up, we're just enabling local mode, stop adding curriculum/fallback scope.Why: I saw it start talking about curriculum fallbacks and drifting from the actual task, so I cut it off and re-narrowed scope.
- +47mFail · +47mFirst deploy attempt failed: the Space repo has a 1GB storage cap and the 1.68GB Q4 GGUF alone blew past it.
- +47mAgent did · +47mReworked local_inference.py to download the GGUF from a separate HF model repo at runtime instead of bundling it in the Space payload; shrank the deploy payload from 1.7GB to under 1MB; uploaded the GGUF to the model repo and redeployed successfully.
- +1h 26mFail · +1h 26mSpace build failed again with exit 137 OOMKilled, caused by pulling huge CUDA torch wheels and compiling llama-cpp-python from source.
- +1h 27mAgent did · +1h 27mPinned Space Python to 3.11, forced CPU-only torch wheels, and pointed pip at a prebuilt llama-cpp-python CPU wheel index instead of source-building; verified the wheel resolves, then redeployed and the build succeeded.
- +2h 7mPrompt · +2h 7mHold up — after your latest change, single-word playback takes ~2 minutes and full sentence playback takes ~35 seconds. That's way too slow.
- +2h 7mIntervention · +2h 7mDeveloper had the agent revert the on-demand live VoxCPM word-generation feature it had just wired up, keeping only the unrelated warning-suppression fix, since live TTS was unacceptably slow for the child-facing UI.Why: Live VoxCPM per-word generation was way too slow for a kid clicking on words, so I had it ripped out and kept only the harmless logging fix while we plan a cached-audio approach instead.
- +2h 23mOutcome · +2h 23mLocal Off the Grid mode verified end-to-end on the deployed HF Space (faster-whisper + llama.cpp MiniCPM, ~10s per reading attempt); dual Modal/local toggle preserved; drafted a fresh-session prompt to tackle TTS latency next via pre-generated/cached word audio.
Next time I'd lead with the Space's hard constraints (1GB repo limit, free-tier build memory, no CUDA) before letting the agent draft a plan, since two of our failures were pure packaging/build issues we could have anticipated upfront. The correction that mattered most was reining in scope right after the first 'implement the plan' prompt — it kept drifting into curriculum/fallback design when the actual ask was just 'make local mode work in the Space we already have.' Killing the live per-word VoxCPM feature immediately after seeing 2-minute latency was the right call too; it saved us from shipping something that would embarrass the demo.