Add local GGUF inference path and hybrid Modal/local toggle for read-along app
Session from read-along-ai-agent-traces on Hugging Face by kingkw1 · MIT · condensed by Coders Talk
Add an offline inference path (ASR, TTS, and a fine-tuned MiniCPM judge quantized to GGUF) as an alternative to the existing Modal cloud endpoints, then expose a toggle in the Gradio UI to switch between them.
- +0Prompt · +0Write a standalone scripts/convert_to_gguf.py that downloads kingkw1/minicpm-phonetic-evaluator and quantizes it to Q4_K_M GGUF using huggingface_hub and llama.cpp, plus the exact bash commands to run it.
- +0mAgent did · +0mInspected repo layout, wrote scripts/convert_to_gguf.py (download via huggingface_hub, convert with llama.cpp's convert_hf_to_gguf.py, quantize to Q4_K_M), updated .gitignore, verified with py_compile and --help.
- +11mFail · +11mRunning the conversion locally crashed: the .venv Python was picking up a Conda libtorch_python.so, causing 'undefined symbol: _PyCode_SetExtra' during the llama.cpp conversion subprocess.
- +11mAgent did · +11mPatched the script to sanitize the conversion subprocess environment (stripping Conda vars, LD_LIBRARY_PATH, PYTHONPATH), retried the conversion, and confirmed the quantization log produced both the f16 and q4_k_m GGUF files (~1.7GB).
- +14mPrompt · +14mGenerate local_inference.py with three functions mirroring the Modal stubs: local_transcribe_audio (faster-whisper tiny.en), local_synthesize_speech (VoxCPM), local_ask_minicpm_judge (llama-cpp-python with the GGUF judge), matching API_CONTRACT_SPEC.md and app.py's wrapper signatures.
- +15mAgent did · +15mRead API_CONTRACT_SPEC.md, ROADMAP.md, modal_inference.py and app.py to match call signatures, then wrote local_inference.py with lazy-loaded models and verified it compiled and imported cleanly.
- +24mFail · +24mAfter unmasking the swallowed exceptions per the dev's debugging request, running test_local.py surfaced real failures: wrong Python interpreter, a missing audio fixture path, Conda/Torch library contamination, and a mismatched torchaudio CUDA wheel.
- +25mAgent did · +25mIteratively fixed each root cause: switched to the project .venv interpreter, added audio path resolution for the real fixture location, added a sanitized re-exec guard against Conda library leakage, reinstalled a CPU-matched torchaudio wheel, installed voxcpm, and disabled its optional ModelScope denoiser call; test_local.py then passed end to end.
- +35mPrompt · +35mWire a hybrid-engine toggle into app.py: a gr.Radio for 'Turbo Mode (Modal)' vs 'Off the Grid Mode (Local)', route evaluate_reading and TTS through the chosen engine, and commit directly to main.
- +35mAgent did · +35mAdded imports for the local functions, a gr.Radio toggle in app.py, routing logic (with a compatibility shim for existing single-arg test doubles), threaded the toggle into microphone and TTS button events, ran py_compile and pytest, then committed the change on main as 348fafe.
- +1h 1mOutcome · +1h 1mDual-mode branch feature/dual-mode ahead of origin with 2 commits (local inference + hybrid toggle); PR description drafted summarizing local_inference.py, convert_to_gguf.py, and the app.py toggle for merge into main.
Next time I'd ask for the environment sanitization (Conda vs .venv Torch conflicts) to be handled up front, since that same libtorch_python.so issue bit me twice, once during GGUF conversion and again during local_inference.py testing. The correction that actually moved things forward was insisting on removing the broad try/except blocks so the agent could see real stack traces instead of guessing at '[ASR_ERROR]' and None returns. Once errors were unmasked, the fixes (path resolution, env sanitization, torchaudio wheel, disabling the denoiser) came quickly.