Fix over-prescribing Action Plan prompt in student risk system
Session from homeroom-copilot-open-traces on Hugging Face by ravi2505 · MIT · condensed by Coders Talk
Diagnose why an LLM-generated student Action Plan feature scores poorly for Low/Moderate risk students (over-prescribing interventions) and revise the prompt to be risk-aware without touching retrieval, scoring, or root-cause code.
- +0Prompt · +0Evaluation shows low scores for Low/Moderate risk plans due to over-prescribed interventions; analyze whether the Action Plan generation prompt is the root cause.
- +0mAgent did · +0mInspected action_plan_generator.py, intervention_engine.py, and app.py to trace what the model sees; found the prompt forces a full 4-week plan format for all risk levels and root-cause text mislabels 'No incidents' as 'Behavior concerns'.
- +5mPrompt · +5mImplement only the Action Plan prompt revision with risk-aware rules (Low: monitoring only, max 1 action/week; Moderate: max 1-2 interventions; preserve High/Critical); do not touch scoring, retrieval, root cause, model config, or UI.
- +5mAgent did · +5mPatched action_plan_generator.py to add risk-aware planning rules, frame retrieved interventions as options not a checklist, and restrict Sources Used to interventions actually used; verified syntax.
- +6mFail · +6mAttempted to run the actual generation pipeline locally to test the new prompt, but the venv was missing spaces, torch, transformers, and accelerate, so real Qwen generation couldn't be run locally.
- +6mAgent did · +6mVerified the new prompt inputs (risk-aware rules present) via the non-LLM pipeline for two sample students, then summarized the prompt changes as implemented.
- +31mPrompt · +31mHere are the new Qwen-generated plans for Student Low A and Student Moderate A using the revised prompt; check the scores.
- +32mAgent did · +32mScored the new outputs: modest improvement (overall quality up from ~2.0 to 3.0) but still over-prescribing, especially unsupported monitoring/engagement mentions for stable students.
- +35mAgent did · +35mUpdated evaluation artifacts (case_001-004.md, evaluation_dataset.json, gpt_judge_results.json, gpt_judge_summary.md) with new Low/Moderate outputs for a second pair of students and recomputed full-system averages, leaving High/Critical scores unchanged.
- +38mOutcome · +38mFinal risk-group breakdown: Low 3.00/2.00/2.00/3.00/4.00/4.00/3.00, Moderate same as Low, High 4.00 across the board, Critical ~4.00/3.50; prompt fix improved Low/Moderate overall quality from ~2.0-2.5 to 3.0 but over-intervention persists.
Next time I'd ask upfront for the prompt's exact output-format constraints, not just the risk-level instructions, since the rigid 4-week structure was quietly forcing intervention-heavy plans regardless of risk. Scoping the fix strictly to the prompt (no retrieval/scoring changes) was the right call and gave a real but partial improvement. I still need another pass on root-cause wording and stricter guardrails against inferring concerns from stable data before Low/Moderate scores will match High/Critical quality.