Build GPT-judge evaluation pipeline for Qwen student action plans
Session from homeroom-copilot-open-traces on Hugging Face by ravi2505 · MIT · condensed by Coders Talk
Export existing risk/intervention data into evaluation case files, score the Qwen-generated action plans against a 7-criteria rubric, and diagnose why low/moderate risk plans scored poorly.
- +0Prompt · +0Build evaluation/case_001.md through case_008.md from existing profile, risk, root-cause and intervention outputs for 8 students, without calling Qwen or generate_text(), leaving Action Plan section blank with [PASTE ACTION PLAN HERE].
- +0mAgent did · +0mFixed the export script's placeholder text to match exactly, ran the developer-only export utility, and generated all 8 case files plus a JSON dataset without touching Qwen or generation code.
- +1mAgent did · +1mPasted the provided Qwen action plans into the 8 case files' Action Plan sections and synced evaluation_dataset.json; self-recovered from a PowerShell quoting error on the first sync attempt.
- +5mPrompt · +5mScore each of the 8 cases as an educational intervention specialist across relevance, correctness, root cause alignment, evidence alignment, actionability, clarity and overall quality; export to gpt_judge_results.json and gpt_judge_summary.md.
- +5mAgent did · +5mRead all 8 case files, scored them per rubric, exported JSON/Markdown judge results, and identified that low-risk plans over-intervened (inventing concerns like tutoring/parent meetings) while high/critical plans scored well (~4.0 overall).
- +21mPrompt · +21mImplement only the Action Plan prompt revision: risk-aware rules capping Low Risk to monitoring/1 action per week and Moderate Risk to 1-2 interventions, without touching risk scoring, retrieval, model config or UI. Regenerate one Low and one Moderate plan and compare before/after.
- +22mAgent did · +22mRevised the prompt rules in src/action_plan_generator.py, validated syntax, but found the local venv lacked torch/transformers/spaces needed to actually run Qwen generation, so verified the new prompt inputs via the non-LLM pipeline instead.
- +51mPrompt · +51mHere are the new Qwen-generated plans for Student Low B and Student Moderate B under the revised prompt; update the evaluation and rescore.
- +52mAgent did · +52mReplaced the Low/Moderate action plan sections in case files 1-4 with the revised outputs, resynced the JSON dataset, and recomputed the judge scores/summary while leaving High/Critical cases unchanged.
- +54mOutcome · +54mFull-system judge averages after the prompt revision: Relevance 3.50, Correctness 3.00, Root Cause Alignment 2.88, Evidence Alignment 3.38, Actionability 4.00, Clarity 4.00, Overall Quality 3.50 — up from an initial Correctness of 2.75 and Root Cause Alignment of 2.50, though Low Risk plans still over-infer mild concerns.
Next time I'd ask for the risk-group breakdown (Low/Moderate/High/Critical) from the very first scoring pass instead of asking for it turn by turn, since that grouping is what actually revealed the real problem was the fixed 4-week output format, not retrieval or root-cause analysis. Restricting the fix to a prompt-only change in src/action_plan_generator.py, with an explicit 'do not modify' list, kept the change scoped and measurable, which is what let us show Correctness and Root Cause Alignment improve without touching risk scoring or retrieval.