Add checkpoint resume and fix HF push for YOLO26n training on Modal
Session from kirana-detective-build-traces on Hugging Face by build-small-hackathon · MIT · condensed by Coders Talk
Add checkpoint/resume support to a YOLO26n training script that got interrupted mid-run, then fix the HuggingFace push pipeline so trained artifacts (best.pt, best.onnx, class_names.json) actually land in the repo.
- +0Prompt · +0Training got cancelled when laptop restarted mid-run — can checkpointing be added so it resumes?
- +0mAgent did · +0mRead train_yolo26n.py, moved merged dataset onto the Modal volume, and added resume logic checking for last.pt to call model.train(resume=True)
- +5mFail · +5mResumed run crashed with an optimizer state_dict mismatch error from ultralytics when loading last.pt
- +5mAgent did · +5mAdded a fallback: try resume=True, catch the optimizer mismatch, fall back to weights-only resume with a lower learning rate for remaining epochs
- Break · 3h 16m away, not counted
- +6mPrompt · +6mTraining finished but HF repo isn't updated with new file and README — check yolo_output.txt and the Modal volume, then push
- +6mAgent did · +6mFound best.pt was never uploaded and the model card would've embedded 1831 class bullet points; wrote a new push_yolo_to_hf.py uploading best.pt, ONNX and a clean model card
- +13mIntervention · +13mAfter running the push, HF still only showed the old class_names.json and onnx — asked to use modal CLI to list volume paths and mtimes properly before touching the script againWhy: the fix didn't actually work and I wanted to see the real state of the volume before trusting another patch
- +13mAgent did · +13mListed volume contents via modal CLI, diagnosed that HF's LFS dedup silently skipped the onnx upload and dropped best.pt from the same commit, then rewrote the script to stage files locally and use upload_folder for one atomic commit
- +16mIntervention · +16mPush still reported 'no files modified, skipping commit' — asked to instead download the files from Modal locally and write a script to upload from thereWhy: the Modal-side push kept getting silently skipped by HF dedup no matter how it was batched, so I wanted the files local so I could control the upload directly
- +16mOutcome · +16mDownloaded best.pt, best.onnx and class_names.json locally from the Modal volume, then wrote upload_yolo_to_hf.py using create_commit/CommitOperationAdd to force an atomic commit that bypasses HF's LFS dedup skip
Next time I'd ask upfront for a push script that uses create_commit/CommitOperationAdd from the start, since upload_file and upload_folder both got silently skipped by HF's LFS dedup and wasted two rounds. The intervention that actually mattered was making it list the Modal volume contents with the CLI directly instead of trusting the script's own success output — that's what surfaced the real problem instead of another blind patch.