Debug Modal QLoRA fine-tuning of MiniCPM-V-4.6
Session from kirana-detective-build-traces on Hugging Face by build-small-hackathon · MIT · condensed by Coders Talk
Get a Modal-hosted training script fine-tuning openbmb/MiniCPM-V-4.6 on synthetic Indian invoice images to run end-to-end, after unsloth and version conflicts blocked it.
- +0Prompt · +0Fix Modal training run: unsloth requires transformers<=5.5.0 but the script pins transformers[torch]>=5.7.0, causing a dependency conflict.
- +0mAgent did · +0mDiagnosed the unsloth/transformers version clash and pinned transformers to <=5.5.0 in train_minicpm_v.py to satisfy unsloth's constraint.
- +8mFail · +8mNew error: MiniCPM-V-4.6 isn't supported on transformers==5.4.0 and needs >=5.7.0 — the version pin just traded one conflict for another.
- +11mAgent did · +11mResearched the MiniCPM-V-CookBook, concluded unsloth can never bridge the gap, and rewrote the whole script to drop unsloth in favor of native PEFT + BitsAndBytes QLoRA with a custom VLMCollator.
- +32mFail · +32mTraining crashed inside PEFT: task_type="CAUSAL_LM" forces a wrapper class requiring prepare_inputs_for_generation, which the custom MiniCPMV4_6Model doesn't implement.
- +33mAgent did · +33mSet task_type=None to use the base PeftModel, then hit a missing build_conversation_input_ids method and a DataLoader-subprocess crash; moved all preprocessing before PEFT wrapping and dropped dataloader workers.
- +42mFail · +42mNew crash: apply_chat_template chokes on inline PIL images with 'argument of type PngImageFile is not iterable', and every sample gets skipped.
- +44mAgent did · +44mIterated through processor/collator fixes across many rounds: text-only chat templates, AutoProcessor called with both text and images, image-placeholder token discovery, and repeated tensor-shape mismatches on pixel_values/target_sizes, adding debug prints and a DebugTrainer to see exact shapes hitting the model.
- +1h 21mFail · +1h 21mTraining still fails with 'Multimodal features and tokens do not match, tokens: 0, features: 454' — the tokenizer-based chat template never inserted image pad tokens.
- +1h 24mOutcome · +1h 24mSwitched to processor.apply_chat_template with structured {"type": "image"} messages so pad tokens get inserted correctly, and added a main_dryrun entrypoint that loads the model on CPU and checks pad-token counts in ~2 minutes without a full paid training run. Fix applied but full training success not yet confirmed in this session.
Next time I'd start by checking the model's actual transformers/peft requirements against unsloth's supported range before writing any training code — that single incompatibility cost the whole first hour. The fix that actually moved things forward was dropping unsloth entirely for native PEFT+bitsandbytes and then adding debug prints of every tensor shape, since MiniCPM-V-4.6's custom vision pipeline kept silently mismatching pixel_values/target_sizes in ways the errors alone didn't explain. I still hadn't seen a clean full training run finish by the end of the session.