Add placement-based spatial verification for generated PPTX slides (clean-slides)
Session from clean-slides-sessions on Hugging Face by thomasmustier · MIT · condensed by Coders Talk
Solve the problem of AI agents being unable to visually verify slide layouts (overlapping labels, misaligned charts) by researching and implementing a structured, non-black-box verification approach for the clean-slides library.
- +0Prompt · +0I keep struggling to get the agent to visually verify slide output (overlaps, mispositioned labels on waterfall charts). I've tried zooming, render tests, warnings, ASCII translation - all rough. Others must face this in analogous cases like frontend?
- +0mAgent did · +0mExplored the clean-slides codebase (render.py, verification.py, screenshot.py, renderer.py, spec.py, sizing.py) to understand the code→renderer→pixels pipeline and existing overlap/boundary/text-fit checks, then proposed framing the problem as code bugs vs renderer interpretation mismatches.
- Break · 35m away, not counted
- +1mPrompt · +1mPlease can you use search to research this? Write your findings to a .md
- +2mAgent did · +2mRan ~12 web searches on VLM spatial reasoning limits, Galen framework, chip-design DRC, SlideAgent, OCR bounding boxes, and Claude computer-use, then wrote a 295-line findings doc (VISUAL-VERIFICATION-RESEARCH.md) concluding vision models architecturally lose pixel precision and structural/placement-based checks are the way forward.
- +15mPrompt · +15mOk, can we think through how we would do this? I would like a robust but easy to understand solution. I'm most concerned about brittleness or creating a black box
- +19mPrompt · +19mThe tool is intended solely for agent use, so it's about verifying arbitrary pptx files the agent generates/edits - but let's worry about generation first
- +19mAgent did · +19mDesigned a 'Placement' data model: renderer-emitted shapes tagged with role and group (e.g. bar/label sharing group='Revenue'), with standalone check functions (overlap, bounds, group alignment, spacing) instead of a framework. Wrote the design into section 4 of the research doc.
- +31mPrompt · +31mOk, let's implement
- +31mAgent did · +31mImplemented clean_slides/placement.py (Placement, PlacementIssue, check_overlaps/bounds/group_alignment/spacing) plus 22 new tests in test_placement.py; fixed a pyright unused-function warning; all 43 project tests pass and pyright reports 0 errors.
- +42mAgent did · +42mRan the new checks against a real generated deck (clean) and a simulated waterfall chart with deliberate misalignment, correctly flagging a 0.5in off-center label and uneven bar spacing; built and opened an actual misaligned waterfall .pptx to visually confirm.
- +49mAgent did · +49mNoted the placement module isn't wired into the generate/verify pipeline yet, created GitHub issue #13 documenting the remaining integration work, updated the research doc with implementation status, and pushed both the feature commit and the docs commit.
- +51mOutcome · +51mShipped placement.py + 22 tests (43 total passing, 0 pyright errors), a 295+ line research doc, and GitHub issue #13 tracking pipeline integration; two commits pushed to main.
Next time I'd start by telling the agent up front that this tool is agent-only and used solely for fresh generation, not editing - that scoping cut a lot of wasted exploration. Making it dig through the actual renderer/solver code before proposing solutions, and pushing back on the 'black box' risk, is what got us to a design I actually trust: plain check functions over a tagged Placement list rather than a heuristic vision-based approach. The module works and is tested, but it's still not wired into the real generate/verify pipeline, which is the obvious next step.