Fix duplicate MTGO pauper league dates and stale crawl data
Session from merchantscroll-traces on Hugging Face by vedalken · MIT · condensed by Coders Talk
Investigate why two pauper leagues showed up under the same calendar date and fix the underlying crawler/pipeline logic so league dates and deck counts stay accurate.
- +0Prompt · +0Two pauper leagues both show up under Jun 17 2026 — how is this possible?
- +0mAgent did · +0mTraced the bug to MTGO's `starttime` field reflecting the last publish date rather than the league week, causing an old Nov 2025 league to be grouped under Jun 17 2026 once it got new decks. Added a canonical_starttime() helper using site_name as the source of truth, wired it into utils.py, crawler.py, pipeline.py and player_stats.py, updated tests, and re-ran pytest (found 41 dates affected repo-wide).
- +0mPrompt · +0mIt's odd the real Jun 17 league only has 1 entry — run the crawler to check.
- +0mAgent did · +0mRan a live crawl and found MTGO already had 9 decks for the Jun 17 league versus 1 stored locally, because the pipeline only crawls tournaments it has never seen. Added same-day re-crawl logic to pipeline.py (matching existing entrypoint.py behavior), re-ran the full pipeline to refresh the data, and confirmed unit tests still pass.
- +0mPrompt · +0mCan we run the crawler more often — is once an hour too much?
- +0mAgent did · +0mReviewed the current every-2-hours GitHub Actions cron, counted typical request volume per run (2 MTGO fetches, up to 5-10 on busy days), and concluded hourly is safe, estimating ~24 runs/day and ~50-100 requests/day.
- +0mPrompt · +0mUpdate the crawler schedule and align all the referenced text.
- +0mAgent did · +0mChanged the GitHub Actions cron from `0 */2 * * *` to `0 * * * *` and updated every '2 hours'/'2-hour' reference across index.html meta tags, docs/sow.md, and docs/features/installable-webapp.md to say hourly, confirming no remaining stale references.
- +0mPrompt · +0mReferencing a terminal error, asked the agent to fix a pre-commit failure.
- +0mAgent did · +0mInvestigated pre-commit's Python 3.10 virtualenv, found pip missing from the uv-managed venv and a broken cached pre-commit environment, tried installing pip via uv, cleaning pre-commit's cache, and running without VIRTUAL_ENV set, but was still inspecting the broken cached venv contents when the transcript ends.
- +0mOutcome · +0mSession ends with three shipped fixes (duplicate-date grouping via canonical_starttime, same-day league re-crawl, hourly cron + doc updates) all passing pytest, while the separate pre-commit/pip venv issue remains unresolved.
Next time I'd start by asking it to check whether starttime actually reflects the league date at all, since that assumption caused the whole bug — that one question would have saved several rounds of grepping. The correction that mattered was pushing back with 'the real Jun 17 league only has 1 entry, run the crawler to check,' which uncovered the second, separate stale-data bug the agent hadn't noticed. The pre-commit/venv fix was still unresolved when the session ended, so I'd pick that back up separately.