Measure Anthropic vs OpenAI provider token overhead in Pi
Session from pine-of-glass-sessions on Hugging Face by thomasmustier · MIT · condensed by Coders Talk
Investigate whether Anthropic's API is silently adding hidden content beyond what Pi sends, by running near-empty probe requests against both providers and comparing captured payload sizes to reported token usage.
- +0Prompt · +0Figure out if Anthropic is adding hidden extra content vs OpenAI by simulating minimal Pi requests and comparing token/cache-write usage against payload sizes.
- +0mAgent did · +0mExplored repo structure, read pi-contextimate docs/scripts and existing context-accounting notes to understand tooling for measuring provider overhead.
- +1mAgent did · +1mRan probe-live-prefix.mjs against openai-codex/gpt-5.5 and anthropic/claude-opus-4-8 with varying flag combinations (no-tools, no-skills, default) to compare payload sizes and token usage between providers.
- +3mAgent did · +3mFixed a bug in probe-live-prefix.mjs where the summary script assumed OpenAI-shaped payloads (instructions/input) and mis-measured Anthropic payloads (system/messages), then verified with a smoke test and git diff check.
- +5mAgent did · +5mWrote up findings to /tmp/pi-provider-token-probe-2026-06-05.md: no evidence of large hidden Anthropic content; the token delta looks explained by Claude 4.8 tokenization/accounting plus prompt-cache write accounting.
- +6mPrompt · +6mExplain the patch to probe-live-prefix.mjs in more detail.
- +7mPrompt · +7mPush the patch.
- +7mFail · +7mPush rejected: remote main had moved ahead since the local commit was based on an older origin/main.
- +7mAgent did · +7mFetched origin/main, rebased the fix commit on top, and pushed successfully as 67c48fe.
- +21mPrompt · +21mFile a GH issue proposing to refine how 'system prompt' accounting works so tool/extension additions aren't hidden inside a monolithic system prompt bucket.
- +21mOutcome · +21mExperiment concluded no large hidden Anthropic payload content; fixed and pushed probe-live-prefix.mjs (commit 67c48fe) and filed GitHub issue #9 for refining system-prompt/tool token accounting UX.
I'd open next time by asking directly for a script that captures raw provider payloads alongside token usage, since that's what actually let us settle the hidden-content question. The rebase-before-push step was the only real snag, and fetching origin/main and rebasing cleanly resolved it without any data loss. The write-up at the end was worth doing since it turned a one-off experiment into something reusable for the next token-accounting question.