Trace how Claude Code and Codex estimate /context token usage
Session from pine-of-glass-sessions on Hugging Face by thomasmustier · MIT · condensed by Coders Talk
Understand how Claude Code's /context command estimates token usage internally, then compare it to how OpenAI Codex estimates context/token usage, to inform pine-of-glass's own estimation approach.
- +0Prompt · +0Look at tmustier/cc-inv - how does Claude Code actually do the context estimate in /context and /context all?
- +0mAgent did · +0mCloned cc-inv, traced /context command through context.tsx, context-noninteractive.ts, analyzeContext.ts and tokenEstimation.ts to find how token estimates are computed and rendered.
- +2mPrompt · +2mHow do the estimates work?
- +2mAgent did · +2mExplained the hybrid approach: Anthropic token-counting API for most buckets (system prompt, tools, message history), rough string-length estimates for some sub-breakdowns, and actual last-response usage for the headline total.
- +3mPrompt · +3mPlease document this locally in a gitignored file.
- +4mAgent did · +4mWrote findings to .pi/cc-context-estimates.md and confirmed via git status --ignored that the .pi/ directory is already gitignored.
- +4mPrompt · +4mDoes openai/codex have a similar thing? It's OSS, so clone to inspect if needed.
- +4mAgent did · +4mCloned codex, traced token_usage.rs, protocol.rs TokenUsage struct, sse/responses.rs, and context_manager/history.rs, finding Codex relies mostly on server-reported usage plus a simple approx_token_count helper (bytes/4) rather than a rich local estimator like Claude Code's.
- +13mPrompt · +13mSo Codex uses bytes/4, not chars/4? And we currently estimate with chars, not bytes - would bytes be better?
- +16mOutcome · +16mSession ended with analysis only (no code changes): confirmed Codex uses ceil(UTF-8 bytes/4), found pine-of-glass currently estimates via JS .length (UTF-16 code units, labelled 'chars'), and concluded bytes would likely be a more portable primitive but existing calibration in docs/pi-contextimate.md needs care before switching.
I'd start the prompt by naming both repos and the exact question (how do the token estimates work, and are we doing it right) instead of exploring cc-inv first and only later remembering to compare against Codex and our own code. The useful correction was asking it to actually document findings locally in a gitignored file - without that the investigation would have stayed ephemeral chat output. No agent mistakes here; it was a clean read-only research session.