docs: design for image input — paste a screenshot, ask about it - #89
docs: design for image input — paste a screenshot, ask about it#89ndemianc wants to merge 3 commits into
Conversation
Sorry, something went wrong.
The interaction being copied — ⌘V a screenshot into the composer, no upload step — is
the easy part. This documents the four things in our code that break quietly when the
first image goes through, and the cost model that decides how bytes should be shaped
before they leave the webview.
The four, each verified against develop rather than recalled:
1. conversation carries content as a STRING (extension.js:1752). There is no shape an
image can take. This is the structural change; everything else follows.
2. translate.js SILENTLY DROPS any block it doesn't recognise. On OpenAI-compatible
providers an attached image would vanish between composer and wire, and the model
would answer confidently about text it never saw — no error, no log line. A user
would reasonably conclude the model hallucinates. Worth fixing on its own merits,
before images, which is why it's slice I1.
3. `vision: true` is ALREADY in the catalog, per model, and nothing reads it. There's
a supportsToolsForModel and no supportsVisionForModel. Half the gate exists.
4. estimateMsgTokens is JSON.stringify(m).length / 4 — sound for text, catastrophic
for base64: a 1MB screenshot books ~333,000 phantom tokens, more than most context
windows. The same estimate drives findCompactionCut, so pasting one screenshot
would evict real conversation history. This is the bug that would have shipped as
"long conversations forget things after I paste an image".
ON THE NUMBERS. I checked the vision API rather than trusting my prior, and the prior
was wrong: cost is not w×h/750, it is ⌈w/28⌉ × ⌈h/28⌉ visual tokens over 28px patches,
with a per-tier cap (2576px/4784 tokens on 4.7+, 1568/1568 below). I reimplemented the
resize rule and reproduced the documented figures exactly — 1092² → 1521, 1000² → 1296,
1920×1080 → 2691 — so the cost table in §1 is arithmetic, not estimate.
That verification changed a decision. Token cost is ALREADY capped server-side, so
client-side downscaling is not a defence against a token blowup — it is a deliberate
fidelity-for-cost trade (4784 → 1792 on a 4K grab) and a defence against the wire. The
doc says so rather than implying downscaling is load-bearing for cost safety.
Also recorded: writing §D3 I ran a 1160×480 capture through `sips -Z 1568` and it GREW,
40KB → 89KB, because the tool scaled it up to meet the cap. Never upscale — the rule is
min(1, cap/longEdge), and a factor of 1 means pass the original bytes through untouched,
which also avoids stacking compression artifacts on screenshots of text.
Seven slices, I1–I7, each independently shippable with bypass-verifiable exit criteria.
Deferred with reasons: Files API upload (Anthropic-only, wins on repeat turns), PDFs,
coordinates, client-side OCR.
Building a live cost calculator from the doc's numbers caught two things the prose had glossed. The resize rule is not an iterative shrink. My first implementation stepped the scale down by 1% until the patch grid fit, which gets the TOKEN COUNT right every time but misreports the sent dimensions — 1447x814 where the docs say 1456x819. Replaced with a binary search for the largest scale whose grid fits the cap, which is what the rule actually is. Checked against every worked example in the vision docs: token count matches on all twelve, dimensions on eleven. The twelfth is one standard-tier row a single pixel wide of the reference (1270 vs 1269, same 1564 tokens) — a rounding convention I could not derive from six data points, and the doc now says so rather than claiming the rule was reproduced "exactly". Two rows in the cost table were computed with the stepping version and are corrected: the macOS retina grab is 2380x1546 (not 2377x1544) and the 12 MP photo is 2212x1659 at 4740 tokens (not 2193x1645 / 4661). Both were mine, neither came from the docs. None of this moves a decision — the 1568 default and the "server already caps the cost" argument rest on the 4K row, which was right. But a plan whose own arithmetic disagrees with its live calculator is worse than one with no calculator.
There was a problem hiding this comment.
Pull request overview
Adds a design document outlining how LevelCode’s AI chat should support image input (e.g., pasting screenshots), focusing on the current codebase constraints, a verified vision token cost model, and a staged implementation plan (I1–I7) to avoid silent failures and incorrect token accounting.
Changes:
- Introduces docs/IMAGES.md describing the end-to-end image pipeline (webview normalization → host storage → provider request shaping).
- Documents four current “quiet breakages” (string-only content, silent block drops, unused vision gating, and naive token estimation) and their implications.
- Specifies concrete design decisions (block widening, image-first ordering, normalization rules, content-addressed storage, and capability gating) plus slice-by-slice exit criteria.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sorry, something went wrong.
|
|
||
| This is the most dangerous thing in the list. On any OpenAI-compatible provider — which is most of them through the gateway — an attached image would **vanish between the composer and the wire**, and the model would answer confidently about text it never saw. No error, no warning, no log line. A user would reasonably conclude the model is hallucinating. | ||
|
|
||
| **The vision capability is already modelled, and nothing reads it.** `providers/catalog.js` carries `vision: true` per model and has done since the multi-provider work: |
There was a problem hiding this comment.
Correct — describeCaps reads c.vision (catalog.js:162) to render "vision" in the model picker's detail line, and getModelChoices passes it through at :192. "Nothing reads it" was wrong; nothing gates on it is the actual gap, and that is what the doc says now.
Fixed here, and the gate itself now exists — supportsVisionForModel landed in the implementation branch.
Sorry, something went wrong.
| } | ||
| ``` | ||
|
|
||
| Sound for text. For a base64 image it charges roughly **one third of the byte count as tokens** — a 1 MB screenshot books ~333,000 phantom tokens, which is larger than most context windows. `findCompactionCut` would fire on the first screenshot and evict real conversation history to make room for an image that actually costs ~4,800. This is not a rounding error; it is the meter reading the wrong quantity entirely. |
There was a problem hiding this comment.
You are right, and this was the one worth catching. I asserted a causal chain without opening the function I named.
findCompactionCut cuts on message count and goal boundaries — len - keepRecent, then isGoalBoundary — and never reads a token number. compactAgentMemory calls estimateMsgTokens only for beforeMsgTokens/afterMsgTokens, which land in a dbg line and the return value. The function's own comment says it: used only for the UI meter.
So the live consequence is narrower than I wrote: the meter reads wildly high the moment an image is attached, telling someone to start a new chat when they are nowhere near full. Still worth fixing — and it becomes a correctness bug rather than a display one the day any auto-compaction policy keys off that number. The doc says that now.
I also corrected the same overstatement where it had already reached shipped code: the comment on imageBlockTokens in imageCost.js and its test both repeated it. A false causal claim in the source outlives one in a design note, because the next person reads the comment and not this thread.
Sorry, something went wrong.
| | High-resolution | Claude 4.7 and later | 2576 px | 4784 | | ||
| | Standard | everything else | 1568 px | 1568 | | ||
|
|
||
| Images above either limit are **downscaled server-side, preserving aspect ratio**. I reimplemented the rule and checked it against every worked example in the documentation: the **token count matches on all twelve** (1092² → 1521, 1000² → 1296, 1920×1080 → 2691, 3840×2160 → 2576×1449 at 4784), and the sent dimensions match on eleven — one standard-tier row lands a single pixel off (1270 vs 1269 wide, same 1564 tokens), a rounding convention I could not derive from six data points. Cost is exact; geometry is exact to a pixel: |
There was a problem hiding this comment.
Fair — the paragraph describes a one-pixel difference and then says "exact to a pixel", which reads as exact equality. Now: Cost is exact; geometry is correct to within a pixel.
Sorry, something went wrong.
Three review comments on #89, all correct. THE ONE THAT MATTERS. The doc claimed a bad token estimate would make findCompactionCut evict real conversation history on the first pasted screenshot. That is false. findCompactionCut cuts on message count and goal boundaries and never reads a token number; compactAgentMemory uses estimateMsgTokens only for its before/after report. The function's own comment says so — "used only for the UI meter". I asserted a causal chain without opening the function it named. The live consequence is narrower and the doc now says so: the context meter reads wildly high the moment an image is attached, telling someone to start a new chat when they are nowhere near full. Still worth fixing, and it becomes a correctness bug rather than a display one the day anything automatic keys off that number. Also corrected: - "nothing reads it" about the vision flag — describeCaps DOES read it to render "vision" in the model picker. Nothing GATES on it, which is the actual gap. - "geometry is exact to a pixel" read as exact equality when the paragraph had just described a one-pixel difference. Now "correct to within a pixel".
Design only — no feature code. Seven slices, I1–I7.
The target is what Cursor and the Claude Code console already do: screenshot, ⌘V into the composer, ask why it looks wrong. That part is easy. This documents what breaks in our code when the first image goes through, and the cost model that decides how the bytes should be shaped before they leave the webview.
Four things that break quietly
Verified against develop, not recalled. Three of the four fail silently, which is the reason to write them down before writing feature code.
The third would have shipped as a mystery report: "long conversations forget things after I paste a screenshot."
The numbers changed a decision
I checked the vision API rather than trusting my prior, and the prior was wrong — cost is not w × h / 750, it is ⌈w/28⌉ × ⌈h/28⌉ over 28px patches, capped per tier (2576px/4784 tokens on 4.7+, 1568/1568 below).
That changed the design. Token cost is already capped server-side. Sending a 12 MB PNG does not buy more than 4784 tokens of fidelity — it buys latency. So client-side downscaling is not a defence against a token blowup, as I would otherwise have written it; it is a deliberate fidelity-for-cost trade (4784 → 1792 on a 4K grab) and a defence against the wire.
I reimplemented the resize rule and checked it against every worked example in the docs: token count matches on all twelve, dimensions on eleven — one standard-tier row lands a single pixel off, and the doc says so rather than claiming exactness.
Slices
I1 is worth merging regardless of images — the silent block drop is a live bug today.
Every exit criterion is written to be bypass-verifiable.
Deferred, with reasons
Files API upload (Anthropic-direct only; wins on repeat turns, not the first), image output (Claude does not generate images), PDF blocks, coordinates, client-side OCR.
Also
A readable version of this plan, with a live visual-token calculator built from the verified formula: https://claude.ai/code/artifact/a31690f7-79f1-4f6e-9902-3c32fd7f91de