What The Model Sees

By the end of this session you will be able to:

• Trace an image from file bytes to the vectors that sit in the language model's token sequence, naming each stage
• Compute, before you send anything, the exact pixel dimensions a given model will see for one of your screenshots
• Decide between downscaling a whole frame and cropping the region you care about, and justify it with numbers

The five stages between your file and the model

You attach a PNG. The model answers. Between those two events are five stages, and every failure you will debug in this course lives in one of them.

1. Decode. Your file becomes a pixel array. Metadata does not survive: Claude does not parse or receive EXIF, so orientation flags, capture time and camera model are gone. Animated GIFs lose everything after frame one.
2. Resize. The pixel array is scaled down to fit the model's native resolution. This is where the 4000px screenshot stops being a 4000px screenshot.
3. Patch. The resized image is cut into a grid of fixed-size squares, 28x28 pixels on Claude. Each square is flattened and linearly embedded into one vector.
4. Encode. Those vectors run through a vision transformer, so each patch ends up carrying information about the patches around it. A patch of white pixels in the middle of a form knows it is inside a form.
5. Merge and project. A small trained network maps the encoder's output vectors into the language model's embedding space, and places them in the token sequence next to your text.

The language model never sees pixels. It sees a run of vectors from stage five, in the same sequence as the words you typed. That is why it can be confidently wrong about something you can read perfectly well: by the time the text side gets involved, the evidence has already been resized, cut into patches and compressed into a fixed number of vectors.

The merger is the part that gets skipped

Most explanations jump from "vision encoder" to "the model reads the image" and skip the join. The join matters.

Encoder and language model are trained separately and have different hidden widths. The encoder emits, say, 1152-dimensional vectors; the decoder expects its own. Something has to translate. That something is the projector, and in current open-weight VLMs it is a two-layer MLP, often called the merger because it does two jobs:

• It groups neighboring patch vectors. Qwen3-VL concatenates each block of adjacent patches into one vector before projecting, cutting the number of visual tokens the language model attends over by a factor of four.
• It projects the grouped vector into the language model's embedding space.

That MLP is trained to make the projected vectors behave like tokens the decoder already understands. When people say a model "does not see fine detail", the merger is often more of the answer than the encoder: four patches of dense small text get squeezed into one vector, and whatever did not survive is gone downstream.

Claude does not publish its projector architecture. It does publish the patch size and the resolution limits, which is what you need to reason about your own images.

What actually happens to a 4000px screenshot

Claude finds the largest aspect-preserving size that satisfies both limits at once:

• Edge limit: neither side exceeds the maximum edge length. 2576 px on Claude Sonnet 5 and other high-resolution-tier models, 1568 px on standard-tier models.
• Visual token limit: ceil(width / 28) * ceil(height / 28) must not exceed the model's budget. 4784 on the high-resolution tier, 1568 on standard.

For nearly every photo and screenshot, the token limit is what binds, not the edge limit. This is the single most common error: people scale to the long edge by hand and get a different image than the one the model saw. A 1920x1080 screenshot resizes to 1456x819, not 1568x882.

Compute it instead of guessing:

import math
 
 
def count_image_tokens(width: int, height: int) -> int:
    return math.ceil(width / 28) * math.ceil(height / 28)
 
 
def resized_size(width, height, max_edge=1568, max_tokens=1568):
    def fits(w, h):
        return (math.ceil(w / 28) * 28 <= max_edge
                and math.ceil(h / 28) * 28 <= max_edge
                and count_image_tokens(w, h) <= max_tokens)
 
    if fits(width, height):
        return (width, height)
    if height > width:
        resized_h, resized_w = resized_size(height, width, max_edge, max_tokens)
        return (resized_w, resized_h)
 
    aspect_ratio = width / height
    lo, hi = 1, width
    while lo + 1 < hi:
        mid = (lo + hi) // 2
        if fits(mid, max(round(mid / aspect_ratio), 1)):
            lo = mid
        else:
            hi = mid
    return (lo, max(round(lo / aspect_ratio), 1))
 
 
# A 4K screenshot on Claude Sonnet 5 (high-resolution tier):
print(resized_size(3840, 2160, max_edge=2576, max_tokens=4784))  # (2576, 1449)
 
# The same file on a standard-tier model:
print(resized_size(3840, 2160))                                  # (1456, 819)

After resizing, Claude pads the bottom and right edges up to the next multiple of 28. That padding holds no content, so if you ask for bounding boxes, normalize against the resized dimensions, never the padded ones.

Not every provider solves this the same way. Gemini tiles larger images into 768x768 crops rather than squashing the whole frame down. Different strategy, same constraint: a fixed budget of encoder inputs per image.

Downscaling is lossy in a way you can predict

Do the arithmetic on your own content. That 4K screenshot went to 2576 wide, a scale factor of 0.67, so UI text rendered at 12px is now 8px tall and a patch that held two glyphs now holds four. Below roughly 10px of glyph height, strokes stop being separable and the model starts producing plausible strings rather than reading them, which is worse than failing because it looks like an answer. On a standard-tier model the same file goes to 1456 wide, a factor of 0.38, and that text lands at 4.5px.

The fix is not a bigger image. Past the tier limit, extra pixels are discarded before the encoder runs, so a 6000px and a 4000px capture of the same screen arrive identical. Send less of the screen instead:

from PIL import Image
 
# Same file as above, so resized_size is already defined.
shot = Image.open("screenshot.png")
print(shot.size)  # (3840, 2160)
 
# Option A: the whole frame, pre-resized to exactly what the model will see.
shot.resize(resized_size(*shot.size, max_edge=2576, max_tokens=4784)).save("full.png")
 
# Option B: the region you actually care about, untouched.
shot.crop((1200, 300, 2200, 1000)).save("crop.png")

crop.png is 1000x700, under both limits, so it is not resized at all: every pixel reaches the encoder.

Try it

Take a screenshot of your own screen at native resolution, containing some small text.

1. Run resized_size on its dimensions for both tiers. Write down both answers before you send anything.
2. Send the full frame to claude-sonnet-5 and ask it to read one specific short string from that small text, exactly as written.
3. Crop about 1000x700 around the same string and send the crop with the identical question.

import base64
import anthropic
 
client = anthropic.Anthropic()
 
with open("crop.png", "rb") as f:
    data = base64.standard_b64encode(f.read()).decode()
 
message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64",
                                         "media_type": "image/png",
                                         "data": data}},
            {"type": "text", "text": "Read the text in the status bar, exactly as written."},
        ],
    }],
)
print(message.content[0].text)

Success condition: the crop returns the string correctly, the full frame fails or invents a similar-looking string, and you predicted the full frame's resized dimensions to the pixel beforehand.

Common mistakes

• Scaling to the long edge by hand. The token limit binds first on almost every real image. 1920x1080 goes to 1456x819, and if you assumed 1568x882 every coordinate you compute is off.
• Sending a bigger file to get more detail. Anything past the tier limit is discarded before the encoder, so a 6000px capture buys you nothing over a 2576px one.
• Upscaling a small image. Interpolation invents no information. A 200x100 crop stretched to 1500x750 just gives the encoder more patches of the same blur.
• Recompressing to JPEG on the way out. Lossy compression on text is the artifact class that makes small glyphs unreadable, and it compounds across passes.
• Assuming the model sees your file's orientation. No metadata reaches it. If your capture pipeline writes an EXIF rotation flag, rotate the pixels yourself first.

Where this goes next

You now know what the model sees. The next session, Counting Visual Tokens, turns that into arithmetic you can budget with: Gemini's 258 tokens per PDF page against Claude's patch formula, priced over a thousand pages.