Frame selection is the whole game: notes from making LLMs watch video

A video is not an article about the video

Why feed a model a video at all, when a writeup of the same thing costs way fewer tokens?

Because an article is someone's compression of an event. A human watched it, decided what mattered, threw the rest away. The framing, the timing, the stuff on screen they didn't think was relevant. When an LLM reads the article, it learns inside that author's choices. It can't recover what got cut, and it can't disagree with a selection it never saw.

Give the model the video and the compression step moves to the model. It sees what the demo actually looked like, not what the reviewer said it looked like. It notices the error message that flashed by in a tutorial nobody bothered to transcribe. It catches that the "quick setup" took eleven cuts. Same subject, very different position: reading a witness statement vs being the witness. That's why the token cost is worth paying, and why the rest of this post is about spending it well.

The budget problem

Most "video understanding" today is a transcript plus frames sampled on a timer. One frame per second, or per ten seconds. Uniform sampling fails in both directions at once: it buries the model in near-identical frames of a talking head, and it skips right past the one second where something actually happened. Same root cause both times: the sampler has no idea what changed.

The constraint that shapes everything is token budget. Images are the most expensive thing you can put in a context window. Once you accept "about 100 to 150 frames per video, period", extraction stops being the problem. Selection is the problem. Every kept frame has to earn its slot.

Scene detection, and why a fixed threshold isn't enough

The first pass is standard: ffmpeg's scene score. Take a frame at every scene change, plus a low density floor so a long uncut shot still produces something. All in one chronological pass, because dedup later wants to compare true neighbours.

Fixed thresholds die on a specific kind of content: animation and slow camera work. A cartoon character squashing and stretching, or a slow pan, changes constantly but never sharply. The score never crosses the line and the sampler sleeps through the whole thing. The fix is boring but it works: compute per-frame scene scores in a metadata-only pass, then keep a frame whenever its score beats a multiple of the rolling average. High-motion footage raises its own bar, quiet footage lowers it. Nobody has to tune anything.

Deduplication: one channel became three

Dedup started as one comparator and grew a channel every time real footage embarrassed it.

1. The global channel

Downscale to a 16×16 RGB signature, count cells that moved more than 25/255 in any channel, drop the frame if fewer than about 8% changed. RGB, not grayscale, because a red-to-green cut with equal luminance looks identical to a grayscale comparator. And you compare against a sliding window of the last four kept frames, not just the previous one. Otherwise A-B-A cutting (interview shot, reaction shot, back to interview) re-admits a shot the model already saw, just because a different frame sat in between.

2. The action channel

A percentage threshold is structurally blind to small subjects. A person taking up 0.5% of a wide shot can never change 8% of the pixels no matter what they do, so the one second that mattered gets deduplicated away. I didn't find this in a benchmark. A user found it after running the tool on 2,181 real videos. The patch: on a 32×32 grid, if even a handful of cells change hard (more than 45/255), the frame counts as new regardless of percentage. Small subject, sharp change, kept.

3. The settled channel

The global channel also can't see small local state changes: a caption swap, a line of ink appearing on a whiteboard, a UI element updating. Those measure 0.0% at 16×16. So the third channel works on a 192×192 signature and looks for pixels that differ strongly from every frame in the kept window. It allows a plus-minus 1 pixel shift when matching, because film weave, sensor jitter and grain otherwise register as change everywhere.

The guards matter more than the detector here. It only runs when the scene is otherwise static, since motion is the other channels' job. Candidate pixels also have to survive a second, stricter tolerance pass. That kills soft-contrast drift like smoke dissipating, while ink and text keep a hard core and survive. And every triggered keep raises the gate with a decaying cooldown, so a flag that "settles" every second can't grab a frame each time, while a single new text card passes at the base gate. Without the cooldown, this one channel would eat the entire frame budget on one waving flag.

Fusing frames with the transcript

Text comes from embedded subtitles when the video ships them, otherwise local Whisper. Two details cost real debugging time:

• Whisper sometimes hallucinates a segment far past the end of the video on music-only audio. Build a timeline that trusts segment timestamps and a phantom line swallows it. So everything gets clamped to the media duration, plus one second of tolerance, because duration is often integer-truncated upstream.
• Handing the model a transcript and a pile of frames and letting it join them "by timestamp" works fine on short videos, then quietly misaligns on long ones. A frame gets quoted against the wrong sentence. So the join is precomputed: every transcript segment lists the frames that fall inside it, and silences longer than 1.5 seconds become frame-only spans. One clock. Nothing left for the reader to join.

Everything lands as plain files: JPEGs, a text transcript, and a manifest that tells the model how to read the package. The output is deliberately boring. That's what makes it portable across models.

Serving it over MCP

The newest surface is an MCP server: pip install "claude-real-video[mcp]", then crv-mcp. Any MCP client (Claude Desktop, Claude Code, Cursor) can call watch_video on a URL or file and get the fused result back. Two budget decisions carried straight over. Frames get resized to 768px on the long side before going back inline, because full-resolution frames burn context for zero visual gain. And analyses are cached per source, so a follow-up question about the same video never re-downloads or re-extracts anything.

Trying it

Everything above ships in the free, MIT-licensed tool. One command in, plain files out:

# needs Python 3.10+ and ffmpeg
pip install "claude-real-video[whisper]"
crv "https://youtube.com/watch?v=..."   # or a local file
→ frames/*.jpg      # the selected keyframes
→ transcript.txt    # timestamped, fused with the frames
→ MANIFEST.txt      # tells any LLM how to read the package

Drag the output into any vision-capable model, or let an MCP client drive the whole pipeline itself via crv-mcp.

What this still doesn't capture

Keyframes and a fused transcript tell the model what was on screen and what was said. They don't carry how the camera moves, the cutting rhythm, or the tone of a voice. That's a separate perception problem and out of scope for this post. But for the everyday case of "let the model actually watch this thing before answering", selection plus fusion covers a surprising amount of ground, entirely on your own machine.

Code, including everything described above: github.com/HUANGCHIHHUNGLeo/claude-real-video (MIT).