There are three routes for getting an LLM to "watch" a video. I have walked all three, and one of them got stress-tested by a user across 2,181 videos. Here are the results.
Shortest path, and to be fair: ask "what is happening in this video" and Gemini is genuinely strong. The cost is structural. The video leaves your machine, repeat runs are not guaranteed to match, and the same evidence cannot be handed to another model. When an answer looks off, you have nothing to check it against, so you rerun and hope.
The design is honest: OpenCV pulls keyframes, Whisper transcribes, a local vision model (Llama 3.2 11B) describes each frame, and you get a reconstructed write-up at the end. The problem is that your LLM never sees the video. It reads another model's opinion of the video. Every frame the vision model gets wrong becomes a fact your LLM repeats with confidence. And you are now feeding a second model in the loop.
This is the underrated one. Yes, I built it, so judge the argument and not the author. crv turns a video into something an LLM can read: scene-aware frame extraction, real source timestamps, a transcript with speaker labels, plus a MANIFEST that tells the agent how to read the folder. No second model sits in the middle. Claude (or GPT, or a local model) looks at the actual frames and cites frame_012 @ 00:03:41. All of it runs on your machine.
One user ran crv across his entire photo library, 2,181 videos in four days, and sent me the failure list sorted by severity. Two of those bugs changed how I see the whole category:
A small object crosses a still frame, the whole-frame change rate barely moves, and the frame gets thrown out as a duplicate. A bird across the sky, a hand reaching into shot, exactly the frames a human would keep. Every tool that dedups on whole-frame difference has this blind spot. crv now looks at change in local blocks. Tools that sample at fixed intervals dodge the bug by keeping everything, and your context window pays for it.
After extraction, dedup and renaming, most pipelines can no longer map back to the source time. The model can describe that slide but cannot tell you the minute and second it appeared. crv parses the real PTS from ffmpeg and writes it into frames.json. The same user re-checked with a 22-minute lecture: 60 frames mapped back to the source, 60 out of 60.
Neither fix came out of my own tests. Both came from a user with a pile of real video. That is the truth about this category: the hard bugs live in the places a demo never goes.
pip install "claude-real-video[fast]" npx skills add HUANGCHIHHUNGLeo/claude-real-video
MIT open source, runs entirely on your machine.
Get it on GitHub See crv Pro