The underrated AI video-reading tool
There are three routes for getting an LLM to "watch" a video. I have walked all three, and one of them got stress-tested by a user across 2,181 videos. Here are the results.
黃志弘 Leo
Published 2026-09-11 · Updated 2026-10-07 · 1 readers
The three routes on the table
One. Throw it straight at a multimodal model (Gemini)
Shortest path, and to be fair: ask "what is happening in this video" and Gemini is genuinely strong. The cost is structural. The video leaves your machine, repeat runs are not guaranteed to match, and the same evidence cannot be handed to another model. When an answer looks off, you have nothing to check it against, so you rerun and hope.
Two. Use a description pipeline (byjlw/video-analyzer, about 1,600 stars as of October 2026)
The design is honest: OpenCV pulls keyframes, Whisper transcribes, a local vision model (Llama 3.2 11B) describes each frame, and you get a reconstructed write-up at the end. The problem is that your LLM never sees the video. It reads another model's opinion of the video. Every frame the vision model gets wrong becomes a fact your LLM repeats with confidence. And you are now feeding a second model in the loop.
Three. Hand the evidence to your own LLM (claude-real-video)
This is the underrated one. Yes, I built it, so judge the argument and not the author. crv turns a video into something an LLM can read: scene-aware frame extraction, real source timestamps, a transcript with speaker labels, plus a MANIFEST that tells the agent how to read the folder. No second model sits in the middle. Claude (or GPT, or a local model) looks at the actual frames and cites frame_012 @ 00:03:41. All of it runs on your machine.
What 2,181 videos taught me
One user ran crv across his entire photo library, 2,181 videos in four days, and sent me the failure list sorted by severity. Two of those bugs changed how I see the whole category:
Percentage dedup is blind by design
A small object crosses a still frame, the whole-frame change rate barely moves, and the frame gets thrown out as a duplicate. A bird across the sky, a hand reaching into shot, exactly the frames a human would keep. Every tool that dedups on whole-frame difference has this blind spot. crv now looks at change in local blocks. Tools that sample at fixed intervals dodge the bug by keeping everything, and your context window pays for it.
Timestamps have to survive the whole pipeline
After extraction, dedup and renaming, most pipelines can no longer map back to the source time. The model can describe that slide but cannot tell you the minute and second it appeared. crv parses the real PTS from ffmpeg and writes it into frames.json. The same user re-checked with a 22-minute lecture: 60 frames mapped back to the source, 60 out of 60.
Neither fix came out of my own tests. Both came from a user with a pile of real video. That is the truth about this category: the hard bugs live in the places a demo never goes.
Which one to use when
- You want a quick question answered about one video and the upload does not bother you → Gemini
- You want a ready-made text description and you already run Ollama → video-analyzer
- You want your own LLM reasoning locally, reproducibly, with timestamps → claude-real-video
pip install "claude-real-video[fast]"
npx skills add HUANGCHIHHUNGLeo/claude-real-videoRelated pages
Let your LLM actually watch the video.
MIT open source, runs entirely on your machine.
Get it on GitHub See crv Pro