claude-real-video — Let Any LLM Actually Watch and Understand Video

The Video Problem That Won't Go Away

Here's the state of play with LLMs and video in mid-2026:

• ChatGPT reads the transcript, not the picture. It can tell you what was said, but not what was shown.
• Claude flat-out rejects video files. You get "I can't process video" and a suggestion to upload images instead.
• Gemini (the only major model with native video support) uploads your video to Google and samples at a fixed interval — usually 1 frame per second. That means you miss fast cuts and drown in near-identical frames of a static presentation.

This isn't a model limitation — it's a pipeline problem. LLMs can already understand images. A single frame is just an image. The hard part is deciding which frames matter and how to present them.

Enter claude-real-video — an MIT-licensed, open-source tool that solves this with a smart pipeline: scene-change detection, sliding-window deduplication, Whisper transcription, and a clean frame manifest any LLM can ingest. It runs entirely on your machine. Your video never leaves.

What Makes This Different From Every Other "LLM Watches Video" Script

There are a dozen GitHub repos that claim to let Claude watch video. Most of them do the naive thing: ffmpeg -r 1 — grab one frame per second, dump them all into a folder, and hand them to an LLM with a transcript.

That works for a 30-second clip. For a 10-minute screencast you're looking at 600 near-identical frames and a context bill that would make an API key cry.

claude-real-video takes a smarter approach with a four-stage pipeline:

1. Scene-change detection. Instead of a fixed interval, it uses ffmpeg's select filter with scene-change sensitivity. Every time the visual content actually changes — cut, transition, new slide — it captures a frame. You control the sensitivity with --scene (default: 0.30, lower = more frames).

2. Density floor. Even in static sections, it guarantees at least one frame every N seconds (--fps-floor, default: 1.0). So a 60-second shot of a talking head still produces one frame, not zero.

3. Sliding-window deduplication. This is the killer feature. New frames are compared against the last N kept frames (default window: 4). If fewer than 8% of pixels changed, the frame is dropped. That means a 10-minute slide deck collapses from 600 frames to maybe 10-15 actual unique slides. A fast-cut music video catches every visual beat because each frame is different.

4. Whisper transcription. Audio is extracted and transcribed with OpenAI Whisper, including language auto-detection. You get SRT, VTT, and a plain-text MANIFEST.txt that pairs each key frame with its timestamp and transcript segment.

The output is a clean folder with:

• Deduplicated key frames (actual images)
• MANIFEST.txt — the golden file any LLM can read
• Timestamped transcript files
• Optionally: the full soundtrack (--keep-audio) and a visual report.html showing every frame decision

Compare this with what else is out there:

You feed the model fewer, more meaningful frames. Cheaper context, better understanding.

What This Unlocks for AI Agents

This isn't just "Claude can watch a YouTube video." The tool's URL support (via yt-dlp) and local file pipeline means it fits into a much wider range of agentic use cases:

Video content summarization. Point it at a 45-minute conference talk, get 8-12 key slides plus a full transcript. Feed the MANIFEST.txt to any LLM for a structured summary with visual references. No more "I watched the video and here's what I think based on the transcript only."

Surveillance and camera analysis. Local execution means security footage never leaves your network. A cron job can feed recent clips through claude-real-video, extract key frames of motion events, and hand them to an LLM for scene understanding. "Did anyone enter the restricted area between 2-3 AM?" becomes a question you can ask without uploading sensitive footage anywhere.

Podcast and interview analysis. With --keep-audio, you get the full soundtrack plus key frames. Audio-capable models can hear tone and delivery while also seeing any visual elements (slide decks, guest reactions, whiteboards).

Automated video QA for agents. Imagine an MCP server wrapping this tool. An agent discovers a video URL, calls the server to extract frames + transcript, and answers questions about the video content. This is the pattern mcp-video-server started exploring — and claude-real-video makes the frame extraction side dramatically smarter.

Private video processing. Medical training videos, internal product demos, client meetings — anything with confidentiality constraints. The tool never uploads, never phones home.

The Bigger Pattern: Smart Extraction as the Universal Bridge

What claude-real-video is doing is part of a wider shift I've been tracking all year. The industry is quietly converging on a pattern:

Don't make the model handle raw sensory data. Extract intelligently, then prompt.

We saw this with shot-scraper's video frame grabber. We saw it with the MCP video server's 14-tool pipeline. We saw it with the video-frames-skill that pinned 768px as the universal image resolution sweet spot across models. And now claude-real-video adds scene-aware deduplication to the mix — arguably the most important piece yet, because token costs scale linearly with frame count, and most video content is 90% redundant frames.

The interesting development here is that each iteration makes the pipeline smarter, not just faster. Early approaches dumped everything and let the model sort it out. That's expensive and unreliable. claude-real-video does the sorting before the model ever sees a frame — scene detection, dedup, transcription — and hands the LLM a clean, minimal representation.

This is the same trajectory we saw with RAG: raw data → chunking → embedding → retrieval → prompt. Video understanding is following the same arc: raw video → scene detection → frame selection → transcription → manifest → prompt.

The Local-First Advantage

One detail that shouldn't be overlooked: this tool runs entirely locally. yt-dlp downloads, ffmpeg processes, Whisper transcribes — all on your machine. The only network call is downloading the source video (and even that's optional if you point it at a local file).

That matters more than most developers realize. The current regulatory climate is making cloud video processing increasingly fraught. Meta banned Claude Code over distillation concerns. Alibaba banned it over backdoor markers. These are text-only tools. Video is a much richer attack surface — frame extraction from internal security feeds, transcription of proprietary audio, the works.

A local pipeline lets you use any LLM you want (Claude, GPT, Gemini, open-weight models running locally) while keeping the raw video on your hardware. The LLM only sees the extracted text manifest and a handful of key frames. That's a much narrower privacy surface.

The Catch

It's not all roses. A few limitations worth noting:

Whisper CLI is a dependency. You need openai-whisper installed separately. On a beefy machine that's fine; on a laptop it's a 2-3 GB download and noticeable GPU usage.

Scene-change sensitivity is a dial, not a switch. Getting the right --scene value for a given video type takes experimentation. A talking-head interview needs different sensitivity than a fast-cut action trailer. The --report flag helps — it generates an HTML visualization showing every frame decision — but you'll still iterate.

150 frame hard cap. The --max-frames default of 150 is sensible (it keeps token costs under control) but means very long videos with frequent scene changes will still lose information. Bump it if you need to, but watch your context window.

No streaming support yet. You process the full video, then hand it to the LLM. Real-time or near-real-time video analysis isn't the use case here.

Bottom Line

claude-real-video is one of the most practical open-source tools I've seen this month for the simple reason that it takes video understanding from "technically possible with enough engineering effort" to "one command and you're done."

It's not trying to be a full agent framework or an MCP server — it's a focused, well-executed pipeline that does one thing well: turn a video URL into an LLM-friendly manifest. That's exactly the kind of tool that disappears into larger workflows and makes them work better.

For agent builders, this is worth adding to your toolkit right now. The combination of local execution, smart deduplication, and clean LLM output makes it a building block for a surprising range of applications. And the pattern — extract less, understand more — is one I expect to see replicated across every sensory modality before the year is out.

GitHub: HUANGCHIHHUNGLeo/claude-real-video — MIT license, Python 3.10+, ffmpeg required.