GitHub - shepnerd/vidify: Vidify is an ASR-first video understanding agent that turns YouTube, local files, and live streams into timelines, searchable Q&A, reports, and highlight clips

GitHub

Vidify is a video understanding agent. Give it a YouTube URL, HTTP video URL, or local video and get structured analysis, searchable indexes, Q&A, highlights, reports, and live-stream understanding.

Vidify is organized as layered entrypoints, a core orchestrator, composable workflows and skills, model adapters, and cache-backed outputs.

What It Does

CapabilityDescriptionAnalyzeDownload media, extract subtitles/metadata, run ASR when needed, and build timelinesUnderstandCaption frames, read OCR text, detect objects, analyze emotion, and translate transcriptsSearch & AskBuild a FAISS index over transcript, frames, and metadata for evidence-backed Q&AEditDetect highlights, export clips, and optionally assemble reelsStreamProcess webcams or RTMP/HTTP streams with adaptive segmentation and live Q&AOperateRetry transient failures, degrade optional skills gracefully, emit progress events, and run hooksVidify is ASR-first: subtitles and speech usually carry the main story, so visual model calls are skipped when transcript coverage is sufficient. See

Project Overview

for the full processing flow.

Try It in 2 Minutes

pip install -e . python -m agent.main analyze youtube "https://www.youtube.com/watch?v=..." --mode briefExample output shape:

{ "video": { "source": {"type": "youtube", "uri": "https://www.youtube.com/watch?v=..."}, "duration_sec": 1234.5, "resolution": {"w": 1920, "h": 1080} }, "timeline": { "chapters": [ {"start": 0.0, "end": 120.3, "title": "...", "summary": "..."} ], "events": [ { "start": 33.2, "end": 41.8, "text": "...", "evidence": {"asr_segment_ids": ["seg_12"], "frame_ids": ["f_0032"]} } ] }, "asr": { "segments": [ {"id": "seg_12", "start": 33.2, "end": 35.1, "text": "..."} ] } }Quick Start

1. Install

pip install -e .System requirements: Python 3.11+, ffmpeg, and yt-dlp.

Optional feature groups:

pip install -e ".[asr,ocr,emotion,live,serving]" pip install -r requirements-full.txt2. Configure

cp .env.example .envEdit .env when you need custom model endpoints, model names, cache paths, or web search credentials. Full details are in

Configuration

3. Start Model Serving

Vidify expects an OpenAI-compatible multimodal endpoint, usually vLLM:

# vLLM >= 0.19.0 is required for Qwen3.5 support. pip install "vllm>=0.19.0" bash scripts/serving_qwen3_5.shManual example:

vllm serve Qwen/Qwen3.5-9B \ --host 0.0.0.0 --port 8000 \ --max-model-len 65536 \ --reasoning-parser qwen3 \ --allowed-local-media-path $(pwd)/cacheSee

Deployment

for GPU, Ascend/NPU, Docker, and validation commands.

4. Run

CLI:

python -m agent.main analyze youtube "https://www.youtube.com/watch?v=..." --mode detailed python -m agent.main analyze local media/example.mp4 --mode brief python -m agent.main analyze local media/example.mp4 --mode ask --question "What changed?"REST API and web UI:

uvicorn server.app:app --host 0.0.0.0 --port 9000 curl -X POST http://localhost:9000/analyze \ -H 'Content-Type: application/json' \ -d '{"source_type":"youtube","uri":"https://www.youtube.com/watch?v=...","mode":"detailed"}'Open http://localhost:9000 for the web interface.

Workflow Modes

brief is the canonical lightweight mode. quick is still accepted as a legacy alias in the CLI and API.

ModeUse It ForExamplebriefFast ASR-first summarypython -m agent.main analyze youtube URL --mode briefdetailedOCR, object detection, emotion, translation, and richer timelinespython -m agent.main analyze youtube URL --mode detailedaskQuestion-answering over an indexed videopython -m agent.main analyze youtube URL --mode ask --question "What are the conclusions?"highlightsClip export and optional reelspython -m agent.main analyze youtube URL --mode highlightsreportStructured report generation, optionally with web searchpython -m agent.main analyze youtube URL --mode report --include-web-searchliveWebcam, RTMP, or HTTP stream understandingpython -m agent.main analyze local webcam --mode liveSee

Workflows

and

API Reference

for complete parameters and request schemas.

Hermes

This repo ships a Hermes-native skill at .agents/skills/media/vidify.

python -m agent.main hermes install-skillThe installer symlinks the skill into ~/.hermes/skills/media/vidify by default. Use --strategy copy for a standalone copy. The legacy openclaw/ skill remains available for older setups.

Testing

Run the fast test suite:

pytest tests/Validate against an existing model endpoint:

bash scripts/run_test_gpu.sh --api-base http://localhost:8000/v1 --video media/my_video.mp4 python scripts/test_all.py --video-path media/my_video.mp4 --api-base http://localhost:8000/v1See

Testing Guide

for focused tests, YouTube E2E validation, and hardware-specific notes.

Repository Layout

PathPurposeagent/core/Orchestration, schemas, events, hooks, retries, segmenting, and parallel executionagent/extensions/skills/Reusable video, audio, retrieval, and analysis unitsagent/extensions/workflows/User-facing workflow compositionagent/extensions/models/Model adapters and direct-loading helpersserver/FastAPI app, SSE endpoints, and web routestemplates/Web UI templatesscripts/Serving, validation, and demo helpersdocs/Architecture, workflow, deployment, and API documentationcache/Runtime artifacts; do not commit generated outputsDocumentation

DocumentContents

Project Overview

ASR-first design, capability map, and processing flow

Deployment

vLLM serving, GPU validation, Ascend/NPU helpers, and Docker

Live Streaming

Webcam/stream architecture, CLI/API usage, and config

Production Features

Retries, graceful degradation, parallelism, progress events, hooks, and logging

Architecture

Data models, cache structure, model interfaces, and orchestrator

Workflows