Video summarization reduces long footage to what matters: a written account of what happens, a list of the moments worth watching with their timecodes, or a shortened cut containing only those moments.
Most useful summaries come from two sources at once. What is said is recovered by transcription, which is cheap and precise; what is shown is recovered by sampling frames and describing them, which is expensive, so the sampling rate is the main cost decision in the whole pipeline. A language model then combines both into an account. For footage that is mostly speech — meetings, lectures, interviews — the transcript carries almost all the information. For surveillance and operational footage, where little is said, the visual side does the work.
Models in this group take a recorded video file as input and are applied frame by frame. Each specification table gives three hardware tiers — Minimum, the smallest setup on which the model runs correctly; Medium, the usual production configuration; and High, a configuration sized for peak volume. The rate is the number of sample inputs processed per hour on that hardware. Use these rates for initial sizing. Before production, benchmark your own data to validate accuracy, latency, throughput and cost. The sample input here is one minute of 1080p video at 25 frames per second.
Video-LLaMA 3
Vendor: Alibaba Cloud (DAMO Academy)
What it does: watches sampled video with its audio and writes an account of what happens, following your instructions about what to report. The most capable general option.
Requirement
Minimum
Medium
High
GPU type
RTX 4090 (reduced precision)
A100 80 GB
2× H100 80 GB
VRAM
16 GB
80 GB
160 GB combined
vCPUs
8
16
32
RAM
32 GB
64 GB
200 GB
Server
1× RTX 4090 24 GB
1× A100 SXM 80 GB
2× H100 SXM 80 GB
Rate (video minutes processed per hour)
≈ 25
≈ 90
≈ 220
InternVideo2
Vendor: Shanghai AI Laboratory
What it does: understands actions and events across a stretch of footage rather than describing single frames, which is what identifies the moments worth keeping.
Requirement
Minimum
Medium
High
GPU type
RTX 4090
A100 80 GB
2× A100 80 GB
VRAM
20 GB
80 GB
160 GB combined
vCPUs
8
16
32
RAM
32 GB
64 GB
128 GB
Server
1× RTX 4090 24 GB
1× A100 SXM 80 GB
2× A100 SXM 80 GB
Rate (video minutes processed per hour)
≈ 40
≈ 150
≈ 380
VideoMAE V2
Vendor: Nanjing University
What it does: classifies actions in short clips cheaply, used as a first pass to find the stretches that a heavier model should examine.
Requirement
Minimum
Medium
High
GPU type
RTX 3090
RTX 4090
A100 80 GB
VRAM
12 GB
24 GB
80 GB
vCPUs
8
12
24
RAM
32 GB
48 GB
96 GB
Server
1× RTX 3090 24 GB
1× RTX 4090 24 GB
1× A100 SXM 80 GB
Rate (video minutes processed per hour)
≈ 300
≈ 1,100
≈ 2,800
Qwen2.5-VL 7B / 72B
Vendor: Alibaba Cloud
What it does: describes individual sampled frames in detail, which is how specific objects, text and conditions in the footage reach the summary.
Requirement
Minimum
Medium
High
GPU type
RTX 4090 (7B, reduced precision)
A100 80 GB (7B, full precision)
2× H100 80 GB (72B model)
VRAM
16 GB
80 GB
160 GB combined
vCPUs
8
16
32
RAM
32 GB
64 GB
200 GB
Server
1× RTX 4090 24 GB
1× A100 SXM 80 GB
2× H100 SXM 80 GB
Rate (video minutes processed per hour)
≈ 40
≈ 160
≈ 90 (72B model, higher accuracy)
faster-whisper
Vendor: SYSTRAN
What it does: transcribes the speech, which for meetings and lectures carries most of the content at a fraction of the cost of visual analysis.
Requirement
Minimum
Medium
High
GPU type
RTX 3060
RTX 4090
A100 80 GB
VRAM
8 GB
24 GB
80 GB
vCPUs
6
12
24
RAM
16 GB
32 GB
64 GB
Server
1× RTX 3060 12 GB
1× RTX 4090 24 GB
1× A100 SXM 80 GB
Rate (video minutes processed per hour)
≈ 1,200
≈ 4,200
≈ 10,800
Llama 3.3 70B
Vendor: Meta
What it does: combines the transcript and the frame descriptions into the final account, with timecodes against each point so any claim can be checked against the footage.
Requirement
Minimum
Medium
High
GPU type
2× RTX 4090 (reduced precision)
H100 80 GB
4× H100 80 GB
VRAM
48 GB combined
80 GB
320 GB combined
vCPUs
16
24
64
RAM
64 GB
128 GB
512 GB
Server
2× RTX 4090 24 GB
1× H100 SXM 80 GB
4× H100 SXM 80 GB
Rate (video minutes processed per hour)
≈ 600
≈ 2,400
≈ 9,000
FFmpeg
Vendor: FFmpeg project
What it does: samples the frames, cuts the shortened version and writes the delivery file. No GPU needed, and the sampling choice here sets the cost of everything else.
Requirement
Minimum
Medium
High
GPU type
No GPU required
No GPU required
GPU-accelerated decode (NVENC/NVDEC)
VRAM
—
—
8 GB
vCPUs
2
8
16
RAM
4 GB
16 GB
32 GB
Server
CPU instance, 2 vCPU
CPU instance, 8 vCPU
1× RTX 4090 24 GB
Rate (video minutes processed per hour)
≈ 6,000
≈ 20,000
≈ 60,000
Choosing between them
The right pipeline depends on whether your footage is speech-led or visual, how long it is, and whether the output is a written account or a shortened cut. Our consultants review your material and the summary you need, then recommend the sampling rate and the models.
At the start of a project we may run a short proof of concept on a sample of your own data, measuring the accuracy and the throughput the model actually achieves on your material. That replaces the estimates on this page with real figures, so the cost and the schedule for the full engagement are known before it is committed.
Share a representative sample, expected volume, latency target and deployment location for Video summarization. We benchmark the shortlisted models, recommend the lowest-cost GPU configuration that meets the target, and scale it from pilot capacity to a dedicated production cluster.