VideoAdAgent Bench v1

Headline — all arms as 0–1 scores

Why we built this benchmark

Until now, there has not been a public benchmark that grades AI video systems on the end-to-end task of producing a finished ad. Existing video benchmarks — VBench, Video Arena, EvalCrafter — measure per-frame fidelity on single-shot text-to-video output. None of them measure whether the output would function as an ad: hook strength, brand alignment, on-screen typography, hallucination cost, production polish. VideoAdAgent Bench fills that gap.

A stunning eight-to-twelve-second clip is not an ad. An ad has to hold a brand at the right angle, render the product name without garbling letters, build a narrative arc with a hook in the first three seconds, hit a fixed duration and aspect ratio, carry voiceover or music, and survive a production QA review without product hallucinations, anatomy glitches, or text artifacts that would force a re-roll. Today's text-to-video models do not attempt most of these problems. The agents that wrap them either do, badly, or skip them entirely. The point of this benchmark is to measure exactly that gap.

Mirroring real ad production

Each brief is structured the way our internal creative team writes one for production: product description, brand guidelines, narrative intent (problem-solution / before-after / demo / testimonial / lifestyle), platform constraints (duration, aspect ratio, required on-screen text strings), and the input assets the agent should ground itself in — logo, product reference image, persona reference. The agent receives the brief and returns a finished MP4. We grade the MP4 the way a creative director grades a campaign before approval: not against a checklist of features, but against whether the output is shippable.

The brief set spans every major ad-production pattern plus targeted stress tests against the hardest axes of multi-scene production: anchor storytelling, persona consistency, brand-asset reuse, structured CTA typography, and persuasion-arc compliance. Every brief uses fictional brands and AI-generated product mocks to avoid real-customer IP. The full set is pre-registered — published to a tagged commit before a single output was generated — so no system in the benchmark could have been tuned to the specific briefs.

Arm 1 — Agent vs Agent (Creatify vs HeyGen V3, Luma)

Strictly agent-vs-agent comparison. Both systems get the same brief and produce a finished ad; LLM judges (Opus 4.7 + GPT-5) see the brief and score 8 ad-rubric dimensions. Win-rate = % of pairs Creatify or HeyGen wins outright (consensus across both judges + both seat positions). Per-dimension scores = average score the judges gave each system across all pairs (un-swapped via position flag).

Arm 2 — Ad Quality (Creatify vs raw T2V baselines)

Multi-judge pairwise where judges see the brief and score 8 ad-rubric dimensions: ad_effectiveness, brand_alignment, visual_quality, motion_quality, hook_strength, cta_clarity, narrative_arc, overall_preference. Per-dimension scores below are per-system averages across every pair the system appeared in (un-swapped via position flag). Competing agents (HeyGen V3, Luma) are evaluated separately in Arm 1; this arm contains only Creatify and the raw text-to-video foundation-model baselines.

Two baseline variants per foundation model. Single-scene mirrors how Veo 3.1, Kling 3.0, and Seedance 2.0 are actually used today: the brief becomes a single prompt, the model returns one 8–12s clip. Multi-scene is a fairer variant we built for the baselines: an LLM decomposes the brief into per-scene prompts (the same decomposition step Creatify Agent does internally), each scene is generated separately, and the clips are concatenated. The multi-scene variant gives the foundation models access to the same scene-planning capability the agentic systems have, so the remaining gap is everything beyond scene planning — typography, audio, brand grounding, hallucination control. Both variants appear in the table below; the single-scene rows surface what raw foundation-model output looks like in its natural usage pattern.

Arm 3 — Video Production Quality (Creatify vs raw T2V baselines)

Arm 3 grades the finished video as a deliverable, separate from how well it answers the brief. Two complementary measurements: (1) a hallucination & visual-defect score from Claude Opus 4.7 vision on 15 frames per video — the dominant signal for whether output is shippable to a paying advertiser; and (2) deterministic mechanical-compliance checks on the rendered MP4 — duration, aspect, audio, on-screen typography — the kind of checks an ad-ops team would run before a campaign launch. These blend into the Arm 3 score with the hallucination component weighted 3× (it is the dominant defect signal). Per-frame visual-quality LLM-judge dims (aesthetic, imaging, dynamic motion) are reported on the radar chart above but excluded from the Arm 3 composite — they penalize multi-scene structure by construction (a single continuous 8-second shot scores higher than a 3-cut 30-second ad regardless of which is the better deliverable). Competing agents (HeyGen V3, Luma) are evaluated head-to-head against Creatify under Arm 1, and their Arm 3 production scores are also reported here for completeness.

Conclusion

What's next

v2 will scale the benchmark along two axes. First, more briefs covering vertical-specific ad patterns: e-commerce, app install, B2B SaaS, healthcare, education. Second, a closed-loop track that measures the value of multi-turn revision — the production experience single-shot bench numbers can't capture. We are also folding campaign-level signals (CTR, ROAS) into a domain-specific AdReward to drive DPO-style alignment of the underlying I2V models, replacing the static rubric with a learned, ad-effectiveness-calibrated judge.

Acknowledgements

Creatify Agent and this benchmark stand on the shoulders of a community of researchers working on multi-agent video, video-quality evaluation, ad-effectiveness rubrics, and closed-loop quality assurance. Notable threads we drew on:

• Multi-agent video generation: Mora, VideoDirectorGPT, FilmAgent, StoryAgent, VISTA.
• Video-quality benchmarks: VBench / VBench-2.0 for per-frame quality dim definitions; VideoGen-Eval for agent-based evaluation methodology; Artificial Analysis Video Arena for pairwise + position-bias control patterns.
• Reward modeling for video: VideoReward / Flow-DPO, UnifiedReward — these inform the trajectory we expect Creatify Agent to take next: turning campaign-level CTR/ROAS into a domain-specific AdReward that drives DPO-style alignment of the underlying generation models.
• Ad-specific rubrics: Springboards Creativity Benchmark, AdsQA, the Pitt Ads Dataset, "Decoding the Hook" for first-3s evaluation.
• Closed-loop quality assurance: Meta-Harness (Lee, Khattab, Finn et al.) on end-to-end optimization of evaluation harnesses; Learning Beyond Gradients (Weng) on critique-and-revise loops over explicit policies as a durable alternative to gradient-based fine-tuning.

For the full architecture of the Creatify Agent system, the design choices behind the closed-loop QA gate, and the production examples behind these numbers, read the companion technical report: Creatify Agent — a closed-loop agent built for video advertising. The benchmark itself — briefs, baseline runners, scorers, and aggregation — is open-source under Apache 2.0 at github.com/creatify-ai/VABench.