VBench Leaderboard

"Which Video Generation Model is better?"
🏆 Welcome to the leaderboard of the VBench! 🎦 A Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 Spotlight)

Comprehensive Dimensions: We carefully decompose video generation quality into 16 comprehensive dimensions to reveal individual model's strengths and weaknesses.
Human Alignment: We conducted extensive experiments and human annotations to validate robustness of VBench.
Valuable Insights: VBench provides multi-perspective insights useful for the community.

Join Leaderboard: Please see the instructions for 3 options to participate. One option is to follow VBench Usage info, and upload the generated result.json file here. After clicking the Submit here! button, click the Refresh button. Model Information: What are the details of these Video Generation Models? See HERE

Credits: This leaderboard is updated and maintained by the team of VBench Contributors.

Model-Unnamed: 0_level_1	Animals-mAM	Animals-mLGM	Nature-mAM	Nature-mLGM	Shows-mAM	Shows-mLGM	Daily Life-mAM	Daily Life-mLGM	Sports-mAM	Sports-mLGM	Entertainments-mAM	Entertainments-mLGM	Vehicles-mAM	Vehicles-mLGM	Indoor-mAM	Indoor-mLGM	Tutorial-mAM	Tutorial-mLGM
Video-CCAM-v1.2	29.18	42.13	31.04	45.14	26.96	38.39	26.61	37.82	24.35	36.36	28.47	46.41	26.89	37.29	24.01	31.16	16.68	56.62

Model-Unnamed: 0_level_1	Animals-mAM	Animals-mLGM	Nature-mAM	Nature-mLGM	Shows-mAM	Shows-mLGM	Daily Life-mAM	Daily Life-mLGM	Sports-mAM	Sports-mLGM	Entertainments-mAM	Entertainments-mLGM	Vehicles-mAM	Vehicles-mLGM	Indoor-mAM	Indoor-mLGM	Tutorial-mAM	Tutorial-mLGM
GPT-4o	29.18	42.13	31.04	45.14	26.96	38.5	26.61	37.82	26.5	36.36	28.47	46.41	26.89	37.29	24.01	31.16	16.68	22.3
Gemini-2-Flash	27.78	36.26	23.42	28.58	28.79	38.39	23.73	29.81	24.35	31.26	33.35	52	24.8	31.59	22.28	27.43	38.33	56.62
Video-Llama3	21.43	26.1	24.37	31.35	24.13	31.12	18.74	22.1	22.41	28.51	24.7	31.58	23.15	29.17	20	24.43	24.68	33.74
Qwen2.5-VL	26.32	36.26	27.07	38.12	23.74	32.13	26.1	35.92	23.24	31.39	20.52	27.36	29.94	33.52	25.61	35.04	2.76	2.87
Qwen2-VL	16.57	18.34	16.46	18.37	21.85	27.75	12.8	13.81	16.34	18.59	20.75	24.82	16.41	18.59	15.1	16.69	17.29	22.23
InternVL-2.5	20.98	28.59	17.74	22.44	20.1	28.76	17.7	22.67	18.86	24.98	15.34	18.59	20.31	27.1	17.72	22.49	9.35	10.76
Llava-Video	20.34	25.96	24.57	33.88	25.03	37.03	20.08	25.58	21.33	28.78	20.24	27.1	23.76	34.2	20.07	25.4	21.97	32.82
VideoChat2	18.5	22.58	20.23	25.05	19.11	24.67	16.03	18.98	16.88	20.23	18.27	21.46	15.91	18.93	15.68	18.65	7.09	7.6
Oryx-1.5	13.58	14.7	11.05	11.83	20	24.4	9.82	10.39	14.39	16.03	16.14	18.85	14.09	15.43	14.3	15.6	10.34	11.78
Video-CCAM-v1.2	23.47	39.49	26.19	50.25	28.54	56.27	19.96	29.73	21.99	35.18	16.42	22.06	25.59	46.08	21.22	32.89	15.03	19.89
TimeChat	18.26	22.52	12.61	14.14	13.75	15.86	12.48	14.02	13.63	15.64	11.17	12.27	15.46	18.22	13.7	15.64	6.05	6.53
VTimeLLM	14.54	16.93	20.28	26.24	22.93	33.5	20.33	27.43	16.12	19.99	15.65	18.46	18.12	23.4	18.36	22.88	8.67	9.56
TRACE	12.35	13.77	10.28	11.23	18.85	22.75	8.59	9.25	15.31	17.41	13.99	15.72	12.87	14.32	10.91	11.92	15.19	17.34
Sa2VA	21.01	27.03	13.62	16.6	23.38	29.13	17.89	23.25	17.01	19.69	13.28	14.9	21.36	26.25	18.44	22.96	10.91	11.97

VBench, a comprehensive benchmark suite for video generative models. We design a comprehensive and hierarchical Evaluation Dimension Suite to decompose "video generation quality" into multiple well-defined dimensions to facilitate fine-grained and objective evaluation. For each dimension and each content category, we carefully design a Prompt Suite as test cases, and sample Generated Videos from a set of video generation models. For each evaluation dimension, we specifically design an Evaluation Method Suite, which uses carefully crafted method or designated pipeline for automatic objective evaluation. We also conduct Human Preference Annotation for the generated videos for each dimension, and show that VBench evaluation results are well aligned with human perceptions. VBench can provide valuable insights from multiple perspectives.

VBench Leaderboard

"Which Video Generation Model is better?"
🏆 Welcome to the leaderboard of the VBench! 🎦 A Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 Spotlight)

Comprehensive Dimensions: We carefully decompose video generation quality into 16 comprehensive dimensions to reveal individual model's strengths and weaknesses.
Human Alignment: We conducted extensive experiments and human annotations to validate robustness of VBench.
Valuable Insights: VBench provides multi-perspective insights useful for the community.

Credits: This leaderboard is updated and maintained by the team of VBench Contributors.

Submit on VBench Benchmark Introduction

🎈

Please note that you need to obtain the file evaluation_results/*.json by running VBench in Github. You may conduct an Offline Check before uploading.
Then, pack these JSON files into a ZIP archive, ensuring that the top-level directory of the ZIP contains the individual JSON files.
Finally, upload the ZIP archive below. ⚠️ Uploading generated videos or images of the model is invalid! ⚠️ Submissions that do not correctly fill in the model name and model link may be deleted by the VBench team. The contact information you filled in will not be made public.

✉️✨ Submit your i2v model evaluation json file here!

Here is a required field

Model name

Revision Model Name(Optional)

Please select the way user can access your model. You can update the content by revision_name, or contact the VBench Team.

Project Page/Paper Link/Github/HuggingFace Repo

Your Team Name(If left blank, it will be user upload)

E-Mail(Will not be displayed)

The following is optional and will be synced to [GitHub] (https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models)

Time of Publish

resolution

model fps

model frame count

model video length

model checkpoint

github commit id

pipeline format

Click to Upload a ZIP File