Welcome to VATEX Captioning Challenge!


Overview


We are pleased to announce the VATEX Captioning Challenge 2019! The challenge will be hosted at the 3rd Workshop on Closing the Loop Between Vision and Language, ICCV 2019.

Please stay tuned for more information!


Dates


  • Challenge Launch: Aug 6th, 2019.
  • CodaLab Submission Deadline: Oct 1st, 2019.
  • Report Submission Deadline: Oct 15th, 2019 23:59 UTC.
  • The winners will be announced at the 3rd CLVL workshop, ICCV 2019 on Oct 28th, 2019.

Prize


The 1st VATEX Captioning Challenge has ended! We plan to archive the competition results from CodaLab to the official VATEX website and further contribute to the vision-and-language research community. Meanwhile, we have several rewards for the winning teams, which will be announced at the 3rd CLVL workshop on Oct 28th, 2019. To be eligible for result archives and consideration for awards, we kindly request you to send the following information to vatex.org@gmail.com using your main contact email:

  • Team name.
  • Team members.
  • The username used in CodaLab submissions.
  • An arXiv link to a 2-4 page report, which describes your systems (including data processing, methods, experimental results, etc.) using the ICCV 2019 paper template. Code release is also encouraged to facilitate future research.
In addition to Outstanding Performance Awards that recognize top winners, we will also select the Best Method Award among all the reports based on the novelty and interestedness of methods. The deadline for the above steps is October 15th, 2019 23:59 UTC.


Guidelines


The VATEX dataset is a new large-scale multilingual video description dataset, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSRVTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. Please refer to our ICCV paper for more details. This VATEX Captioning Challenge aims to benchmark progress towards models that can describe the videos in various languages such as English and Chinese.

Dataset Download

Please refer to the details at the Download page. You can download English/Chinese captions and video features from the page.


Submission

The challenge is hosted at the CodaLab. Please go to the challenge page to submit your models.


Challenge Phases
  • Dev (English/Chinese): This phase evaluates algorithms on the VATEX validation set. We recommend using this Dev phase for algorithm validation. This phase should not be used for reporting results in the paper. A submission needs to consist of results on the entire validation set to be considered as a valid submission.
  • Test (English/Chinese): This phase evaluates algorithms on VATEX public test set. We recommend using this phase for reporting comparison numbers in academic papers. A submission needs to consist of results on the entire test set to be considered as a valid submission. This phase is aimed at the final evaluation of the model and one is not allowed to create multiple submissions using multiple teams.

Leaderboard


Offical Leaderboard

Note: This is the learderboard for tracking results of published papers and concluded challenges. Please email us if you would like to report your published results here. For real-time result tracking of ongoing challenges, please check the real-time leaderboard hosted on CodaLab.

♦  English Captioning

Rank Model BLEU-4 Meteor Rouge-L CIDEr
1 Baseline Shared Encoder

(Wang et al. ICCV 2019)

28.4 21.7 47.0 45.1
2 Naive Baseline

(Wang et al. ICCV 2019)

28.1 21.6 46.9 44.3
3 Baseline Shared Encoder-Decoder

(Wang et al. ICCV 2019)

27.9 21.6 46.8 44.2
♦  Chinese Captioning

Rank Model BLEU-4 Meteor Rouge-L CIDEr
1 Baseline Shared Encoder-Decoder

(Wang et al. ICCV 2019)

24.9 29.8 51.7 35.0
2 Baseline Shared Encoder

(Wang et al. ICCV 2019)

24.9 29.7 51.6 34.9
3 Naive Baseline

(Wang et al. ICCV 2019)

24.9 29.7 51.5 34.7