<  Back to the Polytechnique Montréal portal

Movie description

Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville and Bernt Schiele

Article (2017)

Published Version
Terms of Use: Creative Commons Attribution.
Download (2MB)
Cite this document: Rohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C., Larochelle, H., ... Schiele, B. (2017). Movie description. International Journal of Computer Vision, 123(1), p. 94-120. doi:10.1007/s11263-016-0987-1
Show abstract Hide abstract


Audio description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. We introduce the Large Scale Movie Description Challenge (LSMDC) which contains a parallel corpus of 128,118 sentences aligned to video clips from 200 movies (around 150 h of video in total). The goal of the challenge is to automatically generate descriptions for the movie clips. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in the challenges organized in the context of two workshops at ICCV 2015 and ECCV 2016.

Uncontrolled Keywords

Movie description; Video description; Video captioning; Video understanding; Movie description dataset; Movie description challenge; Long short-term memory network; Audio description; LSMDC

Open Access document in PolyPublie
Subjects: 2700 Technologie de l'information > 2700 Technologie de l'information
2700 Technologie de l'information > 2708 Traitement d'images et traitement vidéo
Department: Département de génie informatique et génie logiciel
Research Center: Non applicable
Funders: Max Planck Society, FITweltweit-Program of the German Academic Exchange Service (DAAD)
Date Deposited: 06 Dec 2018 12:14
Last Modified: 08 Apr 2021 10:43
PolyPublie URL: https://publications.polymtl.ca/3529/
Document issued by the official publisher
Journal Title: International Journal of Computer Vision (vol. 123, no. 1)
Publisher: Springer
Official URL: https://doi.org/10.1007/s11263-016-0987-1


Total downloads

Downloads per month in the last year

Origin of downloads


Repository Staff Only