<  Back to the Polytechnique Montréal portal

Movie description

Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher J. Pal, Hugo Larochelle, Aaron Courville and Bernt Schiele

Article (2017)

Open Acess document in PolyPublie and at official publisher
Open Access to the full text of this document
Published Version
Terms of Use: Creative Commons Attribution
Download (1MB)
Show abstract
Hide abstract


Audio description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. We introduce the Large Scale Movie Description Challenge (LSMDC) which contains a parallel corpus of 128,118 sentences aligned to video clips from 200 movies (around 150 h of video in total). The goal of the challenge is to automatically generate descriptions for the movie clips. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in the challenges organized in the context of two workshops at ICCV 2015 and ECCV 2016.

Uncontrolled Keywords

Movie description; Video description; Video captioning; Video understanding; Movie description dataset; Movie description challenge; Long short-term memory network; Audio description; LSMDC

Subjects: 2700 Information technology > 2700 Information technology
2700 Information technology > 2708 Image and video processing
Department: Department of Computer Engineering and Software Engineering
Funders: Max Planck Society, FITweltweit-Program of the German Academic Exchange Service (DAAD)
PolyPublie URL: https://publications.polymtl.ca/3529/
Journal Title: International Journal of Computer Vision (vol. 123, no. 1)
Publisher: Springer
DOI: 10.1007/s11263-016-0987-1
Official URL: https://doi.org/10.1007/s11263-016-0987-1
Date Deposited: 06 Dec 2018 12:14
Last Modified: 11 Apr 2024 03:04
Cite in APA 7: Rohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C. J., Larochelle, H., Courville, A., & Schiele, B. (2017). Movie description. International Journal of Computer Vision, 123(1), 94-120. https://doi.org/10.1007/s11263-016-0987-1


Total downloads

Downloads per month in the last year

Origin of downloads


Repository Staff Only

View Item View Item