Publications

Publication details [#55402]

Starr, Kim Linda, Sabine Braun and Jaleh Delfani. 2020. Taking a Cue From the Human: linguistic and visual prompts for the automatic sequencing of multimodal narrative. In Pedersen, Jan and Anna Matamala Ripoll, eds. Perspectives on Complex Understandings. Special issue of Journal of Audiovisual Translation 3 (2): 140–169.
Publication type
Article in Special issue
Publication language
English
Journal WWW

Abstract

Human beings find the process of narrative sequencing in written texts and moving imagery a relatively simple task. Key to the success of this activity is establishing coherence by using critical cues to identify key characters, objects, actions and locations as they contribute to plot development. In the drive to make audiovisual media more widely accessible (through audio description), and media archives more searchable (through content description), computer vision experts strive to automate video captioning in order to supplement human description activities. Existing models for automating video descriptions employ deep convolutional neural networks for encoding visual material and feature extraction (Krizhevsky, Sutskever, & Hinton, 2012; Szegedy et al., 2015; He, Zhang, Ren, & Sun, 2016). Recurrent neural networks decode the visual encodings and supply a sentence that describes the moving images in a manner mimicking human performance. However, these descriptions are “blind” to narrative coherence. This study examines the human approach to narrative sequencing and coherence creation using the MeMAD [Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy] film corpus involving five-hundred extracts chosen as stand-alone narrative arcs. The authors examine character recognition, object detection and temporal continuity as indicators of coherence, using linguistic analysis and qualitative assessments to inform the development of more narratively sophisticated computer models in the future.
Source : Based on abstract in journal