Catchments, prosody and discourse
David McNeill | University of Chicago
Francis Quek | Wright State University
Karl-Erik McCullough | University of Chicago
Susan D. Duncan | Wright State University
Nobuhiro Furuyama | University of Chicago
Robert Bryll | Wright State University
Nobuhiro Furuyama | Wright State University
Rashid Ansari | University of Illinois
We present our work on the temporal integration of hierarchies of communicative actions: kinesic, prosodic and discursive. We use the device of the ‘catchment’ as the locus around which this integration proceeds. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. First, we process the video data to obtain the motion traces of both of the subject’s hands using the vector coherence mapping algorithm. We code the gestures to identify the catchments. We recover discourse purposes utilizing a system of guided questions. Finally, we define prosody in terms of the ToBI system. The results of these analyses are compared against the computed motion traces to identify the cues accessible in the gestural and audio data that correlate well with the psycholinguistic analyses. The results show that motion, prosody and discourse structure are integrated at each moment of speaking.
The electronic edition of this article includes audio-visual data.
Keywords: discourse, prosody, gesture groups, temporal integration, discourse purposes, ToBI, catchments
Published online: 06 June 2002
https://doi.org/10.1075/gest.1.1.03mcn
https://doi.org/10.1075/gest.1.1.03mcn
Cited by
Cited by 48 other publications
No author info given
2009.
In Multimodal Metaphor [Applications of Cognitive Linguistics [ACL], ],
Alibali, Martha W., Mitchell J. Nathan, Rebecca Boncoddo & Elizabeth Pier
2019. Managing common ground in the classroom: teachers use gestures to support students’ contributions to classroom discourse.
ZDM 51:2 ► pp. 347 ff.
Alibali, Martha W., Mitchell J. Nathan, Matthew S. Wolfgram, R. Breckinridge Church, Steven A. Jacobs, Chelsea Johnson Martinez & Eric J. Knuth
2014. How Teachers Link Ideas in Mathematics Instruction Using Speech and Gesture: A Corpus Analysis.
Cognition and Instruction 32:1 ► pp. 65 ff.
Arzarello, Ferdinando, Ornella Robutti & Mike Thomas
2015. Growth point and gestures: looking inside mathematical meanings.
Educational Studies in Mathematics 90:1 ► pp. 19 ff.
Arzarello, Ferdinando & Cristina Sabena
2014.
In Networking of Theories as a Research Practice in Mathematics Education [Advances in Mathematics Education, ],
► pp. 31 ff.
Becvar, Amaya, James Hollan & Edwin Hutchins
2008.
In Resources, Co-Evolution and Artifacts [Computer Supported Cooperative Work, ],
► pp. 117 ff.
Bryll, R., R.T. Rose & F. Quek
2005. Agent-Based Gesture Tracking.
IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 35:6 ► pp. 795 ff.
Campbell, Nick
2008.
In Springer Handbook of Speech Processing,
► pp. 505 ff.
Campbell, Nick
2008.
In Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction [Lecture Notes in Computer Science, 5042],
► pp. 107 ff.
Cohn, Neil
2016. A multimodal parallel architecture: A cognitive framework for multimodal interactions.
Cognition 146 ► pp. 304 ff.
Colletta, Jean-Marc, Ramona N. Kunene, Aurélie Venouil, Virginie Kaufmann & Jean-Pascal Simon
2009.
In Multimodal Corpora [Lecture Notes in Computer Science, 5509],
► pp. 54 ff.
Dick, Anthony Steven, Susan Goldin-Meadow, Uri Hasson, Jeremy I. Skipper & Steven L. Small
2009. Co-speech gestures influence neural activity in brain regions associated with processing semantic information.
Human Brain Mapping 30:11 ► pp. 3509 ff.
Feyereisen, Pierre
2006. Further investigation on the mnemonic effect of gestures: Their meaning matters.
European Journal of Cognitive Psychology 18:2 ► pp. 185 ff.
Flecha-García, María L.
2010. Eyebrow raises in dialogue and their relation to discourse structure, utterance function and pitch accents in English.
Speech Communication 52:6 ► pp. 542 ff.
FURUYAMA, NOBUHIRO
2002. Prolegomena of a theory of between-person coordination of speech and gesture.
International Journal of Human-Computer Studies 57:4 ► pp. 347 ff.
Jenkins, Theodore, Marie Coppola & Carl Coelho
2017. Effects of gesture restriction on quality of narrative
production.
Gesture 16:3 ► pp. 416 ff.
Kataoka, Kuniyoshi
2009. A multi-modal ethnopoetic analysis (Part 1): Text, gesture, and environment in Japanese spatial narrative.
Language & Communication 29:4 ► pp. 287 ff.
Kataoka, Kuniyoshi
2010. A multi-modal ethnopoetic analysis (Part 2): Catchment, prosody, and frames of reference in Japanese spatial narrative.
Language & Communication 30:2 ► pp. 69 ff.
Krause, Christina M. & Alexander Salle
2018.
In Signs of Signification [ICME-13 Monographs, ],
► pp. 293 ff.
Krivokapić, Jelena
2014. Gestural coordination at prosodic boundaries and its role for prosodic structure and speech planning processes.
Philosophical Transactions of the Royal Society B: Biological Sciences 369:1658 ► pp. 20130397 ff.
Krivokapić, Jelena, Mark K. Tiede & Martha E. Tyrone
2017. A Kinematic Study of Prosodic Structure in Articulatory and Manual Gestures: Results from a Novel Method of Data Collection.
Laboratory Phonology 8:1
Kröger, Bernd J., Peter Birkholz, Jim Kannampuzha, Emily Kaufmann & Irene Mittelberg
2011. Movements and Holds in Fluent Sentence Production of American Sign Language: The Action-Based Approach.
Cognitive Computation 3:3 ► pp. 449 ff.
Laparle, Schuyler
2021.
In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior [Lecture Notes in Computer Science, 12777],
► pp. 233 ff.
Lempert, Michael
2018. On the Pragmatic Poetry of Pose: Gesture, Parallelism, Politics.
Signs and Society 6:1 ► pp. 120 ff.
Levy, Elena T.
2009. The Mediation of Coherent Discourse by Kinesthetic Reenactment: A Case Study of an Autistic Adolescent, Part II.
Imagination, Cognition and Personality 29:1 ► pp. 41 ff.
Madeo, Renata C. B., Clodoaldo A. M. Lima & Sarajane M. Peres
2017. Studies in automated hand gesture analysis: an overview of functional types and gesture phases.
Language Resources and Evaluation 51:2 ► pp. 547 ff.
Martin, Jean-Claude, Patrizia Paggio, Peter Kuehnlein, Rainer Stiefelhagen & Fabio Pianesi
2008. Introduction to the special issue on multimodal corpora for modeling human multimodal behavior.
Language Resources and Evaluation 42:2 ► pp. 253 ff.
McNeill, David
2015.
In International Encyclopedia of the Social & Behavioral Sciences,
► pp. 109 ff.
Mcneill, David, Elena T. Levy & Susan D. Duncan
2015.
In The Handbook of Discourse Analysis,
► pp. 262 ff.
Mendoza-Denton, Norma & Stefanie Jannedy
2011. Semiotic Layering through Gesture and Intonation.
Journal of English Linguistics 39:3 ► pp. 265 ff.
Mittelberg, Irene
2007. Gesture & Thought. David McNeill, Chicago: Chicago University Press, 2005, 340 pages, $38.00 (hardback). ISBN: 0-226-51462-5..
Metaphor and Symbol 22:3 ► pp. 281 ff.
Mary-Annick Morel
2010. Déflexivité et décondensation dans le dialogue oral en français : marqueurs grammaticaux, intonation, regard et geste.
Langages n° 178:2 ► pp. 115 ff.
Muto, Yumiko, Shoji Takasugi, Tomohito Yamamoto & Yoshihiro Miyake
2009.
In RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication,
► pp. 1022 ff.
Müller, Cornelia & Tag, Susanne
2010. The Dynamics of Metaphor: Foregrounding and Activating Metaphoricity in Conversational Interaction.
Cognitive Semiotics 10:6 ► pp. 85 ff.
Nuessel, Frank
2007. Gesture and language.
Semiotica 2007:166
Payrató, Lluís
2003.
In Handbook of Pragmatics,
► pp. 1 ff.
Pouw, Wim & James A. Dixon
2020. Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles.
Discourse Processes 57:4 ► pp. 301 ff.
Quek, F.
2001.
In Proceedings 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery,
► pp. 80 ff.
Quek, F., R. Bryll, M. Harper, Lei Chen & L. Ramig
2001.
In Proceedings 2nd Annual IEEE International Symposium on Bioinformatics and Bioengineering (BIBE 2001),
► pp. 245 ff.
Quek, Francis, David McNeill, Robert Bryll, Susan Duncan, Xin-Feng Ma, Cemil Kirbas, Karl E. McCullough & Rashid Ansari
2002. Multimodal human discourse.
ACM Transactions on Computer-Human Interaction 9:3 ► pp. 171 ff.
Radford, Luis & Cristina Sabena
2015.
In Approaches to Qualitative Research in Mathematics Education [Advances in Mathematics Education, ],
► pp. 157 ff.
Rodríguez, Lydia
2022. Dickens in Chol.
Language and Cognition ► pp. 1 ff.
Sabena, Cristina
2018.
In Invited Lectures from the 13th International Congress on Mathematical Education [ICME-13 Monographs, ],
► pp. 541 ff.
Sambre, Paul & Kurt Feyaerts
2017. Embodied musical meaning-making and multimodal viewpoints in a trumpet master class.
Journal of Pragmatics 122 ► pp. 10 ff.
Stone, Matthew, Doug DeCarlo, Insuk Oh, Christian Rodriguez, Adrian Stere, Alyssa Lees & Chris Bregler
2004. Speaking with hands.
ACM Transactions on Graphics 23:3 ► pp. 506 ff.
Yasui, Eiko
2013. Collaborative idea construction: Repetition of gestures and talk in joint brainstorming.
Journal of Pragmatics 46:1 ► pp. 157 ff.
Yildirim, Serdar & Shrikanth Narayanan
2009. Automatic Detection of Disfluency Boundaries in Spontaneous Speech of Children Using Audio–Visual Information.
IEEE Transactions on Audio, Speech, and Language Processing 17:1 ► pp. 2 ff.
Yingen Xiong, F. Quek & D. McNeill
2002.
In Proceedings. Fourth IEEE International Conference on Multimodal Interfaces,
► pp. 179 ff.
This list is based on CrossRef data as of 09 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.