Cover not available
Part of
Linguistic Foundations of Narration in Spoken and Sign Languages
Edited by Annika Hübl and Markus Steinbach
[Linguistik Aktuell/Linguistics Today 247] 2018
► pp. 223–250

Ways of expressing action in multimodal narrations – the semiotic complexity of character viewpoint depictions

| Chemnitz University of Technology
| European University Viadrina
| European University Viadrina

Based on an analysis accounting for the whole body as a possible articulator in the depiction of actions, this chapter argues for an expansion of the notion of ‘character viewpoint gestures’ to a notion of ‘multimodal action depiction from a character viewpoint’. Our study shows that speakers may deploy only single articulators, providing a semantically reduced depiction of the action, or they may deploy more bodily articulators and give a semantically rich picture of the event narrated. Our findings suggest a continuum of semiotic complexity, capturing the range of bodily involvement from less pantomimic (single articulators involved) to pantomimic depictions (more articulators involved) of actions. The paper closes by discussing our observations with respect to the notion of ‘constructed action’ and ‘role shift’ in sign languages and by giving some general remarks on the multimodal analysis of narrations.

Article outline
  • 1.Introduction
  • 2.Analyzing multimodal narratives
  • 3.Exemplification of the method
    • Step 1Description of articulators involved in the character viewpoint depiction
    • Step 2Meaning analysis of body movements
      • Step 2.1Analysis of meaning conveyed by each articulator involved in the character-viewpoint depictions
      • Step 2.2Analysis of the meaning in the verbal and interactive context
    • Step 3Determination of degree of semiotic complexity
  • 4.Ways of expression action in multimodal narrations
    • 4.1Character viewpoint depiction involving a single articulator
    • 4.2Character viewpoint depiction involving two articulators
    • 4.3Character viewpoint depiction involving several articulators
  • 5.Continuum of semiotic complexity
  • 6.Concluding thoughts: Iconicity of action depiction and constructed action in gesture and sign language
  • Acknowledgements
  • Notes
  • References
Published online: 25 May 2018
DOI logo
https://doi.org/10.1075/la.247.10bre
References
Beattie, G. & Shovelton, H.
2001 How gesture viewpoint influences what information decoders receive from iconic gestures. In Oralite et gestualite: Interactions et comportements multimodaux dans la communication, C. Cave, I. Guaitella & S. Santi (eds), 283–287. Paris: L’Harmattan.Google Scholar
2002 An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology 93(2): 179–192.DOI logoGoogle Scholar
Birdwhistell, R.
1970 Kinesics and Context. Essays on Body, Motion, Communication. Philadelphia PA: University of Pennsylvania Press.Google Scholar
Bressem, J.
2013 A linguistic perspective on the notation of form features in gestures. In C. Müller et al. (eds), 1079–1098.Google Scholar
2015 Repetition als Mittel der Musterbildung bei redebegleitenden Gesten. In Handbuch Satz, Äußerung, Schema, C. Dürscheid & J. G. Schneider (eds), 422–442. Berlin: Walter de Gruyter.DOI logoGoogle Scholar
Bressem, J., Ladewig, S. H. & Müller, C.
2013 Linguistic Annotation System for Gestures (LASG). In Müller et al.. (eds), 1098–1125.Google Scholar
Bressem, J. & Ladewig, S. H.
2011 Rethinking gesture phases – Articulatory features of gestural movement? Semiotica 184(1–4): 53–91.Google Scholar
Bressem, J. & Müller, C.
2014 The family of AWAY-gestures: Negation, refusal and negative assessement. In Müller et al.. (eds), 1592–1605.Google Scholar
Bühler, K.
1934 Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart: Fischer.Google Scholar
Casey, S. & Emmorey, K.
2009 Co-speech gesture in bimodal bilinguals. Language and Cognitive Processes 24(2): 290–312.DOI logoGoogle Scholar
Cassell, J. & McNeill, D.
1991 Gesture and the poetics of prose. Poetics Today 12(3): 375–404.DOI logoGoogle Scholar
Cormier, K., Quinto-Pozos, D., Sevcikova, Z. & Schembri, A.
2012 Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures. Language and Communication 32: 329–348.DOI logoGoogle Scholar
Davis, M.
1979 Laban analysis of nonverbal communication. In Nonverbal Communication: Readings with Commentary, S. Weitz (ed.), 182–206. Oxford: OUP.Google Scholar
Dudis, P. G
2004 Body partitioning and real-space blends. Cognitive Linguistics 15(2): 223–238.DOI logoGoogle Scholar
Dudis, P. G.
2007 Types of depiction in ASL. Ms, Gallaudet University. [URL]Google Scholar
Duncan, S.
2005 Gesture in signing: A case study in Taiwan Sign Language. Language and Linguistics 6(2): 279–318.Google Scholar
Eibl-Eibesfeldt, I.
1971 Transcultural patterns of ritualized contact behavior. In Behavior and Environment. The Use of Space by Animals and Men, A. H. Esser (ed.), 238–246. New York NY: Plenum.Google Scholar
Ekman, P., Sorenson, E. R. & Friesen, W.
1969 Pan-cultural elements in facial displays of emotions. Science 164(3875): 86–88.DOI logoGoogle Scholar
Ekman, P., Friesen, W. & Hager, J. C.
2002 Facial Action Coding System. Salt Lake City UT: Research Nexus.Google Scholar
Frey, S., Hirsbrunner, H. P., Pool, J. & Daw, W.
1981 Das Berner System zur Untersuchung nonverbaler Interaktion: I. Die Erhebung des Rohdatenprotokolls; II. Die Auswertung von Zeitreihen visuell-auditiver Information. In Methoden der Analyse von Face-to-Face-Situationen, P. Winkler (ed.), 203–268. Stuttgart: Metzler.DOI logoGoogle Scholar
Goodwin, C.
1981 Conversational organization: Interaction between speakers and hearers. New York: Academic Press New York.Google Scholar
Gumperz, J.
1992 Contextualization revisited. In The Contextualization of Language [Pragmatics & Beyond New Series 22], P. Auer & A. di Luzio (eds), 39–54. Amsterdam: John Benjamins.DOI logoGoogle Scholar
Gülich, E.
1970 Makrosyntax der Gliederungssignale im gesprochenen Französisch. Munich: Fink.Google Scholar
Gülich, E. & Hausendorf, H.
2000 Vertextungsmuster Narration. In Text-und Gesprächslinguistik/Linguistics of Text and Conversation, K. Brinker, G. Antos, W. Heinemann & S. F. Sager (eds), 369–385. Berlin: Walter de Gruyter.Google Scholar
Günthner, S.
1996 The prosodic contextualization of moral work: An analysis of reproaches in ‘why’-formats. In Prosody in Conversation, E. Couper-Kuhlen & M. Selting (eds), 271–302. Cambridge: CUP.DOI logoGoogle Scholar
Harrison, S.
2009 The expression of negation through grammar and gesture. In Studies in Language and Cognition, J. Zlatev, M. Andren, M. J. Falck & C. Lundmark (eds), 405–409. Cambridge: CUP.Google Scholar
2014 Head shakes: Variation in form, function, and cultural distribution of a head movement related to “no”. In C. Müller et al.. (eds), 1496–1501.Google Scholar
Hermann, A. & Pendzich, N.-K.
2014 Nonmanual gestures in sign languages. In C. Müller et al.. (eds), 2149–2163.Google Scholar
Herrmann, A. & Steinbach, M.
2012 Quotation in sign languages. In Quotatives: Cross-linguistic and Cross-disciplinary Perspectives [Converging Evidence in Language and Communication Research 15], I. van Alphen & I. Buchstaller (eds), 203–228. Amsterdam: John Benjamins.DOI logoGoogle Scholar
Jakobson, R.
1966 Selected writings. Berlin: Mouton de Gruyter.Google Scholar
Keim, I.
1993 Sprachvariation als konstitutives Merkmal eines sozialen Stils. Am Beispiel einer innerstädtischen Welt „Kleiner Leute“. Soziolinguistik. Der Deutschunterricht 3: 44–56.Google Scholar
Kendon, A.
1967 Some functions of gaze-direction in social interaction. Acta Psychologica 26: 22–63.DOI logoGoogle Scholar
1980 Gesture and speech: Two aspects of the process of utterance. In Nonverbal Communication and Language, M. R. Key (ed.), 207–227. The Hague: Mouton.Google Scholar
1992 Some recent work from Italy on quotable gestures (emblems). Journal of Linguistic Anthropology 2(1): 92–108.DOI logoGoogle Scholar
2000 Language and gesture: Unity or duality. In Language and Gesture, D. McNeill (ed.), 47–63. Cambridge: CUP.DOI logoGoogle Scholar
Kendon, A
2004 Gesture: Visible Action as Utterance. Cambridge: CUP.DOI logoGoogle Scholar
Kendon, A. & Ferber, A.
1973 A description of some human greetings. In Comparative Ecology and Behavior of Primates, R. P. Michael & J. H. Crook (eds), 591–668. New York NY: Academic Press.Google Scholar
Kita, S. & Özyürek, A.
2003 What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48: 16–32.DOI logoGoogle Scholar
Knapp, M., Hall, J. & Horgan, T.
2013 Nonverbal Communication in Human Interaction. Boston MA: Wadsworth Cengage Learning.Google Scholar
Ladewig, S. H.
2010 Beschreiben, suchen und auffordern – Varianten einer rekurrenten Geste. Sprache und Literatur 41(105): 89–111.Google Scholar
2011 Putting the cyclic gesture on a cognitive basis. CogniTextes 6. [URL] (June 2013).DOI logoGoogle Scholar
2014c Creating multimodal utterance. The linear integration of gestures into speech. In: Müller, C. et al. (eds), 1662–1677.Google Scholar
Ladewig, S. H. & Bressem, J.
2013 New insights into the medium hand – Discovering structures in gestures based on the four parameters of sign language. Semiotica 197: 203–231.Google Scholar
Ladewig, S. H.
forthcoming). Integrating Gestures – cognitive grammar multimodal. Berlin: De Gruyter Mouton.
Ladewig, S. H
2014a Recurrent gestures. In C. Müller et al.. (eds), 1558–1575.Google Scholar
2014b The cyclic gesture. In C. Müller et al.. (eds), 1605–1618.Google Scholar
Langacker, R.
2008 Cognitive Grammar: A Basic Introduction. Oxford: OUP.DOI logoGoogle Scholar
Lausberg, H. & Sloetjes, H.
2009 Coding gestural behavior with the NEUROGES – ELAN system. Behavioral Research Methods 41(3): 841–849.DOI logoGoogle Scholar
van Loon, E., Pfau, R. & Steinbach, M.
2014 The grammaticalization of gestures in sign languages. In C. Müller et al. (eds), 2133–2149.Google Scholar
Mandel, M.
1977 Iconic devices in American Sign Language. In On the other hand. New perspectives on American Sign Language, L. A. Friedman (ed.) New York: Academic PressGoogle Scholar
Mayerthaler, W.
1980 Ikonismus in der Morphologie. Zeitschrift für Semiotik 2: 19–37.Google Scholar
McClave, E. Z.
2000 Linguistic functions of head movements in the context of speech. Journal of Pragmatics 32: 855–878.DOI logoGoogle Scholar
McNeill, D.
1992 Hand and Mind. What Gestures Reveal about Thought. Chicago IL: University of Chicago Press.Google Scholar
2005 Gesture and Thought. Chicago IL: University of Chicago Press.DOI logoGoogle Scholar
Mittelberg, I.
2006 Metaphor and Metonymy in Language and Gesture: Discoursive Evidence for Multimodal Models of Grammar. PhD dissertation, Cornell University. Ann Arbor MI: ProQuest/UMI. [URL]Google Scholar
2014 Gestures and iconicity. In C. Müller et al.. (eds), 1712–1732.Google Scholar
Mittelberg, I. & Waugh, L.
2014 Gestures and metonymy. In C. Müller et al.. (eds), 1747–1766.Google Scholar
Müller, C.
1998 Redebegleitende Gesten: Kulturgeschichte, Theorie, Sprachvergleich. Berlin: Arno Spitz.Google Scholar
2004 Forms and uses of the Palm Up Open Hand. A case of a gesture family? In Semantics and Pragmatics of Everyday Gestures, C. Müller & R. Posner (eds), 233–256.Berlin: Weidler VerlagGoogle Scholar
Müller, C
2008 Metaphors. Dead and Alive, Sleeping and Waking. A Dynamic View. Chicago, University of Chicago Press.DOI logoGoogle Scholar
Müller, C.
2010 Wie Gesten bedeuten. Eine kognitiv-linguistische und sequenzanalytische Perspektive. Sprache und Literatur 41(105): 37–68.Google Scholar
2013 Gestures as a medium of expression: The linguistic potential of gestures. In C. Müller et al.. (eds), 202–217.Google Scholar
2014a Gestural modes of representation as techniques of depiction. In Müller et al.. (eds), 1677–1687. Berlin: De Gruyter Mouton.Google Scholar
2014b Gesture as “deliberate expressive movement.” In From Gesture in Conversation to Visible Action as Utterance: Essays in Honor of Adam Kendon, M. Seyfeddinipur & M. Gullberg (eds), 127–152. Amsterdam: John Benjamins.Google Scholar
Müller, C., Bressem, J. & Ladewig S. H.
2013 Towards a grammar of gesture: A form-based view. In Müller et al.. (eds), 707–733.Google Scholar
C. Müller, C., Ladewig, S. H. & Bressem, J.
2013 Gestures and speech from a linguistic perspective: A new field and its history. In C. Müller et al.. (eds), 55–81.Google Scholar
Müller, C., Ladewig S. H. & Teßendorf S
2010 Structures of meaning and reference – Investigating gestures of the concrete and abstract. Paper presented at the 4th Conference of the International Society for Gesture Studies, Frankfurt/Oder, Germany.Google Scholar
Müller, C. & Tag, S.
2010 The dynamics of metaphor: Foregrounding and activating metaphoricity in conversational interaction. Cognitive Semiotics 6: 85–120.DOI logoGoogle Scholar
Müller, C., Cienki, A, Fricke, E., Ladewig, S. H., McNeill, D. & Teßendorf, S.
(eds) 2013 Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 1. Berlin, Boston: De Gruyter Mouton.Google Scholar
Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem
(eds) 2014 Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 2. Berlin, Boston: De Gruyter: Mouton.Google Scholar
Parrill, F.
2009 Dual viewpoint gestures. Gesture 9(3): 271–289.DOI logoGoogle Scholar
2010 Viewpoint in speech-gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 25(5): 650–668.DOI logoGoogle Scholar
Peirce, C. S.
1960 Collected Papers of Charles Sanders Peirce (1931–1958), Vol. I: Principles of Philosophy, Vol. II: Elements of Logic , C. Hartshorne & P. Weiss (eds). Cambridge MA: The Belknap of Harvard University Press.Google Scholar
Pfau, R. & Steinbach, M.
2006 Modality-independent and modality specific aspects of grammaticalization in sign languages. Linguistics in Potsdam 24: 3–98. [URL]Google Scholar
Pörings, R. & Schmitz, U.
2003 Sprache und Sprachwissenschaft. Eine kognitiv orientierte Einführung. Tübingen: Gunter Narr.Google Scholar
Quasthoff, U.
1979 Konversationelle Erzählungen im Sozialamt: Ihr Einsatz als kommunikative Strategien. In Soziologische Analysen. Referate aus den Veranstaltungen der Sektionen der deutschen Gesellschaft für Soziologie und der ad-hoc-Gruppen beim 19. Deutschen Soziologentag, R. Mackensen & F. Sagebiel (eds), 515–534. Berlin: Deutsche Gesellschaft für Soziologie (DGS).Google Scholar
2000 Erzählen als interaktive Gesprächsstruktur. In Text- und Gesprächslinguistik/Linguistics of Text and Conversation, K. Brinker, G. Antos, W. Heinemann & S. F. Sager (eds), 1293–1309. Berlin: Walter de Gruyter.Google Scholar
Quinto-Pozos, D.
2007 Why does constructed action seem obligatory? An analysis of “classifiers” and the lack of articulator-referent correspondence. Sign Language Studies 7(4): 458.DOI logoGoogle Scholar
2014 Enactment as a (signed) language communicative strategy. In C. Müller et al.. (eds), 2163–2170.Google Scholar
Quinto-Pozos, D. & Parrill, F.
2012 Comparing viewpoint strategies used by co-speech gesturers and signers. Paper presented at the 5th Conference of the International Society for Gesture Studies, Lund, Sweden.Google Scholar
Selting, M.
1995 Sprechstile als Kontextualisierungshinweise. In Stilfragen. Jahrbuch 1994 des Instituts für deutsche Sprache, G. Stickel (ed.), 225–256. Berlin: Walter de Gruyter.Google Scholar
Selting, M., Auer, P., Barth-Weingarten, D., Bergmann, J., Bergmann, P., Birkner, K., Couper-Kuhlen, E., Deppermann, E., Gilles, P., Günther, S., Hartung, M., Kern, F., Mertzlufft, C., Meyer, C., Morek, M., Oberzaucher, F., Peter, L., Quasthoff, U. M., Schütte, W., Stukenbrock A. & Uhmann, S.
2009 Gesprächsanalytisches Transkriptionssystem 2 (GAT 2). Gesprächsforschung – Online Zeitschrift zur verbalen Interaktion 10: 353–402.Google Scholar
Seyfeddinipur, M.
2004 Meta-discursive gestures from Iran: Some uses of the ‘Pistol Hand’. In The Semantics and Pragmatics of Everyday Gestures, C. Müller & R. Posner (eds), 205–216. Berlin: Weidler.Google Scholar
Streeck, J
2008 Depicting by gesture. Gesture 8(3): 285–301.DOI logoGoogle Scholar
Sweetser, E.
2012 Introduction: Viewpoint and perspective in language and gesture, from the Ground down. In Viewpoint in Language: A Multimodal Perspective, B. Dancygier & E. Sweetser (eds), 1–22. Cambridge: CUP.DOI logoGoogle Scholar
Sweetser, E. & Stec, K.
2013 Managing multiple viewpoints: Coordinating embedded perspective in multimodal narratives. Paper presented at the ICLC-12, Edmonton, Canada.Google Scholar
Taub, S.
2001 Language from the Body: Iconicity and Metaphor in American Sign Language. Cambridge: CUP.DOI logoGoogle Scholar
Teßendorf, S.
2014 Pragmatic and metaphoric gestures – Combining functional with cognitive approaches. In C. Müller et al.. (eds), 1540–1558.Google Scholar
2016 Actions as sources of gestures. In Beyond Language Boundaries: Multimodal Use in Multilingual Contexts, M. Fernández-Villanueva & K. Jungbluth (eds), 34–54. Berlin, Boston: De Gruyter DOI logoGoogle Scholar
2007 Routes from gesture to language. In Verbal and Signed Languages: Comparing Structures, Constructs and Methodologies, E. Pizzuto, P. Pientrandrea & R. Simone (eds), 107–137. Berlin: Walter de Gruyter.Google Scholar
Wilcox, S., Rossini P. & Antinoro E. P.
2010 Grammaticalization in sign languages. In Sign Languages, D. Brentari (ed.), 332–355. Cambridge: CUP.DOI logoGoogle Scholar
Wilcox, S. & Xavier A. N.
2013 A framework for unifying spoken language, signed language, and gesture. Todas as Letras-Revista de Língua e Literatura 15(1): 88–110.Google Scholar
Cited by

Cited by 8 other publications

Gabarró-López, Sílvia & Laurence Meurant
2022. Contrasting signed and spoken languages. Languages in Contrast 22:2 ► pp. 169 ff. DOI logo
Ingram, Matthew Bruce & Ian Mitchell Wallace
2022. Making a mark: Transforming everyday work of city arborists into dance. Frontiers in Communication 7 DOI logo
Katila, Julia & Johanne S. Philipsen
2019. The intercorporeality of closing a curtain. Pragmatics & Cognition 26:2-3 ► pp. 167 ff. DOI logo
Müller, Cornelia
2018. Gesture and Sign: Cataclysmic Break or Dynamic Relations?. Frontiers in Psychology 9 DOI logo
Philipsen, Johanne S. & Sarah Bro Trasmundi
2019. Gesture reuse as distributed embodied cognition. Gesture 18:1 ► pp. 1 ff. DOI logo
Quinto-Pozos, David, Fey Parrill & Caitie Coons
2022. The interface between grammar and bodily enactment in ASL and English. Languages in Contrast 22:2 ► pp. 195 ff. DOI logo
Savchuk, Svetlana
2021. On the diachronic aspect of gestural communication. Languages and Modalities 1 ► pp. 95 ff. DOI logo
Vandenitte, Sébastien
2022. Making Referents Seen and Heard Across Signed and Spoken Languages: Documenting and Interpreting Cross-Modal Differences in the Use of Enactment. Frontiers in Psychology 13 DOI logo

This list is based on CrossRef data as of 22 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.