Cover not available
Article published In:
Gesture
Vol. 20:1 (2021) ► pp.103–134

Discourse markers in relation to non-verbal behavior

How do speech and body language correlate?

| University of Maribor
| University of Maribor
| University of Maribor
| University of Maribor

The research proposed in this paper focuses on pragmatic interlinks between discourse markers and non-verbal behavior. Although non-verbal behavior is recognized to add non-redundant information and social interaction is not merely recognized as the transmission of words and sentences, the evidence regarding grammatical/linguistic interlinks between verbal and non-verbal concepts are vague and limited to restricted domains. This is even more evident when non-verbal behavior acts in the foreground but contributes to the structure and organization of the discourse. This research focuses on investigating the multimodal nature of discourse markers by observing their linguistic and paralinguistic properties in informal discourse. We perform a quantitative analysis with case studies for representative cases. The results show that discourse markers and background non-verbal behavior tend to follow a similar functionality in interaction. Therefore, by examining them together, one gains more insight into their true intent despite the high multifunctionality of both non-verbal behavior and DMs.

Keywords: multimodal analysis, multimodal discourse, social interaction, metadiscourse, non-verbal behavior
Article outline
  • Introduction
  • Data and methodology
    • The EVA corpus
    • The annotation process
    • Annotating discourse markers in EVA corpus
    • Annotating non-verbal behavior through NCI
  • Results
    • Co-occurrence of DMs and NCIs
    • Co-occurrence of DMs and NCIs according to different classes of DMs
    • Case analysis for individual DM classes
      • Speech formation DMs
      • Contact DMs
      • Feedback DMs
      • Dialogue structure DMs
      • Connective DMs
  • Discussion
  • Conclusion
  • Notes
  • References
Published online: 22 November 2021
DOI logo
https://doi.org/10.1075/gest.20018.mla
References
Aijmer, K.
(2002) English discourse particles: Evidence from a corpus. Amsterdam & Philadelphia: John Benjamins. DOI logoGoogle Scholar
Allwood, J.
(2014) A framework for studying human multimodal communication. In Matej Rojc & Nick Campbell (Eds.), Coverbal synchrony in human-machine interaction (pp. 17–39). Boca Raton, FL: CRC Press.Google Scholar
Allwood, J., Ahlsén, E., Lund, J., & Sundqvist, J.
(2005) Multimodality in own communication management. In Proceedings from the Second Nordic Conference on Multimodal Communication (pp. 1–20). Göteborg: Göteborg University.Google Scholar
Arnold, L.
(2012) Dialogic embodied action: Using gesture to organize sequence and participation in instructional interaction. Research on Language & Social Interaction, 45 (3), 269–296. DOI logoGoogle Scholar
Baiat, G. E., Coler, M., Pullen, M., Tienkouw, S., & Hunyadi, L.
(2013, December). Multimodal analysis of “well” as a discourse marker in conversation: A pilot study. In 2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom) (pp. 283–288).
https://doi.org/
. DOI logoGoogle Scholar
Birdwhistell, R. L.
(2010) Kinesics and context. Philadelphia: University of Pennsylvania Press.Google Scholar
Bonsignori, V. & Camiciottoli, B. C.
(Eds.) (2017) Multimodality across communicative settings, discourse domains and genres. Cambridge: Cambridge Scholars Publishing.Google Scholar
Cassell, J.
(2001) Embodied conversational agents: representation and intelligence in user interfaces. AI magazine, 22 (4), 67.Google Scholar
Chui, K., Lee, C. Y., Yeh, K., & Chao, P. C.
(2018) Semantic processing of self-adaptors, emblems, and iconic gestures: An ERP study. Journal of Neurolinguistics, 471, 105–122. DOI logoGoogle Scholar
Cienki, A. & Müller, C.
(Eds.) (2008) Metaphor and gesture. Amsterdam & Philadelphia: John Benjamins. DOI logoGoogle Scholar
Coker, D. A. & Burgoon, J.
(1987) The nature of conversational involvement and nonverbal encoding patterns. Human Communication Research, 13 (4), 463–494. DOI logoGoogle Scholar
Cooperrider, K.
(2017) Foreground gesture, background gesture. Gesture, 16 (2), 176–202. DOI logoGoogle Scholar
Couper-Kuhlen, E.
(2018) Finding a place for body movement in grammar. Research on Language and Social Interaction, 51 (1), 22–25. DOI logoGoogle Scholar
Crible, L.
(2014) Identifying and describing discourse markers in spoken corpora: Annotation protocol v.8. Technical report. Louvain: Université catholique de Louvain.Google Scholar
Crible, L. & Pascual, E.
(2019) Combinations of discourse markers with repairs and repetitions in English, French and Spanish. Journal of Pragmatics, 1561, 54–67. DOI logoGoogle Scholar
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N.
(1972) The dependability of behavioural measurement: Theory of generalizability for scores and profiles. New York: Wiley.Google Scholar
Degand, L., Cornillie, B., & Pietrandrea, P.
(2013) (Eds.). Discourse markers and modal particles: Categorization and description. Amsterdam & Philadelphia: John Benjamins. DOI logoGoogle Scholar
Ekman, P. & Friesen, W.
(1971) Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17 (2), 124–129. DOI logoGoogle Scholar
Fischer, K.
(2013) Discourse markers. In Caroll Chapelle (Ed.), The encyclopedia of applied linguistics (Vol. 31, 1743–1748). Malden, MA & Oxford: Wiley & Blackwell.Google Scholar
Fitzpatrick, E.
(Ed.) (2007) Corpus linguistics beyond the word: Corpus research from phrase to discourse. Amsterdam & New York: Rodopi. DOI logoGoogle Scholar
Guaitella, I., Santi, S., Lagrue, B., & Cavé, C.
(2009) Are eyebrow movements linked to voice variations and turn-taking in dialogue? An experimental investigation. Language and Speech, 52 (2/3), 207–222. DOI logoGoogle Scholar
Harrison, S., Adolphs, S., Dowens, M. G., Du, P., & Littlemore, J.
(2018) All hands on deck. Negotiation over gesture forms in collaborative discourse. Lingua, 2071, 1–22. DOI logoGoogle Scholar
Hyland, K.
(2005) Metadiscourse: Exploring interaction in writing. London & New York: Continuum.Google Scholar
Keevallik, L.
(2018) What does embodied interaction tell us about grammar? Research on Language and Social Interaction, 51 (1), 1–21. DOI logoGoogle Scholar
Kendon, A.
(2014) Semiotic diversity in utterance production and the concept of ‘language’. Philosophical Transactions of the Royal Society, B 369 (1651), 20130293. DOI logoGoogle Scholar
Knight, D.
(2011) Multimodality and active listenership: A corpus approach. London: Continuum.Google Scholar
Kok, K. I. & Cienki, A.
(2016) Cognitive grammar and gesture: Points of convergence, advances and challenges. Cognitive Linguistics, 27 (1), 67–100. DOI logoGoogle Scholar
Kushch, O., Igualada, A., & Prieto, P.
(2018) Prominence in speech and gesture favour second language novel word learning. Language, Cognition and Neuroscience, 33 (8), 992–1004. DOI logoGoogle Scholar
Lenker, U.
(2018) ‘There’s an issue there…’: Signalling functions of discourse-deictic there in the history of English. Language Sciences, 681, 94–105. DOI logoGoogle Scholar
Lester, J. C., Voerman, J. L., Towns, S. G., & Callaway, C. B.
(1999) Deictic believability: Coordinated gesture, locomotion, and speech in lifelike pedagogical agents. Applied Artificial Intelligence, 13 (4/5), 383–414. DOI logoGoogle Scholar
Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borras-Comes, J., & Prieto, P.
(2018) Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology, 1721, 168–188. DOI logoGoogle Scholar
Lopes, A. C. M.
(2016) Discourse markers. In L. Wetzels, S. Menuzzi, & J. Costa (Eds.), The handbook of Portuguese linguistics. New York: Wiley. DOI logoGoogle Scholar
Lopez-Ozieblo, R. & McNeill, D.
(2017) Exchange on gesture-speech unity: What it is, where it came from. In R. Breckinridge Church, M. W. Alibali, & S. D. Kelly (Eds.), Why gesture? How the hands function in speaking, thinking and communicating (pp. 103–125). Amsterdam & Philadelphia: John Benjamins. DOI logoGoogle Scholar
Maricchiolo, F., Gnisci, A., & Bonaiuto, M.
(2012) Coding hand gestures: A reliable taxonomy and a multi-media support. In Anna Esposito et al. (Eds.), Cognitive behavioural systems (pp. 405–416). Berlin & Heidelberg: Springer. DOI logoGoogle Scholar
Maschler, Y. & Schiffrin, D.
(2015) Discourse markers: Language, meaning, and context. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (pp. 189–221). Hoboken, NJ: Wiley.Google Scholar
McCarthy, M.
(2003) Talking back: “Small” interactional response tokens in everyday conversation. Research on Language and Social Interaction, 36 (1), 33–63. DOI logoGoogle Scholar
McHugh, M. L.
(2012) Interrater reliability: the kappa statistic. Biochemia Medica, 22 (3), 276–282. DOI logoGoogle Scholar
McNeill, D.
(2008) Gesture and thought. Chicago: University of Chicago Press.Google Scholar
McNeill, D., Cassell, J., & Levy, E. T.
(1993) Abstract deixis. Semiotica, 95 (1/2), 5–20. DOI logoGoogle Scholar
McNeill, D., Levy, E., & Duncan, S. D.
(2015) Gesture in discourse. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (pp. 262–289). Hoboken, NJ: Wiley.Google Scholar
Melinger, A. & Levelt, W. J.
(2004) Gesture and the communicative intention of the speaker. Gesture, 4 (2), 119–141. DOI logoGoogle Scholar
Mlakar, I., Verdonik, D., Majhenič, S., & Rojc, M.
(2019) Towards pragmatic understanding of conversational intent: A multimodal annotation approach to multiparty informal interaction – The EVA Corpus. In: International Conference on Statistical Language and Speech Processing. Proceedings (pp. 19–30). Cham: Springer. DOI logoGoogle Scholar
Nevile, M.
(2015) The embodied turn in research on language and social interaction. Research on Language and Social Interaction, 48 (2), 121–151. DOI logoGoogle Scholar
Neville, D. O.
(2015) The story in the mind: The effect of 3D gameplay on the structuring of written L2 narratives. ReCALL, 27 (1), 21–37. DOI logoGoogle Scholar
Nikolaeva, Y.
(2014) Speech-accompanying gestures in Russian: functions and verbal context. In Proceedings of the Third Workshop on Vision and Language (pp. 82–86). Dublin: Dublin City University & Association for Computational Linguistics. DOI logoGoogle Scholar
Paggio, P. & Navarretta, C.
(2017) The Danish NOMCO corpus: multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51 (2), 463–494. DOI logoGoogle Scholar
Peirce, C. S.
(1965) Collected papers, Vol. 51. Cambridge: Harvard University Press.Google Scholar
Redeker, G.
(1990) Ideational and pragmatic markers of discourse structure. Journal of Pragmatics, 14 (3), 367–381. DOI logoGoogle Scholar
Schiffrin, D.
(1987) Discourse markers. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Schlenker, P.
(2018) Iconic pragmatics. Natural Language & Linguistic Theory, 36 (3), 877–936. DOI logoGoogle Scholar
Schourup, L.
(1999) Discourse markers. Lingua, 107 (3), 227–265. DOI logoGoogle Scholar
Trujillo, J. P., Simanova, I., Bekkering, H., & Özyürek, A.
(2018) Communicative intent modulates production and comprehension of actions and gestures: A Kinect study. Cognition, 1801, 38–51. DOI logoGoogle Scholar
Verdonik, D.
(2007) Upravljanje pogovora kot metadiskurzna funkcija. Metabesedilnost v uporabnem jezikoslovju. Jezik in slovstvo, 3 (4), 53–65.Google Scholar
Verdonik, D., Rojc, M., & Stabej, M.
(2007) Annotating discourse markers in spontaneous speech corpora on an example for the Slovenian language. Language Resources and Evaluation, 41 (2), 147–180. DOI logoGoogle Scholar
Verdonik, D., Žgank, A., & Pisanski Peterlin, A.
(2008) The impact of context on discourse marker use in two conversational genres. Discourse Studies, 10 (6), 759–775. DOI logoGoogle Scholar
Vigliocco, G., Perniss, P., & Vinson, D.
(2014) Language as a multimodal phenomenon: implications for language learning, processing and evolution. Philosophical Transactions of the Royal Society of London, B 369 (1651), 20130292.Google Scholar