Article published in:
Gesture
Vol. 20:1 (2021) ► pp. 103–134

Discourse markers in relation to non-verbal behavior

How do speech and body language correlate?

| University of Maribor
| University of Maribor
| University of Maribor
| University of Maribor
The research proposed in this paper focuses on pragmatic interlinks between discourse markers and non-verbal behavior. Although non-verbal behavior is recognized to add non-redundant information and social interaction is not merely recognized as the transmission of words and sentences, the evidence regarding grammatical/linguistic interlinks between verbal and non-verbal concepts are vague and limited to restricted domains. This is even more evident when non-verbal behavior acts in the foreground but contributes to the structure and organization of the discourse. This research focuses on investigating the multimodal nature of discourse markers by observing their linguistic and paralinguistic properties in informal discourse. We perform a quantitative analysis with case studies for representative cases. The results show that discourse markers and background non-verbal behavior tend to follow a similar functionality in interaction. Therefore, by examining them together, one gains more insight into their true intent despite the high multifunctionality of both non-verbal behavior and DMs.
Keywords: multimodal analysis, multimodal discourse, social interaction, metadiscourse, non-verbal behavior
Article outline
  • Introduction
  • Data and methodology
    • The EVA corpus
    • The annotation process
    • Annotating discourse markers in EVA corpus
    • Annotating non-verbal behavior through NCI
  • Results
    • Co-occurrence of DMs and NCIs
    • Co-occurrence of DMs and NCIs according to different classes of DMs
    • Case analysis for individual DM classes
      • Speech formation DMs
      • Contact DMs
      • Feedback DMs
      • Dialogue structure DMs
      • Connective DMs
  • Discussion
  • Conclusion
  • Notes
  • References
Published online: 22 November 2021
https://doi.org/10.1075/gest.20018.mla
References
Aijmer, K.
(2002) English discourse particles: Evidence from a corpus. Amsterdam & Philadelphia: John Benjamins. CrossrefGoogle Scholar
Allwood, J.
(2014) A framework for studying human multimodal communication. In Matej Rojc & Nick Campbell (Eds.), Coverbal synchrony in human-machine interaction (pp. 17–39). Boca Raton, FL: CRC Press.Google Scholar
Allwood, J., Ahlsén, E., Lund, J., & Sundqvist, J.
(2005) Multimodality in own communication management. In Proceedings from the Second Nordic Conference on Multimodal Communication (pp. 1–20). Göteborg: Göteborg University.Google Scholar
Arnold, L.
(2012) Dialogic embodied action: Using gesture to organize sequence and participation in instructional interaction. Research on Language & Social Interaction, 45 (3), 269–296. CrossrefGoogle Scholar
Baiat, G. E., Coler, M., Pullen, M., Tienkouw, S., & Hunyadi, L.
(2013, December). Multimodal analysis of “well” as a discourse marker in conversation: A pilot study. In 2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom) (pp. 283–288). Crossref. CrossrefGoogle Scholar
Birdwhistell, R. L.
(2010) Kinesics and context. Philadelphia: University of Pennsylvania Press.Google Scholar
Bonsignori, V. & Camiciottoli, B. C.
(Eds.) (2017) Multimodality across communicative settings, discourse domains and genres. Cambridge: Cambridge Scholars Publishing.Google Scholar
Cassell, J.
(2001) Embodied conversational agents: representation and intelligence in user interfaces. AI magazine, 22 (4), 67.Google Scholar
Chui, K., Lee, C. Y., Yeh, K., & Chao, P. C.
(2018) Semantic processing of self-adaptors, emblems, and iconic gestures: An ERP study. Journal of Neurolinguistics, 471, 105–122. CrossrefGoogle Scholar
Cienki, A. & Müller, C.
(Eds.) (2008) Metaphor and gesture. Amsterdam & Philadelphia: John Benjamins. CrossrefGoogle Scholar
Coker, D. A. & Burgoon, J.
(1987) The nature of conversational involvement and nonverbal encoding patterns. Human Communication Research, 13 (4), 463–494. CrossrefGoogle Scholar
Cooperrider, K.
(2017) Foreground gesture, background gesture. Gesture, 16 (2), 176–202. CrossrefGoogle Scholar
Couper-Kuhlen, E.
(2018) Finding a place for body movement in grammar. Research on Language and Social Interaction, 51 (1), 22–25. CrossrefGoogle Scholar
Crible, L.
(2014) Identifying and describing discourse markers in spoken corpora: Annotation protocol v.8. Technical report. Louvain: Université catholique de Louvain.Google Scholar
Crible, L. & Pascual, E.
(2019) Combinations of discourse markers with repairs and repetitions in English, French and Spanish. Journal of Pragmatics, 1561, 54–67. CrossrefGoogle Scholar
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N.
(1972) The dependability of behavioural measurement: Theory of generalizability for scores and profiles. New York: Wiley.Google Scholar
Degand, L., Cornillie, B., & Pietrandrea, P.
(2013) (Eds.). Discourse markers and modal particles: Categorization and description. Amsterdam & Philadelphia: John Benjamins. CrossrefGoogle Scholar
Dobrovoljc, K.
(2017) Multi-word discourse markers and their corpus-driven identification: The case of MWDM extraction from the reference corpus of spoken Slovene. International Journal of Corpus Linguistics, 22 (4), 551–582. CrossrefGoogle Scholar
Ekman, P. & Friesen, W.
(1971) Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17 (2), 124–129. CrossrefGoogle Scholar
Fischer, K.
(2013) Discourse markers. In Caroll Chapelle (Ed.), The encyclopedia of applied linguistics (Vol. 31, 1743–1748). Malden, MA & Oxford: Wiley & Blackwell.Google Scholar
Fitzpatrick, E.
(Ed.) (2007) Corpus linguistics beyond the word: Corpus research from phrase to discourse. Amsterdam & New York: Rodopi. CrossrefGoogle Scholar
Guaitella, I., Santi, S., Lagrue, B., & Cavé, C.
(2009) Are eyebrow movements linked to voice variations and turn-taking in dialogue? An experimental investigation. Language and Speech, 52 (2/3), 207–222. CrossrefGoogle Scholar
Harrison, S., Adolphs, S., Dowens, M. G., Du, P., & Littlemore, J.
(2018) All hands on deck. Negotiation over gesture forms in collaborative discourse. Lingua, 2071, 1–22. CrossrefGoogle Scholar
Hyland, K.
(2005) Metadiscourse: Exploring interaction in writing. London & New York: Continuum.Google Scholar
Keevallik, L.
(2018) What does embodied interaction tell us about grammar? Research on Language and Social Interaction, 51 (1), 1–21. CrossrefGoogle Scholar
Kendon, A.
(2014) Semiotic diversity in utterance production and the concept of ‘language’. Philosophical Transactions of the Royal Society, B 369 (1651), 20130293. CrossrefGoogle Scholar
Knight, D.
(2011) Multimodality and active listenership: A corpus approach. London: Continuum.Google Scholar
Kok, K. I. & Cienki, A.
(2016) Cognitive grammar and gesture: Points of convergence, advances and challenges. Cognitive Linguistics, 27 (1), 67–100. CrossrefGoogle Scholar
Kushch, O., Igualada, A., & Prieto, P.
(2018) Prominence in speech and gesture favour second language novel word learning. Language, Cognition and Neuroscience, 33 (8), 992–1004. CrossrefGoogle Scholar
Lenker, U.
(2018) ‘There’s an issue there…’: Signalling functions of discourse-deictic there in the history of English. Language Sciences, 681, 94–105. CrossrefGoogle Scholar
Lester, J. C., Voerman, J. L., Towns, S. G., & Callaway, C. B.
(1999) Deictic believability: Coordinated gesture, locomotion, and speech in lifelike pedagogical agents. Applied Artificial Intelligence, 13 (4/5), 383–414. CrossrefGoogle Scholar
Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borras-Comes, J., & Prieto, P.
(2018) Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology, 1721, 168–188. CrossrefGoogle Scholar
Lopes, A. C. M.
(2016) Discourse markers. In L. Wetzels, S. Menuzzi, & J. Costa (Eds.), The handbook of Portuguese linguistics. New York: Wiley. CrossrefGoogle Scholar
Lopez-Ozieblo, R. & McNeill, D.
(2017) Exchange on gesture-speech unity: What it is, where it came from. In R. Breckinridge Church, M. W. Alibali, & S. D. Kelly (Eds.), Why gesture? How the hands function in speaking, thinking and communicating (pp. 103–125). Amsterdam & Philadelphia: John Benjamins. CrossrefGoogle Scholar
Maricchiolo, F., Gnisci, A., & Bonaiuto, M.
(2012) Coding hand gestures: A reliable taxonomy and a multi-media support. In Anna Esposito et al. (Eds.), Cognitive behavioural systems (pp. 405–416). Berlin & Heidelberg: Springer. CrossrefGoogle Scholar
Maschler, Y. & Schiffrin, D.
(2015) Discourse markers: Language, meaning, and context. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (pp. 189–221). Hoboken, NJ: Wiley.Google Scholar
McCarthy, M.
(2003) Talking back: “Small” interactional response tokens in everyday conversation. Research on Language and Social Interaction, 36 (1), 33–63. CrossrefGoogle Scholar
McHugh, M. L.
(2012) Interrater reliability: the kappa statistic. Biochemia Medica, 22 (3), 276–282. CrossrefGoogle Scholar
McNeill, D.
(2008) Gesture and thought. Chicago: University of Chicago Press.Google Scholar
McNeill, D., Cassell, J., & Levy, E. T.
(1993) Abstract deixis. Semiotica, 95 (1/2), 5–20. CrossrefGoogle Scholar
McNeill, D., Levy, E., & Duncan, S. D.
(2015) Gesture in discourse. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (pp. 262–289). Hoboken, NJ: Wiley.Google Scholar
Melinger, A. & Levelt, W. J.
(2004) Gesture and the communicative intention of the speaker. Gesture, 4 (2), 119–141. CrossrefGoogle Scholar
Mlakar, I., Verdonik, D., Majhenič, S., & Rojc, M.
(2019) Towards pragmatic understanding of conversational intent: A multimodal annotation approach to multiparty informal interaction – The EVA Corpus. In: International Conference on Statistical Language and Speech Processing. Proceedings (pp. 19–30). Cham: Springer. CrossrefGoogle Scholar
Nevile, M.
(2015) The embodied turn in research on language and social interaction. Research on Language and Social Interaction, 48 (2), 121–151. CrossrefGoogle Scholar
Neville, D. O.
(2015) The story in the mind: The effect of 3D gameplay on the structuring of written L2 narratives. ReCALL, 27 (1), 21–37. CrossrefGoogle Scholar
Nikolaeva, Y.
(2014) Speech-accompanying gestures in Russian: functions and verbal context. In Proceedings of the Third Workshop on Vision and Language (pp. 82–86). Dublin: Dublin City University & Association for Computational Linguistics. CrossrefGoogle Scholar
Paggio, P. & Navarretta, C.
(2017) The Danish NOMCO corpus: multimodal interaction in first acquaintance conversations. Language Resources and Evaluation, 51 (2), 463–494. CrossrefGoogle Scholar
Peirce, C. S.
(1965) Collected papers, Vol. 51. Cambridge: Harvard University Press.Google Scholar
Redeker, G.
(1990) Ideational and pragmatic markers of discourse structure. Journal of Pragmatics, 14 (3), 367–381. CrossrefGoogle Scholar
Schiffrin, D.
(1987) Discourse markers. Cambridge: Cambridge University Press. CrossrefGoogle Scholar
Schlenker, P.
(2018) Iconic pragmatics. Natural Language & Linguistic Theory, 36 (3), 877–936. CrossrefGoogle Scholar
Schourup, L.
(1999) Discourse markers. Lingua, 107 (3), 227–265. CrossrefGoogle Scholar
Trujillo, J. P., Simanova, I., Bekkering, H., & Özyürek, A.
(2018) Communicative intent modulates production and comprehension of actions and gestures: A Kinect study. Cognition, 1801, 38–51. CrossrefGoogle Scholar
Verdonik, D.
(2007) Upravljanje pogovora kot metadiskurzna funkcija. Metabesedilnost v uporabnem jezikoslovju. Jezik in slovstvo, 3 (4), 53–65.Google Scholar
Verdonik, D., Rojc, M., & Stabej, M.
(2007) Annotating discourse markers in spontaneous speech corpora on an example for the Slovenian language. Language Resources and Evaluation, 41 (2), 147–180. CrossrefGoogle Scholar
Verdonik, D., Žgank, A., & Pisanski Peterlin, A.
(2008) The impact of context on discourse marker use in two conversational genres. Discourse Studies, 10 (6), 759–775. CrossrefGoogle Scholar
Vigliocco, G., Perniss, P., & Vinson, D.
(2014) Language as a multimodal phenomenon: implications for language learning, processing and evolution. Philosophical Transactions of the Royal Society of London, B 369 (1651), 20130292.Google Scholar