Article published In:
GestureVol. 20:1 (2021) ► pp.103–134
Discourse markers in relation to non-verbal behavior
How do speech and body language correlate?
The research proposed in this paper focuses on pragmatic interlinks between discourse markers and non-verbal
behavior. Although non-verbal behavior is recognized to add non-redundant information and social interaction is not merely
recognized as the transmission of words and sentences, the evidence regarding grammatical/linguistic interlinks between verbal and
non-verbal concepts are vague and limited to restricted domains. This is even more evident when non-verbal behavior acts in the
foreground but contributes to the structure and organization of the discourse. This research focuses on investigating the
multimodal nature of discourse markers by observing their linguistic and paralinguistic properties in informal discourse. We
perform a quantitative analysis with case studies for representative cases. The results show that discourse markers and background
non-verbal behavior tend to follow a similar functionality in interaction. Therefore, by examining them together, one gains more
insight into their true intent despite the high multifunctionality of both non-verbal behavior and DMs.
Article outline
- Introduction
- Data and methodology
- The EVA corpus
- The annotation process
- Annotating discourse markers in EVA corpus
- Annotating non-verbal behavior through NCI
- Results
- Co-occurrence of DMs and NCIs
- Co-occurrence of DMs and NCIs according to different classes of DMs
- Case analysis for individual DM classes
- Speech formation DMs
- Contact DMs
- Feedback DMs
- Dialogue structure DMs
- Connective DMs
- Discussion
- Conclusion
- Notes
-
References
References (57)
Allwood, J.
(
2014)
A
framework for studying human multimodal communication. In
Matej Rojc &
Nick Campbell (Eds.),
Coverbal
synchrony in human-machine
interaction (pp. 17–39). Boca Raton, FL: CRC Press.
Allwood, J., Ahlsén, E., Lund, J., & Sundqvist, J.
(
2005)
Multimodality
in own communication management. In
Proceedings from the Second
Nordic Conference on Multimodal
Communication (pp. 1–20). Göteborg: Göteborg University.
Arnold, L.
(
2012)
Dialogic
embodied action: Using gesture to organize sequence and participation in instructional
interaction.
Research on Language & Social
Interaction, 45 (3), 269–296.
Baiat, G. E., Coler, M., Pullen, M., Tienkouw, S., & Hunyadi, L.
(
2013,
December).
Multimodal analysis of “well” as a discourse marker in conversation: A pilot study. In
2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom) (pp. 283–288).
.
Birdwhistell, R. L.
(
2010)
Kinesics and context. Philadelphia: University of Pennsylvania Press.
Bonsignori, V. & Camiciottoli, B. C.
(Eds.) (
2017)
Multimodality
across communicative settings, discourse domains and
genres. Cambridge: Cambridge Scholars Publishing.
Cassell, J.
(
2001)
Embodied
conversational agents: representation and intelligence in user interfaces.
AI
magazine, 22 (4), 67.
Chui, K., Lee, C. Y., Yeh, K., & Chao, P. C.
(
2018)
Semantic
processing of self-adaptors, emblems, and iconic gestures: An ERP study.
Journal of
Neurolinguistics, 471, 105–122.
Coker, D. A. & Burgoon, J.
(
1987)
The
nature of conversational involvement and nonverbal encoding patterns.
Human Communication
Research, 13 (4), 463–494.
Couper-Kuhlen, E.
(
2018)
Finding
a place for body movement in grammar.
Research on Language and Social
Interaction, 51 (1), 22–25.
Crible, L.
(
2014)
Identifying
and describing discourse markers in spoken corpora: Annotation protocol v.8.
Technical
report. Louvain: Université catholique de Louvain.
Crible, L. & Pascual, E.
(
2019)
Combinations
of discourse markers with repairs and repetitions in English, French and Spanish.
Journal of
Pragmatics, 1561, 54–67.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N.
(
1972)
The
dependability of behavioural measurement: Theory of generalizability for scores and
profiles. New York: Wiley.
Degand, L., Cornillie, B., & Pietrandrea, P.
Ekman, P. & Friesen, W.
(
1971)
Constants
across cultures in the face and emotion.
Journal of Personality and Social
Psychology, 17 (2), 124–129.
Fischer, K.
(
2013)
Discourse
markers. In
Caroll Chapelle (Ed.),
The
encyclopedia of applied
linguistics (Vol. 31, 1743–1748). Malden, MA & Oxford: Wiley & Blackwell.
Fitzpatrick, E.
(Ed.) (
2007)
Corpus
linguistics beyond the word: Corpus research from phrase to discourse. Amsterdam & New York: Rodopi.
Guaitella, I., Santi, S., Lagrue, B., & Cavé, C.
(
2009)
Are
eyebrow movements linked to voice variations and turn-taking in dialogue? An experimental
investigation.
Language and
Speech, 52 (2/3), 207–222.
Harrison, S., Adolphs, S., Dowens, M. G., Du, P., & Littlemore, J.
(
2018)
All
hands on deck. Negotiation over gesture forms in collaborative
discourse.
Lingua, 2071, 1–22.
Hyland, K.
(
2005)
Metadiscourse:
Exploring interaction in writing. London & New York: Continuum.
Keevallik, L.
(
2018)
What
does embodied interaction tell us about grammar? Research on Language and Social
Interaction, 51 (1), 1–21.
Kendon, A.
(
2014)
Semiotic
diversity in utterance production and the concept of ‘language’.
Philosophical Transactions of
the Royal Society, B
369 (1651), 20130293.
Knight, D.
(
2011)
Multimodality
and active listenership: A corpus
approach. London: Continuum.
Kok, K. I. & Cienki, A.
(
2016)
Cognitive
grammar and gesture: Points of convergence, advances and challenges.
Cognitive
Linguistics, 27 (1), 67–100.
Kushch, O., Igualada, A., & Prieto, P.
(
2018)
Prominence
in speech and gesture favour second language novel word learning.
Language, Cognition and
Neuroscience, 33 (8), 992–1004.
Lenker, U.
(
2018)
‘There’s
an issue there…’: Signalling functions of discourse-deictic there in the history of
English.
Language
Sciences, 681, 94–105.
Lester, J. C., Voerman, J. L., Towns, S. G., & Callaway, C. B.
(
1999)
Deictic
believability: Coordinated gesture, locomotion, and speech in lifelike pedagogical
agents.
Applied Artificial
Intelligence, 13 (4/5), 383–414.
Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borras-Comes, J., & Prieto, P.
(
2018)
Beat
gestures help preschoolers recall and comprehend discourse information.
Journal of Experimental
Child Psychology, 1721, 168–188.
Lopes, A. C. M.
(
2016)
Discourse
markers. In
L. Wetzels,
S. Menuzzi, &
J. Costa (Eds.),
The
handbook of Portuguese linguistics. New York: Wiley.
Lopez-Ozieblo, R. & McNeill, D.
Maricchiolo, F., Gnisci, A., & Bonaiuto, M.
(
2012)
Coding
hand gestures: A reliable taxonomy and a multi-media
support. In
Anna Esposito et al. (Eds.),
Cognitive
behavioural
systems (pp. 405–416). Berlin & Heidelberg: Springer.
Maschler, Y. & Schiffrin, D.
(
2015)
Discourse
markers: Language, meaning, and context. In
D. Tannen,
H. E. Hamilton, &
D. Schiffrin (Eds.),
The
handbook of discourse
analysis (pp. 189–221). Hoboken, NJ: Wiley.
McCarthy, M.
(
2003)
Talking
back: “Small” interactional response tokens in everyday conversation.
Research on Language and
Social
Interaction, 36 (1), 33–63.
McHugh, M. L.
(
2012)
Interrater
reliability: the kappa statistic.
Biochemia
Medica, 22 (3), 276–282.
McNeill, D.
(
2008)
Gesture
and thought. Chicago: University of Chicago Press.
McNeill, D., Cassell, J., & Levy, E. T.
(
1993)
Abstract
deixis.
Semiotica, 95 (1/2), 5–20.
McNeill, D., Levy, E., & Duncan, S. D.
(
2015)
Gesture
in discourse. In
D. Tannen,
H. E. Hamilton, &
D. Schiffrin (Eds.),
The
handbook of discourse
analysis (pp. 262–289). Hoboken, NJ: Wiley.
Melinger, A. & Levelt, W. J.
Mlakar, I., Verdonik, D., Majhenič, S., & Rojc, M.
(
2019)
Towards
pragmatic understanding of conversational intent: A multimodal annotation approach to multiparty informal interaction – The
EVA Corpus. In:
International Conference on Statistical Language and
Speech Processing.
Proceedings (pp. 19–30). Cham: Springer.
Nevile, M.
(
2015)
The
embodied turn in research on language and social interaction.
Research on Language and Social
Interaction, 48 (2), 121–151.
Neville, D. O.
(
2015)
The story in the mind: The effect of 3D gameplay on the structuring of written L2 narratives.
ReCALL, 27 (1), 21–37.
Nikolaeva, Y.
(
2014)
Speech-accompanying
gestures in Russian: functions and verbal context. In
Proceedings of
the Third Workshop on Vision and
Language (pp. 82–86). Dublin: Dublin City University & Association for Computational Linguistics.
Paggio, P. & Navarretta, C.
(
2017)
The
Danish NOMCO corpus: multimodal interaction in first acquaintance conversations.
Language
Resources and
Evaluation, 51 (2), 463–494.
Peirce, C. S.
(
1965)
Collected
papers, Vol. 51. Cambridge: Harvard University Press.
Redeker, G.
(
1990)
Ideational and pragmatic markers of discourse structure.
Journal of Pragmatics, 14 (3), 367–381.
Schiffrin, D.
(
1987)
Discourse
markers. Cambridge: Cambridge University Press.
Schlenker, P.
(
2018)
Iconic
pragmatics.
Natural Language & Linguistic
Theory, 36 (3), 877–936.
Schourup, L.
(
1999)
Discourse
markers.
Lingua, 107 (3), 227–265.
Trujillo, J. P., Simanova, I., Bekkering, H., & Özyürek, A.
(
2018)
Communicative
intent modulates production and comprehension of actions and gestures: A Kinect
study.
Cognition, 1801, 38–51.
Verdonik, D.
(
2007)
Upravljanje pogovora kot metadiskurzna funkcija. Metabesedilnost v uporabnem jezikoslovju.
Jezik in slovstvo, 3 (4), 53–65.
Verdonik, D., Rojc, M., & Stabej, M.
(
2007)
Annotating
discourse markers in spontaneous speech corpora on an example for the Slovenian
language.
Language Resources and
Evaluation, 41 (2), 147–180.
Verdonik, D., Žgank, A., & Pisanski Peterlin, A.
(
2008)
The
impact of context on discourse marker use in two conversational genres.
Discourse
Studies, 10 (6), 759–775.
Vigliocco, G., Perniss, P., & Vinson, D.
(
2014)
Language
as a multimodal phenomenon: implications for language learning, processing and
evolution.
Philosophical Transactions of the Royal Society of London, B
369 (1651), 20130292.
Cited by (2)
Cited by 2 other publications
Ben-Moshe, Yotam M. & Yael Maschler
2024.
Hebrew clicks: From the periphery of language to the heart of grammar.
Journal of Pragmatics 229
► pp. 19 ff.
Mlakar, Izidor, Darinka Verdonik, Simona Majhenič & Matej Rojc
2023.
Understanding conversational interaction in multiparty conversations: the EVA Corpus.
Language Resources and Evaluation 57:2
► pp. 641 ff.
This list is based on CrossRef data as of 4 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.