Including facial gestures in gesture-speech ensembles
Janet Bavelas | Department of Psychology, University of Victoria
Jennifer Gerwing | Department of Psychology, University of Victoria
Sara Healing | Department of Psychology, University of Victoria
Conversational facial gestures fit Kendon’s (2004) specifications of the functions of hand gestures. We illustrate how facial gestures in dialogue, like hand gestures, convey referential content as well as serving pragmatic, interpersonal and interactive functions. Hand and facial gestures often occur together, creating an integrated visual image in gesture–speech ensembles. A semantic features analysis demonstrates how speakers adjust their use of these visible versus audible expressive resources according to context. Speakers who were interacting face-to-face (compared to speakers who could not see their addressee) were significantly more likely to rely on their hand and facial gestures than on their words when describing key semantic features, and their gestures were more likely to convey information that was not in their words.
1997“Faces in dialogue.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 334–346. Cambridge: Cambridge University Press.
Bavelas, J., Chovil, N., Lawrie, D.A., and Wade, A
2014“Hand and facial gestures in conversational interaction.” In Handbook of Language and Social Psychology, T. Holtgraves (ed.), 111–130. Oxford: Oxford University Press.
Bavelas, J., Gerwing, J., and Healing, S
in press“The effect of dialogue on demonstrations: direct quotations, facial portrayals, hand gestures, and figurative references.” (accepted for publication in Discourse Processes)
Bavelas, J., Gerwing, J., Sutton, C., and Prevost, D
2008“Gesturing on the telephone: Independent effects of dialogue and visibility.”Journal of Memory and Language 58: 495–520.
1999“Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation.”Semiotica 123 (1–2): 1–30.
Beattie, G., and Shovelton, H
2002“An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.”British Journal of Psychology 93: 179–192.
Birdwhistell, R.L
.1970“Masculinity and femininity as display.”Kinesics and Context: Essays on Body Motion. Philadelphia, PA: University of Pennsylvania.
Brugman, H., and Russel, A
2004“Annotating multimedia/multimodal resources with ELAN.” In
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004)
, M.T. Lino, M.F. Xavier, F. Ferreira, R. Costa, and R. Silva (eds), 2065–2068. Paris: European Language Resources Association.
Brunner, L.J
1979“Smiles can be back channels.”Journal of Personality and Social Psychology 37: 728–734.
Chovil, N
1989Communicative Functions of Facial Displays in Conversation. Ph.D. dissertation, University of Victoria, BC.
Chovil, N
1991/1992 “Discourse-oriented facial displays in conversation.”Research on Language and Social Interaction 25: 163–192.
Chovil, N
1991Social determinants of facial displays. Journal of Nonverbal Behavior 15: 141–153.
Chovil, N
1997“Facing others. A social communicative perspective on facial displays.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 321–333. Cambridge: Cambridge University Press.
Coates, L.J
1991A Collaborative Theory of Inversion: Irony in Dialogue. M.A. thesis, University of Victoria, BC.
de Ruiter, J.P., Bangerter, A., and Dings, P
2012“The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis.”Topics in Cognitive Science 4 (2): 232–248.
Ekman, P
1979“About brows: Emotional and conversational signals.” In Human Ethology, M. von Cranach, K. Foppa, W. Lepenies and D. Ploog (eds), 169–249. Cambridge: Cambridge University Press.
Ekman, P
1985Telling Lies. New York: Norton.
Ekman, P
1997“Should we call it expression or communication?”European Journal of Social Sciences 10: 333–359.
Ekman, P., and Friesen, W.V
1969“The repertoire of nonverbal behavior: Categories, origins, usage, and coding.”Semiotica I: 49–98.
1987“Facial expressions of emotion: Review of literature, 1970–1983.” In Nonverbal Communication and Behavior (2nd ed.), A.W. Siegman and S. Feldstein, eds. (143–224). Hillsdale, NJ: Erlbaum.
2002“A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk.”Semiotica 142: 31–69.
Holler, J., and Beattie, G
2003“How iconic gestures and speech interact in the representation of meaning: Are both aspects really integral to the process?”Semiotica 146: 81–116.
Holler, J., and Beattie, G
2004“The interaction of iconic gesture and speech.” In
Lecture Notes in Computer Science, 5th International Gesture Workshop, Genova, Italy, 2003: Selected Revised Papers
, A. Cammurri and G. Volpe (eds), 63–69. Heidelberg: Springer Verlag.
Holler, J., and Stevens, R
2007“An experimental investigation into the effect of common ground on how speakers use gesture and speech to represent size information in referential communication.”Journal of Language and Social Psychology 26: 4–27.
Holler, J., and Wilkin, K
2009“Communicating common ground: How mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task.”Language and Cognitive Processes 24: 267–289.
Holler, J., Tutton, M., and Wilkin, K
2011“Co-speech gestures in the process of meaning coordination.” In
Proceedings of the 2nd GESPIN – Gesture in Speech and Interaction Conference
, Bielefeld, 5–7, September 2011.
Kendon, A
2004Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
Kraut, R.E., and Johnston, R.E
1979“Social and emotional messages of smiling: An ethological approach.”Journal of Personality and Social Psychology 61: 743–745.
Pine, K.J., Burney, D.J., and Fletcher, B
2010“The semantic specificity hypothesis: When gestures do not depend upon the presence of a listener.”Journal of Nonverbal Behavior 34: 169–178.
Rimé, B
1982“The elimination of visible behavior from social interactions: Effects on verbal, nonverbal and interpersonal variables.”European Journal of Social Psychology 12: 113–129.
Sherzer, J
1973Verbal and Nonverbal Deixis: The Pointed Lip Gesture among the San Blas Cuna. University of Texas at Austin: Institute of Latin American Studies.
Tomasello, M., Hare, B., Lehmann, H., and Call, J
2007“Reliance on head versus eyes in the gaze following of great apes and human infants: the cooperative eye hypothesis.”Journal of Human Evolution 52 (3): 314–320.
Wittenburg, P., Brugman, H., Russel, A., Klassman, A., and Sloetjes, H
2006“ELAN: A professional framework for multimodality research.” In
Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation
, Genoa.
2018. Developmental and cognitive aspects of children’s disbelief comprehension through intonation and facial gesture. First Language 38:6 ► pp. 596 ff.
Clift, Rebecca
2021. Embodiment in Dissent: The Eye Roll as an Interactional Practice. Research on Language and Social Interaction 54:3 ► pp. 261 ff.
Edman, Kristina
2024. Doing mutual understanding in child and family therapy sessions: How three interlocutors calibrate new information. Discourse Studies
Imamura, Mai, Ayane Tashiro, Shiro Kumano & Kazuhiro Otsuka
2023. INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ► pp. 42 ff.
Sharkov, F. I., V. V. Silkin & O. F. Kireeva
2022. Non-verbal signs of personality: Communicative meanings of facial expressions. RUDN Journal of Sociology 22:2 ► pp. 387 ff.
Trujillo, James P. & Judith Holler
2023. Interactionally Embedded Gestalt Principles of Multimodal Human Communication. Perspectives on Psychological Science 18:5 ► pp. 1136 ff.
This list is based on CrossRef data as of 18 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.