Cover not available
Part of
Mobile Eyetracking: New avenues for the study of gaze in social interaction
Edited by Elisabeth Zima and Anja Stukenbrock
[Pragmatics & Beyond New Series 351] 2025
► pp. 24–67

Chapter 2
Why research on gaze in social interaction needs mobile eye tracking

 | University of Freiburg
 | University of Freiburg
 | University of Freiburg
This chapter challenges the prevailing practice in ethnomethodologically inspired interaction research (EMCA) of recording and analyzing gaze in social interactions from an observer’s perspective. Contrary to the assumption that this perspective is ‘natural’, we demonstrate systematic divergence between analysts’ and participants’ viewpoints and argue that ‘the standard procedure’ of video recording does not allow for reliable reconstruction of when interactants’ look to or away from each other. Three intercoder reliability studies, comparing the transcription of mutual gaze in triadic interactions from an observer’s perspective with eye tracking data, support this argument. They reveal the inherent unreliability of gaze coding from an interaction-external perspective, while showing that gaze transcription based on eye tracking data is much less error-prone. It minimizes the need to infer gaze targets from ambiguous bodily cues and thus emerges as the preferred method for accurately reconstructing mutual gaze in interactional sense-making.
Keywords: mobile eye tracking, video recording from a bystander’s perspective, mutual gaze, intercoder reliability
Article outline
  • 1.Introduction
  • 2.Epistemological and methodological questions of video recording in EMCA
  • 3.EMCA methodology and epistemology and the study of human gaze: Video recording versus eye tracking
  • 4.Testing the reliability of gaze transcription in standard EMCA data versus eye tracking data
    • 4.1Study design
    • 4.2Results
      • Study 1a (no sound): Mutual gaze transcription from an observer’s perspective in muted clips
      • Study 1b (observer’s perspective, with sound)
      • Study 2: Transcribing (mutual) gaze on the basis of eye tracking data
  • 5.Conclusions
  • Author queries
  • Notes
  • References
This content is being prepared for publication; it may be subject to changes.
References (54)
References
Auer, Peter, Laner, Barbara, Pfeiffer, Martin, and Kerstin Botsch (in press). “Noticing and Assessing Nature: A Multimodal Investigation of the Format ‘Perception Imperative + Exclamative’ based on Mobile Eye-tracking Data.” In New Perspectives in Interactional Linguistic research, ed. by Margaret Selting, and Dagmar Barth-Weingarten. Amsterdam: Benjamins. DOI logo
Bergmann, Jörg R. 1985. “Flüchtigkeit und methodische Fixierung sozialer Wirklichkeit: Aufzeichnungen als Daten der interpretativen Soziologie.“ In Entzauberte Wissenschaft: Zur Relativität und Geltung soziologischer Forschung, ed. by Wolfgang Bonß, and Heinz Hartmann, 299–320. Göttingen: Schwartz.Google Scholar
Bowers, N. R., Boehm, A. E., Roorda, A. 2019. “The effects of fixational tremor on the retinal image”. Journal of Vision 19(11):8. DOI logoGoogle Scholar
Dalby Kristiansen, Elisabeth, and Gitte Rasmussen. 2021. “Eyetracking Recordings as Data in EMCA Studies: Exploring Possibilities and Limitations.” Social Interaction. Video-Based Studies of Human Sociality 4 (4). DOI logoGoogle Scholar
Deppermann, Arnulf, and Lorenza Mondada. 2018. “Overtaking as an Interactional Achievement: Video Analyses of Participants’ Practices in Traffic.” Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 19: 1–131. [URL]
Goffman, Erving. 1979. “Footing.” Semiotica 25: 1–29. DOI logoGoogle Scholar
Goodwin, Charles. 1981. Conversational Organization: Interaction between Speakers and Hearers. London: Academic Press.Google Scholar
. 1993. “Recording Human Interaction in Natural Settings.” Pragmatics 3 (2): 181–209.Google Scholar
. 1994. “Professional Vision.” American Anthropologist 96 (3): 606–633. DOI logoGoogle Scholar
Gubina, Alexandra. 2021. “Availability, Grammar, and Action Formation: On Simple and Modal Interrogative Request Formats in Spoken German.” Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 19: 1–131. [URL] 22, 272–303.
. 2022. Grammatik des Handelns in der sozialen Interaktion. Göttingen: Verlag für Gesprächsforschung.Google Scholar
Hall, Edward. 1966. The Hidden Dimension. Garden City, NY: Anchor Books.Google Scholar
Heath, Christian, Hindmarsh, Jon, and Paul Luff (eds). 2010. Video in Qualitative Research. Analysing Social Interaction in Everyday Life. London: Sage. DOI logoGoogle Scholar
Hirschauer, Stefan. 2006. “Putting Things into Words. Ethnographic Description and the Silence of the Social.” Human Studies 29 (4): 413–441. [URL]
Holmqvist, Kenneth, Nyström, Marcus, Andersson, Richard, Dewhurst, Richard, Halszka, Jarodzka, and Joost van de Weijer (eds). 2011. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press.Google Scholar
Kendon, Adam. 1973. “The Role of Visible Behaviour in the Organization of Social Interaction.” In Social Communication and Movement: Studies of Interaction and Expression in Man and Chimpanzee, ed. by Mario von Cranach, and Ian Vine, 29–74. New York: Academic Press.Google Scholar
. 1990. “Movement Coordination in Social Interaction: Some Examples Described.” In Conducting Interaction. Patterns of Behaviour in Focused Encounters, ed. by Adam Kendon, 91–115. Cambridge: Cambridge University Press.Google Scholar
Knoblauch, Hubert. 2012. “Videography. Focused Ethnography and Video Analysis.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 69–84. Frankfurt am Main: Peter Lang.Google Scholar
Landis, J. Richard, and Gary G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33 (1): 159–174. DOI logoGoogle Scholar
Laurier, Eric. 2008. “Drinking up Endings: Conversational Resources of the Café.” Language and Communication 28 (2): 165–181. DOI logoGoogle Scholar
Laurier, Eric, and Chris Philo. 2006. “Natural Problems of Naturalistic Video Data.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 183–193. Oxford: Peter Lang.Google Scholar
Laurier, Eric, and Tobias Boelt Back. 2023. “Recurrent Problems and Recent Experiments in Transcribing Video: Live Transcribing in Data Sessions and Depicting Perspective.” In Ethnomethodological Conversation Analysis in Motion: Emerging Methods and Technologies, ed. by Pentti Haddington, Tiina Eilittä, Antti Kamunen, Laura Kohonen-Aho, Tuire Oittinen, Lira Rautiainen, and Anna Vatanen, 245–263. London: Routledge. DOI logoGoogle Scholar
Luckmann, Thomas. 2012. “Some Remarks on Scores in Multimodal Sequential Analysis.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 29–34. Frankfurt am Main: Peter Lang.Google Scholar
Lynch, Michael, and David Bogen. 1994. “Harvey Sacks’s Primitive Natural Science.” Theory, Culture & Society 11 (4): 65–110. DOI logoGoogle Scholar
McHugh, Mary L. 2021. “Interrater Reliability: The Kappa Statistic.” Biochem Med 22 (3): 276–82.Google Scholar
McIlvenny, Paul. 2019. “Inhabiting Spatial Video and Audio Data: Towards a Scenographic Turn in the Analysis of Social Interaction.” Social Interaction. Video-Based Studies of Human Sociality 2 (1). DOI logoGoogle Scholar
Mohn, Elisabeth. 2012. “Permanent Work on Gazes. Video Ethnography as an Alternative Methodology.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 173–182. Frankfurt am Main: Peter Lang.Google Scholar
Mondada, Lorenza. 2008. “Using Video for a Sequential and Multimodal Analysis of Social Interaction: Videotaping Institutional Telephone Calls.” Forum Qualitative Sozialforschung Forum Qualitative Social Research 9 (3). DOI logoGoogle Scholar
. 2009. “Emergent Focused Interactions in Public Places: A Systematic Analysis of the Multimodal Achievement of a Common Interactional Space.” Journal of Pragmatics 41 (10): 1977–1997. DOI logoGoogle Scholar
. 2012a. “Video Recording as the Reflexive Preservation and Configuration of Phenomenal Features for Analysis.” In Video Analysis Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 51–68. Frankfurt am Main: Peter Lang.Google Scholar
. 2012b. “The Conversation Analytic Approach to Data Collection.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 32–56. Oxford: Wiley-Blackwell. DOI logoGoogle Scholar
. 2019. “Practices for Showing, Looking, and Videorecording: The Interactional Establishment of a Common Focus of Attention.” In Embodied Activities in Face-to-Face and Mediated Settings: Social Encounters in Time and Space, ed. by Elisabeth Reber, and Cornelia Gerhardt, 63–104. Cham: Palgrave Macmillan. DOI logoGoogle Scholar
Oloff, Florence. 2022. “The Particle Jako (‘Like’) in Spoken Czech: From Expressing Comparison to Mobilizing Affiliative Responses.” Frontiers in Psychology 12 (662115). DOI logoGoogle Scholar
Pehkonen, Samu, Rauniomaa, Mirka, and Pauliina Siitonen. 2021. “Participating Researcher or Researching Participant? On Possible Positions of the Researcher in the Collection (and Analysis) of Mobile Video Data. Social Interaction. Video-Based Studies of Human Sociality 4 (2). DOI logoGoogle Scholar
Pekarek Doehler, Simona, Polak-Yitzhaki, Hilla, Li, Xiaoting, Stoenica, Ioana Maria, Havlík, Martin, and Leelo Keevallik. 2022. “Multimodal Assemblies for Prefacing a Dispreferred Response: A Cross-Linguistic Analysis.” Frontiers in Psychology 12 (689275). DOI logoGoogle Scholar
Peräkylä, Anssi, and Johanna E. Ruusuvuori. 2012. “Facial Expression and Interactional Regulation of Emotion.” In Emotion in Interaction, ed. by Anssi Peräkylä, and Marija L. Sorjonen, 64–91. Oxford: Oxford University Press. DOI logoGoogle Scholar
Rossano, Federico. 2013. “Gaze in Conversation.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 308–329. Oxford: Wiley-Blackwell.Google Scholar
Sacks, Harvey. [1967] 1992. Lectures on Conversation. (2 Volumes). Oxford: Basil Blackwell.Google Scholar
Schmitz, H. Walter. 2020. “Wenn der Hörer sichtbar wird. Ch. Goodwins „The Interactive construction of a Sentence in Natural Conversation“ und die ethnomethodologische Konversationsanalyse.” In: Sine ira et studio: Disziplinübergreifende Annäherungen an die zwischenmenschliche Kommunikation, ed. by Robin Kurilla, Karin Kolb-Albers, Hannes Krämer, and Karola Pitsch, 97–115. Berlin: Springer. DOI logoGoogle Scholar
Schutz, Alfred. 1972. “The Problem of Social Reality.” In Collected Papers. Volume 1, ed. by Maurice Natanson. 277–293. The Hague: Martinus Nijhoff.Google Scholar
Stivers, Tanya. 2021. “Is Conversation Built for Two? The Partitioning of Social Interaction.” Research on Language and Social Interaction 54 (1): 1–19. DOI logoGoogle Scholar
Stoenica, Ioana-Maria, and Sophia Fiedler. 2022. “Multimodal Practice for Mobilizing Response: The Case of Turn-final Tu Vois ‘You See’ in French Talk-in-Interaction.” Frontiers in Psychology 12: 249–268. DOI logoGoogle Scholar
Stukenbrock, Anja. 2015. Deixis in der face-to-face-Interaktion. Berlin, Boston: de Gruyter. DOI logoGoogle Scholar
. 2018. “Forward-looking: Where do we go with multimodal projections?” In Modalities and Temporalities: Convergences and Divergences of Bodily Resources in Interaction, ed. by Arnulf Deppermann and Jürgen Streeck. 31–68. Amsterdam: Benjamins. DOI logoGoogle Scholar
. 2020. “Meta-perceptive practices.” Frontiers in Psychology. DOI logoGoogle Scholar
. 2021. “Multimodal Gestalts and Their Change Over Time: Is Routinization Also Grammaticalization?” Frontiers in Communication, 6 (6622406). DOI logoGoogle Scholar
Stukenbrock, Anja, and Anh-Nhi Dao. 2019. “Joint Attention in Passing: What Dual Mobile Eye Tracking Reveals About Gaze in Coordinating Embodied Activities at a Market.” In Embodied Activities in Face-to-face and Mediated Settings: Social Encounters in Time and Space, ed. by Elisabeth Reber, and Cornelia Gerhardt, 177–213. Cham: Palgrave Macmillan. DOI logoGoogle Scholar
Tomasello, Michael. 2008. Origins of Human Communication. Cambridge: MIT Press. DOI logoGoogle Scholar
Tomasello, Michael, Hare, Brian, Lehmann, Hagen, and Josep Call. 2007. “Reliance on Head versus Eyes in the Gaze Following of Great Apes and Human Infants: The Cooperative Eye Hypothesis.” Journal of Human Evolution 52 (3): 314–320. DOI logoGoogle Scholar
Tuncer, Sylvaine, and Pentti Haddington. 2019. “Looking at and Seeing Objects: Instructed Vision and Collaboration in the Laboratory.” Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 20: 435–360. [URL]
Wittenburg, Peter, Brugman, Hennie, Russel, Albert, Klassmann, Axel, and Han Sloetjes. 2006. “ELAN: a Professional Framework for Multimodality Research.” In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), 1556–1559. Genua: European Language Resources Association (ELRA).Google Scholar
Zhisheng, Wang, Nagai, Yukari, Zhu, Dan, Liu, Jiahui, and Ninyu Zou. 2019. “Based on Creative Thinking to Museum Lighting Design Influences to Visitors Emotional Response Levels Theory Research.” IOP Conference Series: Materials Science and Engineering 573: 1–7. DOI logoGoogle Scholar
Zima, Elisabeth. 2020. “Gaze and Recipient Feedback in Triadic Storytelling Activities.” Discourse Processes 57 (9): 725–748. DOI logoGoogle Scholar