Cover not available
Part of
Eye-tracking in Interaction: Studies on the role of eye gaze in dialogue
Edited by Geert Brône and Bert Oben
[Advances in Interaction Studies 10] 2018
► pp. 265–300

Chapter 11
Mobile dual eye-tracking in face-to-face interaction

The case of deixis and joint attention

In face-to-face interaction deixis, i.e. (the use of) a particular class of linguistic items that have grammaticalised the space-, time- and person-bound structure of the participants’ subjective orientation in the speech event (Bühler, 1965[1934]), is intricately connected to visible acts of demonstration (prototypically pointing) and joint attention. A growing body of publications within the field of conversation analysis and research on multimodality acknowledges the central role that pointing plays in acts of deictic reference (Eriksson, 2009; Fricke, 2007; Goodwin, 2003; Kendon, 2004; Kita, 2003; Mondada, 2012a; Stukenbrock, 2009, 2014a, 2014b, 2015). Surprisingly, eye gaze has remained an unexplored area although it serves a variety of crucial functions in the participants’ on-line organisation of a joint focus of attention on deictically foregrounded entities in the immediate spatial surroundings (Stukenbrock, 2009, 2010, 2014a, 2014b, 2015). The few existing studies mainly rely on video-recordings that do not allow a precise analysis of eye gaze.

Drawing on innovative mobile eye-tracking technology, my paper explores different forms of gaze behaviour that systematically occur when participants direct their interlocutor’s attention to visible entities in the surroundings by means of deictic pointing. My data consists of mobile eye-tracking recordings undertaken with two pairs of eye-tracking glasses worn by participants in non-laboratory, everyday settings ((1) shopping together at a market, (2) searching for a book in a library, (3) conducting an informal conversation). The analysis is based on frame-precisely synchronised split-screen videos consisting of two complementary eye-tracking videos which allow a moment-by-moment reconstruction of the way in which the participants coordinate talk, body movements and gaze in the emergent interaction.

Keywords: mobile eye tracking, deixis, joint attention, gaze, gesture, conversation analysis, multimodality
Article outline
  • 1.Introduction
  • 2.Data, methodology and challenges
  • 3.Deictic practices in face-to-face interaction
    • 3.1What is deixis?
    • 3.2Verbal deixis combines with bodily pointing
    • 3.3Verbal deixis requests the addressee’s gaze
  • 4.Analyses: Gaze in deictic practices
    • 4.1The gesture-eye-link in demonstratio ad oculos
    • 4.2The target-eye-link in demonstratio ad oculos
    • 4.3The eye-to-eye-link in demonstratio ad oculos
  • 5.Conclusion: On eye-tracking natural gaze practices in demonstratio ad oculos
  • Conventions for the transcription gaze behaviour
    • The target-eye-link
    • The body-eye-link, more specifically the gesture-eye-link
    • The eye-to-eye-link
  • GAT 2 transcription conventions (GAT2, Selting et al. 2009; for the English translation cf. Couper-Kuhlen & Barth-Weingarten 2011)
    • Sequential structure
    • In- and outbreaths
    • Pauses
    • Segmental conventions
    • Laughter
    • Continuers
    • Accentuation
    • Final pitch movements of intonation phrases
    • Other conventions
  • Acknowledgments
  • Notes
  • References
Published online: 13 November 2018
DOI logo
https://doi.org/10.1075/ais.10.11stu
References
Argyle, M. & Cook, M.
(1976) Gaze and Mutual Gaze. London: Cambridge University Press.Google Scholar
Brooks, R. & Meltzoff, A. N.
(2014) Gaze following: A mechanism for building social connections between infants and adults. In M. Mikulincer & P. R. Shaver (Eds.), Mechanisms of social connection: From brain to group (pp.167–183). Washington, D.C.: American Psychological Association.DOI logoGoogle Scholar
Bühler, K.
(1965 [1934]) Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart: Gustav Fischer Verlag.Google Scholar
(2011 [1934]) Theory of Language. The Representational Function of Language. (translated by D. F. Goodwin in collaboration with A. Eschbach). Amsterdam, Philadelphia: John Benjamins.DOI logoGoogle Scholar
Clark, A. T. & Gergle, D.
(2011, September). Mobile Dual Eye-Tracking Methods: Challenges and Opportunities. Paper presented at DUET 2011: Dual Eye Tracking Workshop at ECSCW 2011, Aarhus, Denmark.Google Scholar
(2012, February). Know what I’m talking about? Dual eye-tracking in multimodal reference resolution. Paper presented at DUET 2012: Dual Eye Tracking workshop at CSCW 2012, Bellevue, WA.Google Scholar
Clark, H. H., Schreuder, R. & Buttrick, S.
(1983) Common Ground and the Understanding of Demonstrative Reference. Journal of Verbal Learning and Verbal Behavior, 22(2), 245–258.DOI logoGoogle Scholar
Couper-Kuhlen, E. & Barth-Weingarten, D.
(2011)  A system for transcribing talk in interaction. GAT 2. English translation and adaption of Selting. Margret et al. (2009): Gesprächsanalytisches Transkriptionssystem 2. In: Gesprächsforschung - Online-Zeitschrift zur verbalen Interaktion, 12, 1–51.Google Scholar
Couper-Kuhlen, E. & Selting, M.
(2001) Studies in Interactional Linguistics. Amsterdam: John Benjamins.Google Scholar
Dao, A. N.
in prep.). Blickverhalten bei sprachlichen Lokalisierungen in der Interaktion. PhD thesis Freiburg/Br.
Deppermann, A. & Schmitt, R.
(2007) Koordination. Zur Begründung eines neuen Forschungsgegenstandes. In R. Schmitt (Ed.), Koordination. Analysen zur multimodalen Interaktion (pp.15–54). Tübingen: Narr.Google Scholar
De Stefani, E.
(2010) Reference as an interactively and multimodally accomplished practice. Organizing spatial reorientation in guided tours. In M. Pettorino et al. (Eds.), Spoken Communication (pp.137–170). Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar
De Stefani, E. & Gazin, A. -D.
(2014) Instructional sequences in driving lessons: Mobile participants and the temporal and sequential organization of actions. Journal of Pragmatics, 65, 63–79.DOI logoGoogle Scholar
De Stefani, E. & Mondada, L.
(2014) Reorganizing mobile formations: When ‘guided’ participants initiate reorientation in guided tours. Space and Culture, 17(2), 157–175.DOI logoGoogle Scholar
Diessel, H.
(2006) Demonstratives, joint attention, and the emergence of grammar. Cognitive Linguistics, 17(4), 463–489.DOI logoGoogle Scholar
Duchowski, A. T.
(2003) Eye Tracking Methodology. Theory and Practice. Berlin: Springer.DOI logoGoogle Scholar
Eriksson, M.
(2009) Referring as Interaction: On the Interplay Between Linguistic and Bodily Practices. Journal of Pragmatics, 41, 240–262.DOI logoGoogle Scholar
Fillmore, C.
(1997) Lectures on Deixis. Stanford, CA: CSLI Publ.Google Scholar
Fricke, E.
(2007) Origo, Geste und Raum. Lokaldeixis im Deutschen. Berlin: de Gruyter.Google Scholar
Garfinkel, H.
(1967) Studies in Ethnomethodology. Englewood Cliffs, Nj: Prentice-Hall.Google Scholar
Gergle, D. & Clark, A. T.
(2010) Effects of Shifting Spatial Context on Referential Form. Paper presented at the 20th Annual Meeting of the Society for Text and Discourse.Google Scholar
(2011) See What I’m Saying? Using Dyadic Mobile Eye-Tracking to Study Collaborative Reference. Proceedings of CSCW (2011), 435–444.Google Scholar
Goffman, E.
(1963) Behavior in Public Places. Notes on the Social Organization of Gatherings. New York: The Free Press.Google Scholar
(1981) Forms of Talk. Philadelphia: University of Pennsylvania Press.Google Scholar
Goodwin, C.
(1980) Restarts, Pauses, and the Achievement of a State of Mutual Gaze at Turn-Beginning. Sociological Inquiry, 50(3–4), 272–302.DOI logoGoogle Scholar
(1981) Conversational Organization: Interaction Between Speakers and Hearers. New York: Academic Press.Google Scholar
(2000) Action and embodiment within situated human interaction. Journal of Pragmatics, 32, 1489–1522.DOI logoGoogle Scholar
(2003) Pointing as Situated Practice. In S. Kita (Ed.), Pointing. Where Language, Culture, and Cognition Meet (pp.217–241). Mahwah, NJ: Erlbaum.Google Scholar
Gullberg, M. & Holmqvist, K.
(2006) What speakers do and what addressees look at. Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition, 14(1), 53–82.DOI logoGoogle Scholar
Gullberg, M. & Kita, S.
(2009) Attention to Speech-Accompanying Gestures: Eye Movements and Information Uptake. Journal of Nonverbal Behavior, 33, 251–277.DOI logoGoogle Scholar
Haddington, P., Mondada, L. & Neville, M.
(2013) Interaction and Mobility. Language and the Body in Motion. Berlin, Boston: de Gruyter.DOI logoGoogle Scholar
Hanks, W. F.
(1990) Referential practice. Language and lived space among the Maya. Chicago: University of Chicago Press.Google Scholar
Hanna, J. E. & Brennan, S. E.
(2007) Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57, 596–651.DOI logoGoogle Scholar
Hausendorf, H.
(2003) Deixis and speech situation revisited. The mechanism of perceived perception. In F. Lenz (Ed.), Deictic Conceptualisation of Space, Time and Person (pp.249–269). Amsterdam: John Benjamins.DOI logoGoogle Scholar
Hepburn, A. & Bolden, G.
(2012) The Conversation Analytic Approach to Transcription. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp.57–76). Wiley: Blackwell.DOI logoGoogle Scholar
Hindmarsh, J. & Heath, C.
(2000) Embodied reference: A study on deixis in workplace interaction. Journal of Pragmatics, 32(12), 1855–1878.DOI logoGoogle Scholar
Holler, J. & Kendrick, K.
(2015) Unaddressed participants’ gaze in multi-person interaction: Optimizing recipiency. Frontiers in Psychology, 6(98), 1–14.Google Scholar
Horsley, M., Eliot, M., Knight, B. A. & Reilly, R.
(Eds.) (2014) Current Trends in Eye Tracking Research. Heidelberg: Springer.DOI logoGoogle Scholar
Keevallik, L.
(2013) Here in time and space: Decomposing movement in dance instruction. In P. Haddington, L. Mondada & M. Nevile (Eds.), Interaction and Mobility. Language and the Body in Motion (pp.345–370). Berlin, Boston: de Gruyter.DOI logoGoogle Scholar
Kendon, A.
(1967) Some function of gaze-direction in social interaction. Acta Psychologica, 26, 22–63.DOI logoGoogle Scholar
(1990) Conducting Interaction. Patterns of Behavior in Focused Encounters. Cambridge, New York: Cambridge University Press.Google Scholar
(2004) Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press.DOI logoGoogle Scholar
Kita, S.
(Ed.) (2003) Pointing. Where Language, Culture, and Cognition Meet. Mahwah, NJ: Erlbaum.Google Scholar
Land, M. F.
(2006): Eye movement and the control of actions in everyday life. Progress in Retinal and Eye Research, 25, 296–324.DOI logoGoogle Scholar
Levinson, S. C.
(1983) Pragmatics. Cambridge: Cambridge University Press.DOI logoGoogle Scholar
(2004) Deixis. In L. R. Horn (Ed.), The handbook of pragmatics (pp.97–121). Oxford: Blackwell.Google Scholar
(2012) Action Ascription and Formation. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp.103–130). Wiley: Blackwell.DOI logoGoogle Scholar
Luhmann, N.
(1984) Soziale Systeme. Grundriß einer allgemeinen Theorie. Frankfurt/M.: Suhrkamp.Google Scholar
Mondada, L.
(2007) Multimodal Resources for Turn-Taking: Pointing and the Emergence of Possible Next Speakers. Discourse Studies, 9(2), 195–226.DOI logoGoogle Scholar
(2012a) Deixis: an integrated interactional multimodal analysis. In P. Bergmann, J. Brenning, M. Pfeiffer & E. Reber (Eds.), Prosody and Embodiment in Interactional Grammar (pp.173–206). Berlin, Boston: de Gruyter.DOI logoGoogle Scholar
(2012b) The Conversation Analytic Approach to Data Collection. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp.32–56). Wiley: Blackwell.DOI logoGoogle Scholar
Pomerantz, A.
(1984) Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In J. M. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp.57–101). Cambridge: Cambridge University Press.Google Scholar
van Quine, W. O.
(1960) Word and Object. Cambridge: Technology Press of the Massachusetts Institute of Technology.Google Scholar
Rossano, F.
(2012) Gaze Behavior in Face-to-Face Interaction. Nijmegen: Max Planck Institute for Psycholinguistics Series.Google Scholar
Schegloff, E. A.
(1968) Sequencing in Conversational Openings. American Anthropologist, 70(6), 1075–1095.DOI logoGoogle Scholar
(1996) Confirming Allusions: Toward an Empirical Account of Action. American Journal of Sociology, 102(1), 161–216.DOI logoGoogle Scholar
(2007) Sequence Organization in Interaction. Cambridge: Cambridge University Press.DOI logoGoogle Scholar
Schegloff, E. A. & Sacks, H.
(1973) Opening Up Closings. Semiotica, 8, 289–327.DOI logoGoogle Scholar
Selting, M. et al.
(2009) Gesprächsanalytisches Transkriptionssystem 2 (GAT 2). Gesprächsforschung – Onlinezeitschrift zur verbalen Interaktion, 10, 353–402.Google Scholar
Sidnell, J. & Stivers, T.
(Eds.) (2013) The Handbook of Conversation Analysis. Wiley: Blackwell.Google Scholar
Streeck, J.
(1993) Gesture as communication I: Its coordination with gaze and speech. Communication Monographs, 60, 275–299.DOI logoGoogle Scholar
(2002) Grammars, Words, and Embodied Meanings: On the Uses and Evolution of So and Like. Journal of Communication, 52(3), 581–596.DOI logoGoogle Scholar
(2009) Gesturecraft. The manu-facture of meaning. Amsterdam, Philadelphia: John Benjamins.DOI logoGoogle Scholar
(2014) Mutual gaze and recognition: Revisiting Kendon’s “Gaze direction in two-person conversation”. In M. Seyfeddinipur & M. Gullberg (Eds.), From Gesture in Conversation to Visible Action as Utterance. Essays in honor of Adam Kendon (pp.35–55). Amsterdam/Philadelphia: Benjamins.Google Scholar
Stukenbrock, A.
(2008) “Wo ist der Hauptschmerz?” – Zeigen am eigenen Körper in der medizinischen Kommunikation. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion, 9, 1–33.Google Scholar
(2009) Referenz durch Zeigen: Zur Theorie der Deixis. Deutsche Sprache, 37, 289–315.Google Scholar
(2010) Überlegungen zu einem multimodalen Verständnis der gesprochenen Sprache am Beispiel deiktischer Verwendungsweisen des Ausdrucks “so”. InLiSt –Interaction and Linguistic Structures, 47, 1–23.Google Scholar
(2012) Imagined spaces as a resource in interaction. Bulletin Suisse de Linguistique Appliquée, 96, 141–161.Google Scholar
(2014a) Take the words out of my mouth: Verbal instructions as embodied practices. Journal of Pragmatics, 65, 80–102.DOI logoGoogle Scholar
(2014b) Pointing to an ‘empty’ space: Deixis am Phantasma in face-to-face interaction. Journal of Pragmatics, 74, 70–93.DOI logoGoogle Scholar
(2015) Deixis in der face-to-face-Interaktion. Berlin, Boston: de Gruyter.DOI logoGoogle Scholar
(2016) Deiktische Praktiken: Zwischen Interaktion und Grammatik. In A. Deppermann, H. Feilke & A. Linke (Eds.), Sprachliche und kommunikative Praktiken (pp.81–126). Jahrburch 2015 des Instituts für Deutsche Sprache. Berlin, Boston: de Gruyter.DOI logoGoogle Scholar
(2018) Forward-looking: Where do we go with multimodal projections? In A. Deppermann & Jürgen Streeck (Eds.), Modalities and Temporalities: Convergences and divergences of bodily resources in interaction. Amsterdam: Benjamins, 31–68.Google Scholar
Stukenbrock, A. & Dao, A. N.
accepted). Joint attention in passing. What dual mobile eye-tracking reveals about gaze in coordinating embodied activities on a market.
Stukenbrock, A.
in prep.). Divergent foci of visual attention in joint projects.
Tomasello, M.
(2008) Origins of Human Communication. Cambridge/Massachusetts: MIT Press.Google Scholar
Weiß, C. & Auer, P.
(2016) Das Blickverhalten des Rezipienten bei Sprecherhäsitationen. Eine explorative Studie. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion , 17, 132–167.Google Scholar
Cited by

Cited by 8 other publications

Auer, Peter
2021. Turn-allocation and gaze: A multimodal revision of the “current-speaker-selects-next” rule of the turn-taking system of conversation analysis. Discourse Studies 23:2 ► pp. 117 ff. DOI logo
Auer, Peter
2021. Gaze selects the next speaker in answers to questions pronominally addressed to more than one co-participant. Interactional Linguistics 1:2 ► pp. 154 ff. DOI logo
Balantani, Angeliki
2021. Reference construction in interaction: The case of type-indicative “so”. Journal of Pragmatics 181 ► pp. 241 ff. DOI logo
Neto Carvalho, Isabel, Mandy Schiefner-Rohs & Carina Troxler
2022. Eye-Viewing als Verfahren der erziehungswissenschaftlichen Medienforschung. Fokussierte Videographie mit Eye-Tracking-Technologie am Beispiel der Erforschung von medienbezogenen Praktiken in Schule und Unterricht. ZQF – Zeitschrift für Qualitative Forschung 23:2 ► pp. 262 ff. DOI logo
Rasmussen, Gitte & Elisabeth Dalby Kristiansen
2022. The sociality of minimizing involvement in self-service shops in Denmark: Customers’ multi-modal practices of being, getting, and staying out of the way. Discourse & Communication 16:2 ► pp. 200 ff. DOI logo
Stukenbrock, Anja
2018. Chapter 1. Forward-looking. In Time in Embodied Interaction [Pragmatics & Beyond New Series, 293], ► pp. 31 ff. DOI logo
Stukenbrock, Anja
2020. Deixis, Meta-Perceptive Gaze Practices, and the Interactional Achievement of Joint Attention. Frontiers in Psychology 11 DOI logo
Stukenbrock, Anja
2023. Temporality and the cooperative infrastructure of human communication: Noticings to delay and to accelerate onward movement in mobile interaction. Language & Communication 92 ► pp. 33 ff. DOI logo

This list is based on CrossRef data as of 22 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.