Cover not available
Part of
Linking up with Video: Perspectives on interpreting practice and research
Edited by Heidi Salaets and Geert Brône
[Benjamins Translation Library 149] 2020
► pp. 203–233

Chapter 8
Eye-tracking in interpreter-mediated talk

From research to practice

A starting point for a multimodal analysis of interpreter-mediated interaction is the discussion on the pros and cons of new technologies especially for new forms of distant, remote or offsite interpreting. More specifically, in this contribution is argued that empirical multimodal analyses of participants’ and interpreters’ behavior in real-life interpreting settings may provide much-needed groundwork that helps to fuel this discussion. As one of the most central points of criticism raised by interpreting practitioners is that distant forms of interpreting lack certain characteristics that are typical for on-site or face-to-face interpreting (i.e. distant solutions are not true-to-life, Van Rotterdam & van den Hoogen 2011). It becomes increasingly important to pinpoint what these typical and natural characteristics are. Empirically grounded insights into these characteristics may then, in a research-technology-application loop, feed into innovative technologies, rendering novel interpreting solutions that are more true-to-life.

In this empirical analysis we focus on one specific phenomenon that has received increased attention in the recent literature in multimodal interaction analysis as well as interpreting studies, viz. eye gaze as a semiotic resource employed by speakers as well as hearers. The main questions here are how interpreters typically manage the turn-taking process in real-life interaction, among others by means of gaze, and how important visual access is to the co-participants for a smooth exchange. In order to answer the research questions, the researchers make use of eye-tracking technology to gain detailed information on interpreters’ and primary participants’ gaze behavior.

Article outline
  • 1.Face-to-face vs. remote interpreting
  • 2.Multimodal interaction: A brief technological state-of-the-art
  • 3.Interpreting: Multimodality & eye gaze
  • 4.Dataset & methodological considerations
  • 5.Empirical analysis
    • 5.1Multimodal feedback patterns in interpreter-mediated talk
      • a.Interpreter-directed feedback
      • b.Primary directed feedback
      • c.Dual feedback
    • 5.2Turn-taking in interpreter-mediated interactions
      • Adjacency pairs
      • Multiple turns
  • 6.Conclusion
  • Notes
  • References
  • Appendix. Transcription conventions
Published online: 13 January 2020
DOI logo
https://doi.org/10.1075/btl.149.09vra
References
Auer, Peter
2005 “Projection in interaction and projection in grammar.” Text 25 (1): 7–36. DOI logoGoogle Scholar
Bavelas, Janet., Linda Coates, and Trudy Johnson
2002 “Listener responses as a collaborative process: The role of gaze.” Journal of Communication, 566–580. DOI logoGoogle Scholar
Bot, Hanneke
2005 Dialogue Interpreting in Mental Health. Amsterdam and New York: Rodopi Publishers.Google Scholar
Braun Sabine
2013 “Keep your distance? Remote interpreting in legal proceedings: A critical assessment of a growing practice.” Interpreting: international journal of research and practice in interpreting, 15 (2), pp.200–228. DOI logoGoogle Scholar
Braun, Sabine
2015 “Remote interpreting.” In Routledge Encyclopedia of Interpreting Studies, ed. by Franz Pöchhacker, 346–348. London/New York: Routledge.Google Scholar
2017 “What a micro-analytical investigation of additions and expansions in remote interpreting can tell us about interpreters’ participation in a shared virtual space.” Journal of Pragmatics 107, 165–177. DOI logoGoogle Scholar
Braun, Sabine, and L. Judith Taylor
2012 Videoconference and Remote Interpreting in Criminal Proceedings. Cambridge/Antwerp: Intersentia.Google Scholar
Brenger, Bela, and Irene Mittelberg
(2015) “Shakes, nods and tilts. Motion-capture data profiles of speakers’ and listeners’ head gestures.” In: Proceedings of the 3rd Gesture and Speech in Interaction (GESPIN) Conference, Sept 2015, Nantes. 43–48.Google Scholar
Brône, Geert and Bert Oben
2015 “InSight Interaction. A multimodal and multifocal dialogue corpus.” Language Resources and Evaluation 49: 195–214. DOI logoGoogle Scholar
Brône, Geert, Bert Oben , Annelies Jehoul , Jelena Vranjes, and, Kurt Feyaerts
2017 “Eye gaze and viewpoint in multimodal interaction management.” Cognitive Linguistics 28 (3): 449–484. DOI logoGoogle Scholar
Clayman, Steven E.
2013 “Turn-Constructional Units and the Transition-Relevance-Place.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 150–166, Wiley-Blackwell.Google Scholar
Clark, Herbert
1996 Using Language. Cambridge University press: Cambridge. DOI logoGoogle Scholar
Clark, Herbert, and Susan Brennan
1991 “Grounding in communication.” In Perspectives on socially shared cognition, ed. by Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, Washington: APA Books. DOI logoGoogle Scholar
Davidson, Brad
2002 “A model for the construction of conversational common ground in interpreted discourse.” Journal of Pragmatics 34: 1273–1300. DOI logoGoogle Scholar
2015 “Gaze.” In Routledge Encyclopedia of Interpreting Studies, ed. by Franz Pöchhacker, Nadja Grbic, Peter Mead, and Robin Setton, 168–168, New York: Routledge.Google Scholar
Davitti, Elena, and Sergio Pasquandrea
2017 “Embodied participation: What multimodal analysis can tell us about interpreter-mediated encounters in pedagogical settings.” Elsevier Journal of Pragmatics 107: 105–128. DOI logoGoogle Scholar
Deppermann, Arnulf
2013 “Turn-design at turn-beginnings: Multimodal resources to deal with tasks of turn-construction in German.” Journal of Pragmatics 46 (1): 91–121. DOI logoGoogle Scholar
Ellis, S. Ronald
2004 “Videoconferencing in refugee hearings.” Ellis Report to the Immigration and Refugee Board Audit and Evaluation Committee. [URL].Google Scholar
Englund Dimitrova, Birgitta
1997 “Degree of Interpreter Responsibility in the Interaction Process in Community Interpreting.” In The Critical Link: Interpreters in the Community, ed. by Silvana E. Carr, 147–164, Amsterdam/Philadelphia: Benjamins.Google Scholar
Esteban Causo, Jose
2012 “Conference interpreting with information and communication technologies. Experiences from the European Commission DG Interpretation.” In Videoconference and remote interpreting in criminal proceedings, ed. by Sabine Braun, and Judith Taylor, 227–232 . Antwerp: Intersentia.Google Scholar
Ford, Cecilia, Barbara Fox, and A. Sandra Thompson
1996 “Practices in the construction of turns: The TCU revisted.” Pragmatics 6 ( 3 ): 427–454 . DOI logoGoogle Scholar
Gardner, Rod
2001 When Listeners Talk: Response tokens and listener stance. Amsterdam: J. Benjamins Publishing. DOI logoGoogle Scholar
Gerwing, Jennfer, & Bavelas Janet
(2013) “The social interactive nature of gestures: Theory, assumptions, methods, and findings. In Body-language-communication, Volume I, Contemporary approaches, ed. by Cornelia Müller, Allan Cienki, Ellen Fricke, Silva Ladewig, David McNeil & Sedinha Tessendorf. 816-831. Berlin: Mouton De Gruyter.Google Scholar
Goffman, Erving
1981 Forms of talk. Oxford: Basil Blackwell.Google Scholar
Goodwin, Charles
1981 Conversational Organization. Interaction between Speakers and Hearers. New York, London: Academic.Google Scholar
Hayashi, Makoto
2013 “Turn Allocation and Turn Sharing.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell and Tanya Stivers, 167–190, Wiley-Blackwell.Google Scholar
Heath, Christian
1986 Body Movement and Speech in Medical Interaction. Cambridge, U.K.: Cambridge University Press. DOI logoGoogle Scholar
Holler, Judith, Kobin H.,Kendrick, Marisa Casillas, and Stephen, C. Levinson
2016 Turn-Taking in Human Communicative Interaction. Lausanne: Frontiers Media. DOI logo.Google Scholar
Holler, Judith, and Kobin. H. Kendrick
2015.“Unaddressed participants’ gaze in multi-person interaction: Optimizing recipiency.” Frontiers in Psychology 6: 98. DOI logoGoogle Scholar
Jokinen, Kristiina
2010 “Non-verbal signals for turn-taking & feedback.” Proc. of 7th Int. Conf. on Language Resources & Evaluation (LREC) International Universal Communication Symposium.Google Scholar
Kendon, Adam
1967 “Some functions of gaze-direction in social interaction.” Acta Psychologica 26: 22–63. DOI logoGoogle Scholar
Kipp, Michael
2014 “ANVIL: A Universal Video Research Tool.” In Handbook of Corpus Phonology, ed. by Jacques Durand, Ulrike Gut, and Gjert Kristofferson, 420–436. Oxford: Oxford University Press.Google Scholar
Krystallidou, Demi
2016 “Investigating the Interpreter’s Role(s): The A.R.T. Framework.” Interpreting 18 (2): 172–197. DOI logoGoogle Scholar
Lang, Ranier
1978 “Behavioral aspects of liaison interpreters in Papua New Guinea: some preliminary observations.” In Language Interpretation and Communication, ed. by David Gerver, and H. Wallace Sinaiko, 231–244. New Yotk/London: Plenum Press. DOI logoGoogle Scholar
Lerner, Gene. H.
2003 “Selecting next speaker: The context-sensitive operation of a context-free organization.” Language in Society 32: 177–201. DOI logoGoogle Scholar
Llewellyn-Jones, Peter, and Robert. G. Lee
2014 Redefining the Role of the Community Interpreter: The Concept of role-space. Lincoln: SLI Press.Google Scholar
Li, Shuangyu
2015 “Nine Types of Turn-taking in Interpreter-mediated GP Consultations.” Applied Linguistics Review 6 (1): 73–96. DOI logoGoogle Scholar
Licoppe, Christian, and Clair-Antoine Veyrier
2017 “How to show the interpreter on screen? The normatice organization of visual ecologies in multilingual courtrooms with video links.” Elsevier Journal of Pragmatics 107: 147–164. DOI logoGoogle Scholar
Linell, Per, Linda Jönsson, and Cecilia Wadensjö
1992 “Establishing Communicative Contact through a Dialogue Interpreter.” In LSP Communication/Fachsprachliche Kommunikation, ed. by Annette Grindsted, and Johannes Wagner, 125–142, Tübingen: Narr Verlag.Google Scholar
Mason, Ian
2012 “Gaze, positioning and identity in interpreter-mediated dialogues.” In Coordinating Participation in Dialogue Interpreting, ed. by Claudio. Baraladi and Laura Gavioli, 177–199, Amsterdam/Philadelphia: Benjamins Publishing. DOI logoGoogle Scholar
Merlini, Raffaela, and Roberta Favaron
2005 “Examining the “voice of interpreting” in speech pathology.” Interpreting 7 (2): 263–302. DOI logoGoogle Scholar
Mondada, Lorenza
2007 “Multimodal resources for turn-taking: pointing and emergence of next speakers.” Discourse Studies: 194–225. DOI logoGoogle Scholar
2013 “Video as a tool in the social sciences.” In Body – Language – Communication: An International Handbook on Multimodality in Human Interaction , Vol. 1, ed. by Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill, and Sedinha Teßendorf, 982–992. Berlin: De Gruyter Mouton.Google Scholar
2016 “Challenges of multimodality: Language and the body in social interaction.” Journal of Sociolinguistics 20–3, 336–366. DOI logoGoogle Scholar
Moser-Mercer, Barbara
2003 “Remote interpreting: assessment of human factors and performance parameters’” Communicate! Summer 2003. Available at [URL].Google Scholar
2005 “Remote interpreting: issues of multi-sensory integration in a multilingual task,” Meta, 50 (2), 727–738. DOI logoGoogle Scholar
Oben, Bert, and Geert Brône
2015 “What you see is what you do: On the relation between gaze and gesture in multimodal alignment.” Language and Cognition 7 (4): 546–562. DOI logoGoogle Scholar
Pasquandrea, Sergio
2011 “Managing multiple actions through multimodality: Doctors’ involvement in interpreter-mediated interactions.” Language in Society 40 (4): 455–481. DOI logoGoogle Scholar
Pitsch Karola, Bernhard-Andreas Brüning, Christian Schnier, Holger Dierker, and Sven Wachsmuth
2010 Linking conversation analysis and motion capturing: How to robustly track multiple participants. In Proceedings of the LREC Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality (MMC 2010), ed. by : Michael Kipp, Jean-Claude Martin, Patrizia Paggio, and Dirk Heylen, 63–69.Google Scholar
Price, Erika, Eliseo Pérez-Stable, Dana Nickleach, Monica López, and Leah Karliner
2012 “Interpreter perspectives of in-person, telephonic, and videoconferencing medical interpretation in clinical encounters.” Patient Education & Counseling, 87 (2), 226–232. DOI logoGoogle Scholar
Rossano, Federico
2013 “Gaze in Conversation.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 308–329. Oxford: Wiley-Blackwell.Google Scholar
Roziner, Ilan, and Miriam Shlesinger
2010 “Much ado about something remote: Stress and performance in remote interpreting.” Interpreting 12 (2): 214–247. DOI logoGoogle Scholar
Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson
1974 “A simplest systematics for the organization of turn-taking in conversation.” Language 50: 696–735. DOI logoGoogle Scholar
Schmitt, Reinhold
2005.“Zur multimodalen Struktur von turn-taking.” Gesprächsforschung 6: 17–61.Google Scholar
Selting, Margaret. et al.
2009.“Gesprächsanalytisches Transkriptionssystem 2 (GAT 2).” Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 10: 353–402. [[URL]]Google Scholar
Sloetjes, Han, and Peter Wittenburg
2008 “Annotation by category – ELAN and ISO DCR.” In: Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008).Google Scholar
Stivers, Tania and Federico Rossano
2010 “Mobilizing Response.” Research on Language and Social Interaction 43 (1): 3–31. DOI logoGoogle Scholar
Vandemoortele, Sarah, Stijn De Beugher, Geert Brône, Kurt Feyaerts, Toon Goedemé, Thomas De Baets, and Stijn Vervliet
2016 Into the Wild. Musical Communication in Ensemble Playing. Leuven, Acco.Google Scholar
Van Rotterdam, Peter, and Ronald van den Hoogen
2011 “True-to-life requirements for using videoconferencing in legal proceedings.” In Videoconference and remote interpreting in criminal proceedings, ed. by Sabine Braun, and Judith L. Taylor, 187–197. Guildford: University of Surrey.Google Scholar
Vranjes, Jelena, Geert Brône, and Kurt Feyaerts
2018 “Dual feedback in interpreter-mediated interactions: on the role of gaze in the production of listener responses.” Journal of Pragmatics 134: 15–30. DOI logoGoogle Scholar
Vranjes, Jelena, Hanneke Bot, Kurt Feyaerts, and Geert Brône
(2019) “Displaying recipiency in an interpreter-mediated dialogue: an eye-tracking study. In Eye-tracking in interaction. Studies on the role of eye gaze in dialogue, ed. by Bert Oben and G. Brône . Amsterdam: Benjamins Publishing.Google Scholar
Vranjes, Jelena, Hanneke Bot , Kurt Feyaerts, and Geert Brône
(2019)  “Affiliation in interpreter-mediated therapeutic talk: on the relationship between gaze and head nods.” Interpreting. DOI logoGoogle Scholar
Vranjes, Jelena
2017 “Dual feedback and the management of intersubjectivity in interpreter-mediated talk. Mapping Multimodal Dialogue (MaMuD 4).” Frankfurt/Oder, Germany, 23–24 November 2017.Google Scholar
Wadensjö, Cecilia
1998 Interpreting as interaction. London and New York: Longman.Google Scholar
Cited by

Cited by 4 other publications

Argaman, Einav
2023. Neither speaker nor recipient: The middle-distance look of unaddressed participants. Discourse Studies 25:5 ► pp. 573 ff. DOI logo
Doherty, Stephen, Natalie Martschuk, Jane Goodman-Delahunty & Sandra Hale
2022. An Eye-Movement Analysis of Overt Visual Attention During Consecutive and Simultaneous Interpreting Modes in a Remotely Interpreted Investigative Interview. Frontiers in Psychology 13 DOI logo
Salaets, Heidi & Katalin Balogh
2023. Are interpreters and interpreting technology ready for the post-Covid era?. In Introducing New Hypertexts on Interpreting (Studies) [Benjamins Translation Library, 160], ► pp. 254 ff. DOI logo
Zhang Xiaojun, Gloria Corpas Pastor & Jing Zhang
2023. Chapter 7. Videoconference interpreting goes multimodal. In Interpreting Technologies – Current and Future Trends [IVITRA Research in Linguistics and Literature, 37], ► pp. 169 ff. DOI logo

This list is based on CrossRef data as of 23 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.