Chapter 6
Quantifying the interplay of gaze and gesture in deixis using an experimental-simulative approach
Gaze and gestures have been studied qualitatively, e.g., by Kendon and others (Kendon, 1990; McNeill, 1992; Kendon, 2004; McNeill, 2006). A quantitative assessment of gaze
and gestures in dialogue, in particular regarding precise orientations, positions and timings, however, has only been
possible with the advent of advanced measuring technologies, such as motion capturing or eye tracking. Especially in
dynamic natural environments, when interlocutors are concerned with their surrounding three-dimensional environment, a
precise three-dimensional reconstruction of the set-up is required to analyze the produced multimodal utterances.
In this article we review several of our past projects with a focus on our experimental-simulative approach in which we
combine state-of-the-art tracking technologies with 3D representations and computer simulations to test different
hypotheses in the context of deixis in human-human interaction.
Article outline
- 1.Introduction
- 2.An experimental-simulative approach
- 2.1Quantification using tracking technology
- 2.2Reconstructing a 3D situation model
- 2.3Simulations in the 3D situation model
- 2.4Analysis and results
- 2.5Summary
- 3.Example A: A study on deixis
- 3.1Background
- 3.2Scenario
- 3.3Tracking setup
- 3.4Reconstructed 3D situation model
- 3.5Simulations in the 3D situation model
- 3.6Results
- 3.6.1Distribution of hand positions
- 3.6.2Direction of pointing gestures
- 4.Example B: Deictic gaze of two interlocutors in a search scenario
- 4.1Background
- 4.2Scenario
- 4.3Method
- Tracking
- Reconstructed 3D situation model
- Simulations in the 3D situation model
- 4.4Results
- 4.4.1Automatic annotation of gaze targets
- 4.4.2Identifying patterns of shared gaze
- 4.5Summary
- 5.Example C: Gaze and deixis in shared space of two interlocutors
- 5.1Background
- 5.2Scenario
- 5.3Method
- Tracking
- Reconstructed 3D situation model
- Simulations in the 3D situation model
- 5.4Results
- 5.5Summary
- 6.Conclusion
- 7.Outlook: Affordable tracking solutions
- Acknowledgment
-
Notes
-
References