Cover not available
Part of
Moving Bodies in Interaction – Interacting Bodies in Motion: Intercorporeality, interkinesthesia, and enaction in sports
Edited by Christian Meyer and Ulrich v. Wedelstaedt
[Advances in Interaction Studies 8] 2017
► pp. 193–214

Chapter 8
Sound joined actions in rowing and swimming

The present chapter introduces the method of sonification as a tool for studying intercorporeality and enactment. We show that auditory movement information can support motor perception as well as the control of movements, and explain these effects by mechanisms which are consistent with the enactment approach. Providing additional auditory information about a movement enables the acting individual as well as observers to perceive the movement in exactly the same way via audition. Thus, a sonification can establish a common percept for all interaction partners, which corresponds well to the concept of intercorporeality. Furthermore, we show that sonifications can be specifically designed to constitute a variety of frameworks for the analysis of interpersonal coordination and intercorporeality.

Keywords: sonification, internal modelling, motor control, motion perception, multisensory integration
Article outline
  • Introduction
  • The method of movement sonification
  • Mechanisms of multisensory integration
  • Movement sonification and sports
  • Modifying and optimizing sensorimotor control
  • Activation of the action-observation-system and the motor loop during the observation of a kinematic sonification
  • Retrieval of movement representations
  • Discrimination of rowing patterns
  • Coordinating movements with sonified movements of another person
  • Modification of team performance
  • Summary and conclusion
  • References
Published online: 14 August 2017
DOI logo
https://doi.org/10.1075/ais.8.08sch
References
Allison, T., Puce, A., & McCarthy, G.
(2000) Social perception from visual cues: Role of the STS region. Trends in Cognitive Science, 4:267–278. DOI logoGoogle Scholar
Bangert, M., & Altenmüller, E. O.
(2003) Mapping perception to action in piano practice: a longitudinal DC-EEG study. BMC Neuroscience, 4: 26. DOI logoGoogle Scholar
Boyer, E. O., Babayan, B. M., Bevilacqua, F., Noisternig, M., Warusfel, O., Roby-Brami, A., Hanneton, S., & Viaud-Delmon, I.
(2013) From ear to hand: the role of the auditory-motor-loop in pointing to an auditory source. Frontiers in Computational Neuroscience, 7:26. DOI logoGoogle Scholar
Brock, H., Schmitz, G., Baumann, J., & Effenberg, A. O.
(2012) If motion sounds: Movement sonification based on inertial motion data. Proceedings of the 9th Conference of the International Sports Engineering Association (ISEA), Boston, USA, 556–561.Google Scholar
Calvert, G. A., Brammer, M. J., & Iversen, S. D.
(1998) Crossmodal Identification. Trends in Cognitive Sciences, 2(7), 247–253. DOI logoGoogle Scholar
Cohen, J.
(1988) Statistical Power Analysis for the Behavioral Sciences, 2. Ed., Hillsdale: Lawrence Erlbaum Associates.Google Scholar
D’Ausilio, Novembre G., Fadiga, L., & Keller, P. E.
(2014) What can music tell us about social interation? Trends in Cognitive Sciences, 19(3), 111–114. DOI logoGoogle Scholar
Demos, A. P., Chaffin, R., Begosh, K. T., Daniels, J. R., & Marsh, K. L.
(2012) Rocking to the Beat: Effects of Music and Partner’s Movements on Spontaneous Interpersonal Coordination. Journal of Experimental Psychology: General, 141(1), 49–53. DOI logoGoogle Scholar
Desmurget, M., Reilly, K. T., Richard, N., Szathmari, A., Mottolese, C., & Sirigu, A.
(2009) Movement Intention After Parietal Cortex Stimulation in Humans. Science, 324, 811–813.Google Scholar
Effenberg, A. O.
(1996) Sonification – Ein akustisches Informationskonzept zur menschlichen Bewegung. [Sonification – a concept of acoustic information on human motor behavior]. Schorndorf: Hofmann.Google Scholar
Effenberg, A. O.
(2004) Synergien der Sinne für die Bewegungsregulation. [Synergies of the Senses for the Regulation of Movement.] Frankfurt am Main: Peter Lang.Google Scholar
(2005) Movement sonification: Effects on perception and action. IEEE Multimedia, 12(2), 53–59. DOI logoGoogle Scholar
Effenberg, A. O., Fehse, U., Schmitz, G., Krueger, B., & Mechling, H.
(2016) Movement Sonification: Effects on Motor Learning beyond Rhythmic Adjustements. Frontiers in Neuroscience, 10:219. DOI logoGoogle Scholar
Effenberg, A. O., Fehse, U., & Weber, A.
(2011) Movement Sonification: Audiovisual benefits on motor learning. BIO Web of Conferences, 1, 00022, 1–5.Google Scholar
Ernst, M. O., & Banks, M. S.
(2002) Humans integrate visual and haptic information in a statistically optimal fashion, Nature, 415, 429–433. DOI logoGoogle Scholar
Gaveau, V., Prablanc, C., Laurent, D., Rossetti, Y., & Priot, A-M.
(2014) Visuomotor adaptation needs a validation of prediction error by feedback error. Frontiers in Human Neuroscience, 8, 880. DOI logoGoogle Scholar
Hatada, Y., Rossetti, Y., & Miall, R. C.
(2006) Long-lasting aftereffect of a single prism adaptation: Shifts in vision and proprioception are independent. Experimental Brain Research, 173(3), 415–424. DOI logoGoogle Scholar
Hay, J. C., & Pick, H. I.
(1966) Visual and proprioceptive adaptation to optical displacement of the visual stimulus. Journal of Experimental Psychology, 71(1), 150–158. DOI logoGoogle Scholar
Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G.
(2002) Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons, Science, 297 (5582), 846–848.Google Scholar
Kornysheva, K.
(2011) Function of the Ventral Premotor Cortex in Auditory-Motor Integration of Musical Rhythm. Doctoral thesis: Westfälische Wilhelms-Universität Münster.Google Scholar
Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J., Miner, N.; et al.
(1999) The Sonification Report: Status of the Field and Research Agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display. Santa Fe, NM: ICAD.Google Scholar
Loula, F., Prasad, S, Harber, K., & Shiffrar, M.
(2005) Recognizing people from their movement. Journal of Experimental Psychology: Human Perception & Performance, 31(1), 210–220.Google Scholar
Maulucci, R. A., & Eckhouse, R. H.
(2001) Retraining reaching in chronic stroke with real-time auditory feedback. NeuroRehabilitation, 16, 171–182.Google Scholar
McGann, M.
(2014) Enacting a social ecology: Radically embodied intersubjectivity. Frontiers in Psychology, 5, 1321. DOI logoGoogle Scholar
Ong, N. T., Larssen, B. C., & Hodges, N. J.
(2012) In the absence of physical practice, observation and imagery do not result in updating of internal models for aiming. Experimental Brain Research, 218, 9–19. DOI logoGoogle Scholar
Redding, G. M., & Wallace, B.
(1988) Components of prism adaptation in terminal and concurrent exposure: Organization of the eye-hand coordination loop. Perception & Psychophysics, 44(1), 59–68. DOI logoGoogle Scholar
Robertson, J. V. G., Hoellinger, T., Lindberg, P., Bensmail, D., Hanneton, S., & Roby-Brami, A.
(2009) Effect of auditory feedback differs according to side of hemiparesis: a comparative pilot study. Journal of NeuroEngineering and Rehabilitation, 6:45. DOI logoGoogle Scholar
Rosati, G., Oscari, F., Spagnol, S., Avanzini, F., & Masiero, S.
(2012) Effect of task-related continuous auditory feedback during learning of tracking motion exercises. Journal of NeuroEngineering and Rehabilitation, 9:79. DOI logoGoogle Scholar
Rizzolatti G, Fadiga L, Gallese V., & Fogassi L.
(1996) Premotor Cortex and the Recognition of Motor Actions. Cognitive Brain Research, 3(2), 131–141. DOI logoGoogle Scholar
Scheef, L., Boecker, H., Daamen, M., Fehse, U., Landsberg, M. W., Granath, D. O., Mechling, H., & Effenberg, A. O.
(2009) Multimodal motion processing in area V5/MT: Evidence from an artificial class of audio-visual events. Brain Research, 1252, 94–104. DOI logoGoogle Scholar
Saxe, R.
(2006) Uniquely human social cognition. Current Opinion in Neurobiology, 16, 235–239. DOI logo.Google Scholar
Schmitz G., & Bock, O.
(2014) A Comparison of Sensorimotor Adaptation in the Visual and in the Auditory Modality. PLoS ONE 9(9): e107834. DOI logoGoogle Scholar
Schmitz, G., & Effenberg, A. O.
(2012) Perceptual effects of auditory information about own and other movements. Proceedings of the 18th Annual Conference on Auditory Display, Atlanta, USA, 89–94.Google Scholar
Schmitz, G., Kroeger, D., & Effenberg, A. O.
(2014) A mobile sonification system for stroke rehabilitation. The 20th International Conference on Auditory Display (ICAD2014), New York, NY, (pp.1–7). [URL]
Schmitz, G., Mohammadi, B., Hammer, A., Heldmann, M., Samii, A., Münte, T. F, & Effenberg, A. O.
(2013) Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neuroscience 14:32. DOI logoGoogle Scholar
Schmitz, G., Müller, M., Hornschuh, M., Friedrich, A., Müller, M., & Effenberg, A. O.
(2013) Verbesserung des Passspiels durch zeitsynchrone Akustik. In: Deutscher Fußball-Bund (ed.), 2. DFB-Wissenschaftskongress 2013 (pp. 217–218). Greven: Bitter und Loose GmbH.Google Scholar
Shadmehr, R., Smith, M. A., & Krakauer, J. W.
(2010) Error correction, sensory prediction and adaptation in motor control. Annual Review of Neuroscience, 33, 89–108. DOI logoGoogle Scholar
Simani, M. C., McGuire, L. M. M., & Sabes, P. N.
(2007) Visual-shift adaptation is composed of separable sensory and task-dependent effects. Journal of Neurophysiology, 98(5), 2827–2841. DOI logoGoogle Scholar
Stein, B. E., & Meredith, M. A.
(1993) The Merging of the Senses. Cambridge: MIT Press.Google Scholar
Stein, B. E., & Stanford, T. R.
(2008) Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9(4), 255–266. DOI logoGoogle Scholar
Stoffregen, T. A., & Bardy, B. G.
(2001) On specification and the senses. Behavioral and Brain Sciences, 24(2), 195–213; discussion 213–161. DOI logoGoogle Scholar
Uhlarik, J. J., & Canon, L. K.
(1971) Influence of concurrent and terminal exposure conditions on the nature of perceptual adaptation. Journal of Experimental Psychology, 91(2), 233–239. DOI logoGoogle Scholar
Vinken, P., Kroeger, D., Fehse, U., Schmitz, G., Brock, H., & Effenberg, A. O.
(2013) Auditory Coding of Human Movement Kinematics. Multisensory Research, 26 (6), 533–552. DOI logoGoogle Scholar
Cited by

Cited by 3 other publications

Allen-Collinson, Jacquelyn, Dona L. Hall & Patricia C. Jackman
2023. Intercorporeality in visually impaired running-together: Auditory attunement and somatic empathy. The Sociological Review ► pp. 003802612311634 ff. DOI logo
Effenberg, Alfred O., Tong-Hun Hwang, Shashank Ghai & Gerd Schmitz
2018. Auditory Modulation of Multisensory Representations. In Music Technology with Swing [Lecture Notes in Computer Science, 11265], ► pp. 284 ff. DOI logo
Schmitz, Gerd, Jeannine Bergmann, Alfred O. Effenberg, Carmen Krewer, Tong-Hun Hwang & Friedemann Müller
2018. Movement Sonification in Stroke Rehabilitation. Frontiers in Neurology 9 DOI logo

This list is based on CrossRef data as of 6 june 2023. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.