Cover not available
Article published In:
Journal of Second Language Pronunciation
Vol. 4:2 (2018) ► pp.232–259

Visualizing the acoustic and gestural beats of emphasis in multimodal discourse

Theoretical and pedagogical implications

| Michigan State University

Perceivers’ attention is entrained to the rhythm of a speaker’s gestural and acoustic beats. When different rhythms (polyrhythms) occur across the visual and auditory modalities of speech simultaneously, attention may be heightened, enhancing memorability of the sequence. In this three-stage study, Stage 1 analyzed videorecordings of native English-speaking instructors, focusing on frame-by-frame analysis of time-aligned annotations from Praat and Anvil (video annotation tool) of polyrhythmic sequences. Stage 2 explored the perceivers’ perspective on the sequences’ discourse role. Stage 3 analyzed 10 international teaching assistants’ gestures, and implemented a multistep technology-assisted program to enhance verbal and nonverbal communication skills. Findings demonstrated (a) a dynamic temporal gesture-speech relationship involving perturbations of beat intervals surrounding pitch-accented vowels, (b) the sequences’ important role as highlighters of information, and (c) improvement of ITA confidence, teaching effectiveness, and ability to communicate important points. Findings support the joint production of gesture and prosodically prominent features.

Keywords: gesture, prosody, beats, acoustic, international teaching assistants, entrainment, rhythm, intonation, Praat, Anvil Abstract: Perceivers’ attention is entrained
Article outline
  • 1.Introduction
    • 1.1Gesture and prosody
    • 1.2Theoretical perspectives
    • 1.3Prosody in Korean
    • 1.4L2 nonverbal communication
    • 1.5ITA Training
  • 2.Current study
    • 2.1Stage 1
      • 2.1.1Method
        • 2.1.1.1Participants
        • 2.1.1.2Recording
        • 2.1.1.3Selection of G-phrases for analysis
        • 2.1.1.4Speech annotations
        • 2.1.1.5Video annotations
      • 2.1.2Results
        • 2.1.2.1Example 1
        • 2.1.2.2Example 2
    • 2.2Stage 2
      • 2.2.1Participants
      • 2.2.2Materials and procedure
      • 2.2.3Results
    • 2.3Stage 3
      • 2.3.1Participants
      • 2.3.2Presentation skills program: Materials and procedure
        • 2.3.2.1Step 1: Raising visual awareness
        • 2.3.2.2Step 2: Raising gesture-prosody awareness
        • 2.3.2.3Step 3: Review and practice of speech features
        • 2.3.2.4Step 4: Review and practice of nonverbal communication
        • 2.3.2.5Step 5: Preparation for future classes
        • 2.3.2.6Follow-up
      • 2.3.3Results and discussion
        • 2.3.3.1ITA Example 1
        • 2.3.3.2ITA Example 2
        • 2.3.3.3Student survey on communication skills
        • 2.3.3.4ITA survey
  • 3.General discussion
  • Notes
  • References
Published online: 1 February 2019
DOI logo
https://doi.org/10.1075/jslp.17006.har
References

References

Allen, G. D.
(1972) The location of rhythmic stress beats in English speech. Parts I & II. Language and Speech, 151, 72–100, 179–195. DOI logoGoogle Scholar
Bailey, K. M.
(1982) Teaching in a second language: The communicative competence of non-native speaking assistants (Unpublished doctoral dissertation). University of California, Los Angeles.Google Scholar
Beckman, M. E., Hirschberg, J., & Shattuck-Hufnagel, S.
(2005) The original ToBI system and the evolution of the ToBI framework (pp. 9–54). In S. Jun (Ed.), Prosodic typology: The phonology of intonation and phrasing. Oxford: Oxford University Press. DOI logoGoogle Scholar
Beckman, M. E., & Pierrehumbert, J.
(1986) Intonational structure in Japanese and English. Phonology Yearbook 31, 255–309. DOI logoGoogle Scholar
Bergeson, T. R., Pisoni, D. B., & Davis, R. A. O.
(2003) A longitudinal study of audiovisual speech perception by children with hearing loss who have cochlear implants. The Volta Review, 1031, 347–370.Google Scholar
Biau, E., & Soto-Faraco, S.
(2013) Beat gestures modulate auditory integration in speech perception. Brain and Language, 1241, 143–152. DOI logoGoogle Scholar
Birdwhistell, R.
(1970) Kinesics and context: Essay on body motion communication. Philadelphia, PA: University of Pennsylvania Press.Google Scholar
Boersma, P., & Weenink, D.
(2014) Praat: Doing phonetics by computer [Computer program]. Retrieved from [URL]
Bolinger, D.
(1986) Intonation and its parts: Melody in spoken English. Stanford, CA: Stanford University Press.Google Scholar
Bull, P., & Connelly, G.
(1985) Body movement and emphasis in speech. Journal of Nonverbal Behavior, 91, 169–187. DOI logoGoogle Scholar
Chafe, W.
(1994) Discourse, consciousness, and time: The flow and displacement of conscious experience in speaking and writing. Chicago, IL: The University of Chicago Press.Google Scholar
Clark, J. M., & Paivio, A.
(1991) Dual coding theory and education. Educational Psychology Review, 31, 149–210. DOI logoGoogle Scholar
Clayton, M., Sager, R., & Will, U.
(2005) In time with the music: The concept of entrainment and its significance for ethnomusicology. ESEM (European Seminar in Ethnomusicology) CounterPoint, 11, 1–82.Google Scholar
Condon, W.
(1976) An analysis of behavioral organization. Sign Language Studies, 51, 285–318. DOI logoGoogle Scholar
de Ruiter, J. P.
(2000) The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284–311). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Dimitrova, D., Chu, M., Wang, L., Özyürek, A., & Hagoort, P.
(2016) Beat that word: How listeners integrate beat gesture and focus on multimodal speech discourse. Journal of Cognitive Neuroscience, 281, 1255–1269. DOI logoGoogle Scholar
Flecha-García, M. L.
(2010) Eyebrow raises in dialogue and their relation to discourse structure, utterance function and pitch accents in English. Speech Communication, 521, 542–554. DOI logoGoogle Scholar
Flowerdew, J., & Tauroza, S.
(1995) The effect of discourse markers on second language lecture comprehension. Studies in Second Language Acquisition, 171, 435–458. DOI logoGoogle Scholar
Fraisse, P.
(1982) Rhythm and tempo. In D. Deutsch (Ed.), The psychology of music (pp. 149–180). New York, NY: Academic Press. DOI logoGoogle Scholar
Goldin-Meadow, S., & Alibali, M. W.
(2013) Gesture’s role in speaking, learning, and creating language. Annual Review of Psychology, 641, 257–283. DOI logoGoogle Scholar
Gorsuch, G. J.
(2003) The educational cultures of international teaching assistants and U.S. universities. TESL-EJ, 71. [URL]
Gregersen, T., Olivares-Cuhat, G., & Storm, J.
(2009) An examination of L1 and L2 gesture use: What role does proficiency play? The Modern Language Journal, 931, 195–208. DOI logoGoogle Scholar
Gullberg, M.
(2011) Multilingual multimodality: Communicative difficulties and their solutions in second language use. In J. Streeck, C. Goodwin, & C. LeBaron (Eds.), Embodied interaction: Language and body in the material world (pp. 137–151). Cambridge: Cambridge University Press.Google Scholar
Hardison, D. M.
(2003) Acquisition of second-language speech: Effects of visual cues, context and talker variability. Applied Psycholinguistics, 241, 495–522. DOI logoGoogle Scholar
(2004) Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 81, 34–52. [URL]
(2005) Contextualized computer-based L2 prosody training: Evaluating the effects of discourse context and video input. CALICO Journal, 221, 175–190. DOI logoGoogle Scholar
(2012) Second-language speech perception: A cross-disciplinary perspective on challenges and accomplishments. In S. Gass & A. Mackey (Eds.), The Routledge handbook of second language acquisition (pp. 349–363). London: Routledge.Google Scholar
Hincks, R., & Edlund, J.
(2009) Promoting increased pitch variation in oral presentations with transient visual feedback. Language Learning & Technology, 131, 32–50. [URL]
Inceoglu, S.
(2016) Effects of perceptual training on second language vowel perception and production. Applied Psycholinguistics, 371, 1175–1199. DOI logoGoogle Scholar
Jenkins, S., & Parra, I.
(2003) Multiple layers of meaning in an oral proficiency test: The complementary roles of nonverbal, paralinguistic, and verbal behaviors in assessment decisions. The Modern Language Journal, 871, 90–107. DOI logoGoogle Scholar
Jones, M. R.
(1986) Attentional rhythmicity in human perception. In J. R. Evans & M. Clynes (Eds.), Rhythm in psychological, linguistic, and music processes (pp. 13–40). Springfield, IL: Charles C. Thomas.Google Scholar
Jun, S. -A.
(2005) Korean intonational phonology and prosodic transcription. In S. -A. Jun (Ed.), Prosodic typology: The phonology of intonation and phrasing (pp. 201–229). Oxford: Oxford University Press. DOI logoGoogle Scholar
Kellerman, E., & Van Hoof, A. -M.
(2003) Manual accents. International Review of Applied Linguistics, 411, 251–269. DOI logoGoogle Scholar
Kendon, A.
(1972) Some relationships between body motion and speech: An analysis of an example. In A. Siegman & B. Pope (Eds.), Studies in dyadic communication (pp. 177–210). New York, NY: Pergamon Press. DOI logoGoogle Scholar
Kipp, M.
(2001)  Anvil – A generic annotation tool for multimodal dialogue. In Proceedings of the 7th European Conference on Speech Communication and Technology (pp. 1367–1370). Aalborg, Denmark: Eurospeech.Google Scholar
Krahmer, E., & Swerts, M.
(2007) The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 571, 396–414. DOI logoGoogle Scholar
Large, E. W., & Jones, M. R.
(1999) The dynamics of attending: How people track time-varying events. Psychological Review, 1061, 119–159. DOI logoGoogle Scholar
Leonard, T., & Cummins, F.
(2011) The temporal relation between beat gestures and speech. Language and Cognitive Processes, 261, 1457–1471. DOI logoGoogle Scholar
Levelt, W. J. M.
(1989) Speaking: From intention to articulation. Cambridge, MA: The MIT Press.Google Scholar
Loehr, D.
(2007) Aspects of rhythm in gesture and speech. Gesture, 71, 179–214. DOI logoGoogle Scholar
London, J.
(2012) Hearing in time: Psychological aspects of musical meter (2nd ed.). Oxford: Oxford University Press. DOI logoGoogle Scholar
McCafferty, S. G.
(2002) Gesture and creating zones of proximal development for second language learning. The Modern Language Journal, 861, 192–203. DOI logoGoogle Scholar
McClave, E.
(1994) Gestural beats: The rhythm hypothesis. Journal of Psycholinguistic Research, 231, 45–66. DOI logoGoogle Scholar
McCroskey, J. C., Richmond, V. P., Sallinen, A., Fayer, J. M. & Barraclough, R. A.
(1995) A cross-cultural and multi-behavioral analysis of the relationship between nonverbal immediacy and teacher evaluation. Communication Education, 441, 281–291. DOI logoGoogle Scholar
Mehrabian, A.
(1969) Methods & designs: Some referents and measures of nonverbal behavior. Behavior Research Methods and Instrumentation, 11, 203–207. DOI logoGoogle Scholar
McNeill, D.
(1992) Hand and mind: What gestures reveal about thought. Chicago, IL: University of Chicago Press.Google Scholar
(2006) Gesture and communication. In the Encyclopedia of language and linguistics (2nd ed.) (pp. 58–66). Amsterdam: Elsevier. DOI logoGoogle Scholar
McNeill, D., Levy, E. T., & Duncan, S. D.
(2015) Gesture in discourse. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (2nd ed.) (pp. 262–289). Hoboken, NJ: Wiley & Sons.Google Scholar
Meltzoff, A. N., & Kuhl, P. K.
(1994) Faces and speech: Intermodal processing of biologically relevant signals in infants and adults. In D. J. Lewkowitz & R. Lickliter (Eds.), The development of intersensory perception: Comparative perspectives (pp. 335–369). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Merker, B. H., Madison, G. S., & Eckerdal, P.
(2009) On the role and origin of isochrony in human rhythmic entrainment. Cortex, 451, 4–17. DOI logoGoogle Scholar
Mori, J., & Hayashi, M.
(2006) The achievement of intersubjectivity through embodied completions: A study of interactions between first and second language speakers. Applied Linguistics, 271, 195–219. DOI logoGoogle Scholar
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., Vatikiotis-Bateson, E.
(2004) Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 151, 133–137. DOI logoGoogle Scholar
Munhall, K. G., & Tohkura, Y.
(1998) Audiovisual gating and the time course of speech perception. Journal of the Acoustical Society of America, 1041, 530–539. DOI logoGoogle Scholar
Neu, J.
(1990) Assessing the role of nonverbal communication in the acquisition of communicative competence in L2. In R. C. Scarcella, E. S. Andersen, & S. D. Krashen (Eds.), Developing communicative competence in a second language (pp. 121–138). Boston, MA: Heinle and Heinle.Google Scholar
Port, R. F.
(2003) Meter and speech. Journal of Phonetics, 311, 599–611. DOI logoGoogle Scholar
Rosenblum, L.
(2010) See what I’m saying: The extraordinary powers of our five senses. New York, NY: W.W. Norton & Co.Google Scholar
Rusiewicz, H. L., Shaiman, S., Iverson, J. M., & Szuminsky, N.
(2013) Effects of prosody and position on the timing of deictic gestures. Journal of Speech, Language, and Hearing Research, 561, 458–470. DOI logoGoogle Scholar
Skipper, J. I., van Wassenhove, V., Nusbaum, H. W., & Small, S. L.
(2007) Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex, 171, 2387–2399. DOI logoGoogle Scholar
Sueyoshi, A., & Hardison, D. M.
(2005) The role of gestures and facial cues in second-language listening comprehension. Language Learning, 551, 661–699. DOI logoGoogle Scholar
Summerfield, Q.
(1979) Use of visual information for phonetic perception. Phonetica, 361, 314–331. DOI logoGoogle Scholar
Tajima, K., & Port, R. F.
(2003) Speech rhythm in English and Japanese. In J. Local, R. Ogden, & R. Temple (Eds.), Phonetic interpretation: Papers in laboratory phonology VI (pp. 317–334). Cambridge: Cambridge University Press.Google Scholar
Treffner, P., Peter, M., & Kleidon, M.
(2008) Gestures and phases: The dynamics of speech-hand communication. Ecological Psychology, 201, 32–64. DOI logoGoogle Scholar
Tuite, K.
(1993) The production of gesture. Semiotica, 931, 83–105. DOI logoGoogle Scholar
Volterra, V., & Erting, C. J.
(Eds.) (2002) From gesture to language in hearing and deaf children. Washington, DC: Gallaudet University Press.Google Scholar
Wennerstrom, A.
(2001) The music of everyday speech: Prosody and discourse analysis. Oxford: Oxford University Press.Google Scholar
Cited by

Cited by 10 other publications

Bobkina, Jelena, Elena Domínguez Romero & María José Gómez Ortiz
2023. Kinesic communication in traditional and digital contexts: An exploratory study of ESP undergraduate students. System 115 ► pp. 103034 ff. DOI logo
Hardison, Debra M & Martha C Pennington
2021. Multimodal Second-Language Communication: Research Findings and Pedagogical Implications. RELC Journal 52:1 ► pp. 62 ff. DOI logo
Hardison, Debra M.
2019. Technology-Based Communication Success for Second-Language Learners. In Encyclopedia of Educational Innovation, ► pp. 1 ff. DOI logo
Hardison, Debra M.
2020. Technology-Based Communication Success for Second-Language Learners. In Encyclopedia of Educational Innovation, ► pp. 1 ff. DOI logo
Kim, Yoo Lae, Chen Liu, Pavel Trofimovich & Kim McDonough
2024. Is nonverbal behavior during conversation related to perceived fluency?. TESOL Journal DOI logo
LaSCOTTE, DARREN & ELAINE TARONE
2019. Heteroglossia and Constructed Dialogue in SLA. The Modern Language Journal 103:S1 ► pp. 95 ff. DOI logo
Levis, John M.
2019. Teaching-oriented research. Journal of Second Language Pronunciation 5:1 ► pp. 1 ff. DOI logo
Pennington, Martha C.
2019. ‘Top-Down’ Pronunciation Teaching Revisited. RELC Journal 50:3 ► pp. 371 ff. DOI logo
Tarone, Elaine & Colleen Meyers
2018. Chapter 9. The Mirroring Project. In Speaking in a Second Language [AILA Applied Linguistics Series, 17], ► pp. 197 ff. DOI logo
Tsunemoto, Aki, Rachael Lindberg, Pavel Trofimovich & Kim McDonough
2022. VISUAL CUES AND RATER PERCEPTIONS OF SECOND LANGUAGE COMPREHENSIBILITY, ACCENTEDNESS, AND FLUENCY. Studies in Second Language Acquisition 44:3 ► pp. 659 ff. DOI logo

This list is based on CrossRef data as of 2 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.