Cover not available
Article published In:
How the Brain Got Language: Towards a New Road Map
Edited by Michael A. Arbib
[Interaction Studies 19:1/2] 2018
► pp. 289–317

From actions to events

Communicating through language and gesture

| Brandeis University
In this paper, I argue that an important component of the language-ready brain is the ability to recognize and conceptualize events. By ‘event’, I mean any situation or activity in the world or our mental life, that we find salient enough to individuate as a thought or word. While this may sound either trivial or non-unique to humans, I hope to show that abstracting away events and their participants from the embodied flow of experience is a characteristic unique to humans. This ability is enabled, I will argue, by two critical competencies that act as scaffolds for language-ready thought in the prehuman brain. The first, as argued by Arbib (2006, 2012, 2016) and others, is a sophisticated system of gesture production and understanding in prehumans, which provided a template for schema-like sequencing and slot-filling of information units. The second involves the integration of multiple modalities of expression in the communicative act, in particular, the alignment of co-gestural speech and co-speech gesture. With such computational facilities, action-based gestures can be abstracted away from their associated objects and become full event representations. This view supports the MSH argument for the emergence of more complex linguistic expressions from initially holophrastic units. In particular, actions can be thought of as protoverbs, which through this process are abstracted to full event descriptions, i.e., verbs.
Keywords: gesture, language, action, affordances, events
Article outline
  • 1.Introduction
  • 2.Communication with a common ground
  • 3.The structure of actions and events
  • 4.The interpretation of gesture
  • 5.Gesture sequences
  • 6.Multimodal communication: Gesture-speech ensembles
    • 6.1Co-gestural demonstratives
    • 6.2More expressive co-gestural ensembles
  • 7.Meaning in the absence of common groun
  • 8.Toward a new road map
  • Acknowledgements
  • References
Published online: 17 September 2018
DOI logo
https://doi.org/10.1075/is.17045.pus
References (71)
References
Aboitiz, F. (2013) “How did vocal behavior ‘take over’ the gestural communication system?.” Language and Cognition 5, no. 2–3: 167–176. DOI logoGoogle Scholar
Abner, N., Cooperrider, K., & Goldin-Meadow, S. (2015). Gesture for linguists: A handy primer. Language and linguistics compass, 9(11), 437–451. DOI logoGoogle Scholar
Arbib, M. A. (2002) The Mirror System, Imitation, and the Evolution of Language. Imitation in animals and artifacts, 2291.Google Scholar
ed. (2006) Action to language via the mirror neuron system. Cambridge University Press. DOI logoGoogle Scholar
, (2008). Holophrasis and the protolanguage spectrum. Interaction Studies, 9(1), pp.154–168. DOI logoGoogle Scholar
(2008). From grasp to language: Embodied concepts and the challenge of abstraction. Journal of Physiology-Paris, 102(1–3), 4–20. DOI logoGoogle Scholar
(2012). How the Brain Got Language: The Mirror System Hypothesis. New York & Oxford: Oxford University Press. DOI logoGoogle Scholar
(2013). Precis of how the brain got language: the mirror system hypothesis. Language and Cognition, 5(2–3), 107–131. DOI logoGoogle Scholar
(2016). Towards a computational comparative neuroprimatology: framing the language-ready brain. Physics of life reviews, 161, 1–54. DOI logoGoogle Scholar
Arbib, M. A., Liebal, K. and Pika, S., (2008). “Primate vocalization, gesture, and the evolution of human language.” Current anthropology 491. 61, 1053–1076. DOI logoGoogle Scholar
Arbib, M. A. & Rizzolatti, G. (1997). Neural expectations: a possible evolutionary path from manual skills to language. Communication and Cognition, 291, 393–424.Google Scholar
Armstrong, D. F., Stokoe, W. C. and Wilcox, S. E. 1995. Gesture and the nature of language. Cambridge University Press. DOI logoGoogle Scholar
Armstrong, D. F., & Wilcox, S. (2007). The gestural origin of language. Oxford University Press. DOI logoGoogle Scholar
Asher, N. (1998). Common ground, corrections and coordination. Journal of Semantics.Google Scholar
Asher, N. and Lascarides, A., (2003). Logics of conversation. Cambridge University Press.Google Scholar
Asher, N. and A. Lascarides. (2008). Commitments, beliefs and intentions in dialogue. Proceedings of Londial, 35–42.Google Scholar
Austin, J. L. (1962). How to do things with words. London: Oxford University PressGoogle Scholar
Bach, K. (2003). Speech acts and pragmatics. Blackwell. Guide to the philosophy of language, 147–167.Google Scholar
Bickerton, D. (1990) Language and Species, University of Chicago Press. Bickerton, Derek (2009). Adam’s Tongue. Hill and Wang. DOI logoGoogle Scholar
2009. Adam’s tongue: how humans made language, how language made humans. Macmillan.Google Scholar
Bohn, M., Call, J. and Tomasello, M., (2016). Comprehension of iconic gestures by chimpanzees and human children. Journal of experimental child psychology, 1421, pp.1–17. DOI logoGoogle Scholar
Cangelosi, A. (2010). Grounding language in action and perception: from cognitive agents to humanoid robots. Physics of life reviews 7(2), 139–151. DOI logoGoogle Scholar
Clark, H. H. (1996). Using language. Cambridge University Press. DOI logoGoogle Scholar
Clark, H. H., S. E. Brennan, et al. (1991). Grounding in communication. Perspectives on socially shared cognition 13(1991), 127–149. DOI logoGoogle Scholar
Clark, H. H., Schreuder, R., & Buttrick, S. (1983). Common ground at the understanding of demonstrative reference. Journal of verbal learning and verbal behavior, 22(2), 245–258. DOI logoGoogle Scholar
Cooper, R. and Ginzburg, J. (2015). “Type theory with records for natural language semantics”. Handbook of Contemporary Semantic Theory, pp.375–407. DOI logoGoogle Scholar
Corballis, Michael C., (2003) From hand to mouth: The origins of language. Princeton University Press.Google Scholar
Deacon, Terrence W. (1998). The symbolic species: The co-evolution of language and the brain. WW Norton & Company.Google Scholar
Diessel, H. (2013) “Where does language come from? Some reflections on the role of deictic gesture and demonstratives in the evolution of language.” Language and Cognition 5.2–3: 239–249. DOI logoGoogle Scholar
Engle, R.A. and Clark, H.H. 1995. Using composites of speech, gestures, diagrams and demonstrations in explanations of mechanical devices. In American Association for Applied Linguistics Conference, Long Beach, CA.Google Scholar
Fogassi, L., Coude, G. and Ferrari, P. F., (2013). The extended features of mirror neurons and the voluntary control of vocalization in the pathway to language. Language and Cognition, 5(2–3), pp.145–155. DOI logoGoogle Scholar
Fricke, E. (2013). Towards a unified grammar of gesture and speech: A multimodal approach. Body-language-communication. An international handbook on multimodality in human interaction, 733–754.Google Scholar
Gillespie-Lynch, K., Greenfield, P. M., Lyn, H., & Savage-Rumbaugh, S. (2014). Gestural and symbolic development among apes and humans: support for a multimodal theory of language evolution. Frontiers in psychology, 51. DOI logoGoogle Scholar
Gibson, J. J. (1979). The ecological approach to visual perception: classic edition. Psychology Press.Google Scholar
Gilbert, M. (1992). On social facts. Princeton University Press.Google Scholar
Givòn, T. (1995). Functionalism and grammar. John Benjamins Publishing. DOI logoGoogle Scholar
(1998). On the co-evolution of language, mind, and brain. Evolution of Communication, 2(1), 45–116. DOI logoGoogle Scholar
Glenberg, A. M. & Gallese, V. (2012). Action-based language: a theory of language acquisition, comprehension, and production. Cortex, 48(7), 905–22. DOI logoGoogle Scholar
Goldin-Meadow, S. and Alibali, M.W. 2013. Gesture’s role in speaking, learning, and creating language. Annual review of psychology, 641, pp. 257-283. DOI logoGoogle Scholar
Grice, H. P. (1981). Presupposition and conversational implicature. In C. Peter (ed.), Radical pragmatics. New York: Academic Press, pp. 183–198. Reprinted in Grice, H. P. (1989) Studies in the way of words. Cambridge: Harvard University Press, pp. 269–282.Google Scholar
Hsiao, K. -Y., S. Tellex, S. Vosoughi, R. Kubat, and D. Roy. (2008). Object schemas for grounding language in a responsive robot. Connection Science 20(4), 253–276. DOI logoGoogle Scholar
Hauser, Marc D., Noam Chomsky, and W. Tecumseh Fitch. (2002) “The faculty of language: what is it, who has it, and how did it evolve?.” Science 298.5598: 1569–1579. DOI logoGoogle Scholar
Hockett, C. F., (1960). The origin of speech. Scientific American, 203(3), pp.88–97. DOI logoGoogle Scholar
Iverson, J. M., Capirci, O. and Caselli, M. C., (1994). From communication to language in two modalities. Cognitive development, 9(1), pp.23–43. DOI logoGoogle Scholar
Iverson, J. M., & Goldin-Meadow, S. (2005). Gesture paves the way for language development. Psychological science, 16(5), 367–371. DOI logoGoogle Scholar
Kamp, H. and Reyle, U. 1993. From discourse to logic: Introduction to model-theoretic semantics of natural language, formal logic and discourse representation. Studies in Linguistics and Philosophy. Kluwer.Google Scholar
Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In M. R. Key (Ed.), The relationship of verbal and nonverbal communication, 207–228. The Hague: Mouton.Google Scholar
(1995). Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of pragmatics, 23(3), pp. 247–279.
Lascarides, A. and M. Stone. (2006). Formal semantics for iconic gesture. In Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue (BRANDIAL), pp. 64–71.
. (2009). A formal semantic analysis of gesture. Journal of Semantics 26(4), 393–449. DOI logoGoogle Scholar
Pika, S. and Mitani, J. C., (2009). The directed scratch: evidence for a referential gesture in chimpanzees. The prehistory of language, 11, pp.166–181. DOI logoGoogle Scholar
Pustejovsky, J. (1995). The Generative Lexicon. MIT Press, Cambridge, MA.Google Scholar
(2013). Where things happen: On the semantics of event localization. In Proceedings of isa-9: International workshop on semantic annotation.Google Scholar
Pustejovsky, J. and N. Krishnaswamy. (2014). Generating simulations of motion events from verbal descriptions. In Lexical and Computational Semantics (*SEM 2014). ACL. DOI logoGoogle Scholar
Pustejovsky, J. and J. L. Moszkowicz. (2011). The qualitative spatial dynamics of motion in language. Spatial Cognition & Computation 11(1), 15–44. DOI logoGoogle Scholar
Rizzolatti, G., and Arbib, M. (1998). Language within our grasp. Trends in Neurosciences, 211, 188–194. DOI logoGoogle Scholar
Schlenker, P. (2015). Visible meaning: Sign language and the foundations of semantics. Manuscript, Institut Jean-Nicod and New York University.Google Scholar
Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Vol. 6261). Cambridge university press. DOI logoGoogle Scholar
Stalnaker, R. (2002). Common ground. Linguistics and philosophy 25(5), 701–721. DOI logoGoogle Scholar
Steels, L. (1999). The talking heads experiment: Vol. I. Words and meaning (Special preedition). Brussels: Vrije Universiteit Brussel.Google Scholar
(2011). Modeling the cultural evolution of language. Physics of Life Reviews, 8(4), 339–356. DOI logoGoogle Scholar
(ed), (2012). Experiments in cultural language evolution. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Stout, Dietrich, & Hecht, Erin E. (2017). Evolutionary neuroscience of cumulative culture. Proceedings of the National Academy of Sciences, 114(30), 7861–7868. DOI logoGoogle Scholar
Strickland, B., Geraci, C., Chemla, E., Schlenker, P., Kelepir, M., and Pfau, R. (2015). Event representations constrain the structure of language: Sign language as a window into universally accessible linguistic biases. Proceedings of the National Academy of Sciences. doi: DOI logoGoogle Scholar
Tomasello, M. and M. Carpenter. (2007). Shared intentionality. Developmental science 10(1), 121?125. DOI logoGoogle Scholar
Vendler, Z. (1967). Linguistics and Philosophy. Ithaca: Cornell University Press. DOI logoGoogle Scholar
Volterra, V., Caselli, M. C., Capirci, O., & Pizzuto, E. (2005). Gesture and the emergence and development of language. Beyond nature-nurture: Essays in honor of Elizabeth Bates, 3–40.Google Scholar
von Uexkll, J. (1957). A stroll through the worlds of animals and men: A picture book of invisible worlds, in Schiller, C. H. (ed), Instinctive Behavior: The Development of a Modern Concept. New York: International Universities Press, 5–80.Google Scholar
Wang, I., M. Ben Fraj, P. Narayana, D. Patil, G. Mulay, R. Bangar, R. Beveridge, B. Draper, and J. Ruiz. (2017). Eggnog: A continuous, multimodal data set of naturally occurring gestures with ground truth labels. In Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition.
Whitehead, A. N. (1919), An Enquiry concerning the Principles of Natural Knowledge, Cambridge: Cambridge University Press.Google Scholar
Ziemke, T. & Sharkey, N. E. (2001). A stroll through the worlds of robots and animals: Applying Jakob von Uexkll’s theory of meaning to adaptive robots and artificial life. Semiotica, 2001(134), 701–746. DOI logoGoogle Scholar
Cited by (7)

Cited by seven other publications

Van der sluis, Ielka, Gabriela Matoušková, Hannah Niemeier, Sophia Popp & Josephine Carstens
2022. The clarity and correctness of visualized thrust actions: a description and insights from users and experts. Visual Communication DOI logo
Pustejovsky, James & Nikhil Krishnaswamy
2021. The Role of Embodiment and Simulation in Evaluating HCI: Theory and Framework. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior [Lecture Notes in Computer Science, 12777], ► pp. 288 ff. DOI logo
Pustejovsky, James & Nikhil Krishnaswamy
2021. Embodied Human Computer Interaction. KI - Künstliche Intelligenz 35:3-4 ► pp. 307 ff. DOI logo
Arbib, Michael A., Francisco Aboitiz, Judith M. Burkart, Michael C. Corballis, Gino Coudé, Erin Hecht, Katja Liebal, Masako Myowa-Yamakoshi, James Pustejovsky, Shelby S. Putt, Federico Rossano, Anne E. Russon, P. Thomas Schoenemann, Uwe Seifert, Katerina Semendeferi, Chris Sinha, Dietrich Stout, Virginia Volterra, Sławomir Wacewicz & Benjamin Wilson
2020. The comparative neuroprimatology 2018 (CNP-2018) road map for research on How the Brain Got Language. In How the Brain Got Language – Towards a New Road Map [Benjamins Current Topics, 112], ► pp. 370 ff. DOI logo
Arbib, Michael A., Francisco Aboitiz, Judith M. Burkart, Michael Corballis, Gino Coudé, Erin Hecht, Katja Liebal, Masako Myowa-Yamakoshi, James Pustejovsky, Shelby Putt, Federico Rossano, Anne E. Russon, P. Thomas Schoenemann, Uwe Seifert, Katerina Semendeferi, Chris Sinha, Dietrich Stout, Virginia Volterra, Sławomir Wacewicz & Benjamin Wilson
2018. The comparative neuroprimatology 2018 (CNP-2018) road map for research on How the Brain Got Language . Interaction Studies. Social Behaviour and Communication in Biological and Artificial Systems 19:1-2 ► pp. 370 ff. DOI logo
Krishnaswamy, Nikhil & James Pustejovsky
2018. Deictic Adaptation in a Virtual Environment. In Spatial Cognition XI [Lecture Notes in Computer Science, 11034], ► pp. 180 ff. DOI logo
Krishnaswamy, Nikhil & James Pustejovsky
2022. Affordance embeddings for situated language understanding. Frontiers in Artificial Intelligence 5 DOI logo

This list is based on CrossRef data as of 18 october 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.