From actions to events
Communicating through language and gesture
In this paper, I argue that an important component of the language-ready brain is the ability to recognize and conceptualize
events. By ‘event’, I mean any situation or activity in the world or our mental life, that we find salient enough to individuate
as a thought or word. While this may sound either trivial or non-unique to humans, I hope to show that abstracting away events and
their participants from the embodied flow of experience is a characteristic unique to humans. This ability is enabled, I will
argue, by two critical competencies that act as scaffolds for language-ready thought in the prehuman brain. The first, as argued
by Arbib (2006, 2012, 2016) and others, is a sophisticated system of gesture production and understanding in prehumans, which
provided a template for schema-like sequencing and slot-filling of information units. The second involves the integration of
multiple modalities of expression in the communicative act, in particular, the alignment of co-gestural speech and co-speech
gesture. With such computational facilities, action-based gestures can be abstracted away from their associated objects and become
full event representations. This view supports the MSH argument for the emergence of more complex linguistic expressions from
initially holophrastic units. In particular, actions can be thought of as protoverbs, which through this process are abstracted to
full event descriptions, i.e., verbs.
Article outline
- 1.Introduction
- 2.Communication with a common ground
- 3.The structure of actions and events
- 4.The interpretation of gesture
- 5.Gesture sequences
- 6.Multimodal communication: Gesture-speech ensembles
- 6.1Co-gestural demonstratives
- 6.2More expressive co-gestural ensembles
- 7.Meaning in the absence of common groun
- 8.Toward a new road map
-
Acknowledgements
-
References
References (71)
References
Aboitiz, F. (2013) “How did vocal behavior ‘take over’ the gestural communication system?.” Language and Cognition 5, no. 2–3: 167–176. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Abner, N., Cooperrider, K., & Goldin-Meadow, S. (2015). Gesture for linguists: A handy primer. Language and linguistics compass, 9(11), 437–451. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. (2002) The Mirror System, Imitation, and the Evolution of Language. Imitation in animals and artifacts, 229.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. ed. (2006) Action to language via the mirror neuron system. Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. (2008). From grasp to language: Embodied concepts and the challenge of abstraction. Journal of Physiology-Paris, 102(1–3), 4–20. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. (2012). How the Brain Got Language: The Mirror System Hypothesis. New York & Oxford: Oxford University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. (2013). Precis of how the brain got language: the mirror system hypothesis. Language and Cognition, 5(2–3), 107–131. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. (2016). Towards a computational comparative neuroprimatology: framing the language-ready brain. Physics of life reviews, 16, 1–54. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A., Liebal, K. and Pika, S., (2008). “Primate vocalization, gesture, and the evolution of human language.” Current anthropology 49. 6, 1053–1076. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Arbib, M. A. & Rizzolatti, G. (1997). Neural expectations: a possible evolutionary path from manual skills to language. Communication and Cognition, 29, 393–424.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Armstrong, D. F., Stokoe, W. C. and Wilcox, S. E. 1995. Gesture and the nature of language. Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Armstrong, D. F., & Wilcox, S. (2007). The gestural origin of language. Oxford University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Asher, N. (1998). Common ground, corrections and coordination. Journal of Semantics.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Asher, N. and Lascarides, A., (2003). Logics of conversation. Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Asher, N. and A. Lascarides. (2008). Commitments, beliefs and intentions in dialogue. Proceedings of Londial, 35–42.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Austin, J. L. (1962). How to do things with words. London: Oxford University Press![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bach, K. (2003). Speech acts and pragmatics. Blackwell. Guide to the philosophy of language, 147–167.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bickerton, D. (1990) Language and Species, University of Chicago Press. Bickerton, Derek (2009). Adam’s Tongue. Hill and Wang. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bickerton, D. 2009. Adam’s tongue: how humans made language, how language made humans. Macmillan.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bohn, M., Call, J. and Tomasello, M., (2016). Comprehension of iconic gestures by chimpanzees and human children. Journal of experimental child psychology, 142, pp.1–17. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cangelosi, A. (2010). Grounding language in action and perception: from cognitive agents to humanoid robots. Physics of life reviews 7(2), 139–151. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Clark, H. H. (1996). Using language. Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Clark, H. H., S. E. Brennan, et al. (1991). Grounding in communication. Perspectives on socially shared cognition 13(1991), 127–149. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Clark, H. H., Schreuder, R., & Buttrick, S. (1983). Common ground at the understanding of demonstrative reference. Journal of verbal learning and verbal behavior, 22(2), 245–258. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cooper, R. and Ginzburg, J. (2015). “Type theory with records for natural language semantics”. Handbook of Contemporary Semantic Theory, pp.375–407. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Corballis, Michael C., (2003) From hand to mouth: The origins of language. Princeton University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Deacon, Terrence W. (1998). The symbolic species: The co-evolution of language and the brain. WW Norton & Company.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Diessel, H. (2013) “Where does language come from? Some reflections on the role of deictic gesture and demonstratives in the evolution
of language.” Language and Cognition 5.2–3: 239–249. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Engle, R.A. and Clark, H.H. 1995. Using composites of speech, gestures, diagrams and demonstrations in explanations of mechanical
devices. In American Association for Applied Linguistics Conference, Long Beach, CA.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fogassi, L., Coude, G. and Ferrari, P. F., (2013). The extended features of mirror neurons and the voluntary control of vocalization in the pathway to
language. Language and Cognition, 5(2–3), pp.145–155. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fricke, E. (2013). Towards a unified grammar of gesture and speech: A multimodal approach. Body-language-communication. An international handbook on multimodality in human interaction, 733–754.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Gillespie-Lynch, K., Greenfield, P. M., Lyn, H., & Savage-Rumbaugh, S. (2014). Gestural and symbolic development among apes and humans: support for a multimodal theory of language
evolution. Frontiers in psychology, 5. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Gibson, J. J. (1979). The ecological approach to visual perception: classic edition. Psychology Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Gilbert, M. (1992). On social facts. Princeton University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Glenberg, A. M. & Gallese, V. (2012). Action-based language: a theory of language acquisition, comprehension, and production. Cortex, 48(7), 905–22. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goldin-Meadow, S. and Alibali, M.W. 2013. Gesture’s role in speaking, learning, and creating language. Annual review of psychology, 64, pp. 257-283. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Grice, H. P. (1981). Presupposition and conversational implicature. In C. Peter (ed.), Radical pragmatics. New York: Academic Press, pp. 183–198. Reprinted in Grice, H. P. (1989) Studies in the way of words. Cambridge: Harvard University Press, pp. 269–282.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hsiao, K. -Y., S. Tellex, S. Vosoughi, R. Kubat, and D. Roy. (2008). Object schemas for grounding language in a responsive robot. Connection Science 20(4), 253–276. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hauser, Marc D., Noam Chomsky, and W. Tecumseh Fitch. (2002) “The faculty of language: what is it, who has it, and how did it evolve?.” Science 298.5598: 1569–1579. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hockett, C. F., (1960). The origin of speech. Scientific American, 203(3), pp.88–97. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Iverson, J. M., Capirci, O. and Caselli, M. C., (1994). From communication to language in two modalities. Cognitive development, 9(1), pp.23–43. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Iverson, J. M., & Goldin-Meadow, S. (2005). Gesture paves the way for language development. Psychological science, 16(5), 367–371. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kamp, H. and Reyle, U. 1993. From discourse to logic: Introduction to model-theoretic semantics of natural language, formal logic and discourse
representation. Studies in Linguistics and Philosophy. Kluwer.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In M. R. Key (Ed.), The relationship of verbal and nonverbal communication, 207–228. The Hague: Mouton.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kendon, A. (1995). Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of pragmatics, 23(3), pp. 247–279.
Lascarides, A. and M. Stone. (2006). Formal semantics for iconic gesture. In Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue (BRANDIAL), pp. 64–71.
Lascarides, A. and M. Stone. (2009). A formal semantic analysis of gesture. Journal of Semantics 26(4), 393–449. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pika, S. and Mitani, J. C., (2009). The directed scratch: evidence for a referential gesture in chimpanzees. The prehistory of language, 1, pp.166–181. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pustejovsky, J. (1995). The Generative Lexicon. MIT Press, Cambridge, MA.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pustejovsky, J. (2013). Where things happen: On the semantics of event localization. In Proceedings of isa-9: International workshop on semantic annotation.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pustejovsky, J. and N. Krishnaswamy. (2014). Generating simulations of motion events from verbal descriptions. In Lexical and Computational Semantics (*SEM 2014). ACL. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pustejovsky, J. and J. L. Moszkowicz. (2011). The qualitative spatial dynamics of motion in language. Spatial Cognition & Computation 11(1), 15–44. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Rizzolatti, G., and Arbib, M. (1998). Language within our grasp. Trends in Neurosciences, 21, 188–194. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Schlenker, P. (2015). Visible meaning: Sign language and the foundations of semantics. Manuscript, Institut Jean-Nicod and New York University.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Vol. 626). Cambridge university press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Stalnaker, R. (2002). Common ground. Linguistics and philosophy 25(5), 701–721. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Steels, L. (1999). The talking heads experiment: Vol. I. Words and meaning (Special preedition). Brussels: Vrije Universiteit Brussel.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Steels, L. (2011). Modeling the cultural evolution of language. Physics of Life Reviews, 8(4), 339–356. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Stout, Dietrich, & Hecht, Erin E. (2017). Evolutionary neuroscience of cumulative culture. Proceedings of the National Academy of Sciences, 114(30), 7861–7868. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Strickland, B., Geraci, C., Chemla, E., Schlenker, P., Kelepir, M., and Pfau, R. (2015). Event representations constrain the structure of language: Sign language as a window into universally accessible
linguistic biases. Proceedings of the National Academy of Sciences. doi: ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tomasello, M. and M. Carpenter. (2007). Shared intentionality. Developmental science 10(1), 121–125. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Vendler, Z. (1967). Linguistics and Philosophy. Ithaca: Cornell University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Volterra, V., Caselli, M. C., Capirci, O., & Pizzuto, E. (2005). Gesture and the emergence and development of language. Beyond nature-nurture: Essays in honor of Elizabeth Bates, 3–40.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
von Uexkll, J. (1957). A stroll through the worlds of animals and men: A picture book of invisible worlds, in Schiller, C. H. (ed), Instinctive Behavior: The Development of a Modern Concept. New York: International Universities Press, 5–80.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Wang, I., M. Ben Fraj, P. Narayana, D. Patil, G. Mulay, R. Bangar, R. Beveridge, B. Draper, and J. Ruiz. (2017). Eggnog: A continuous, multimodal data set of naturally occurring gestures with ground truth labels. In Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition.
Whitehead, A. N. (1919). An Enquiry concerning the Principles of Natural Knowledge, Cambridge: Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ziemke, T. & Sharkey, N. E. (2001). A stroll through the worlds of robots and animals: Applying Jakob von Uexkll’s theory of meaning to adaptive robots
and artificial life. Semiotica, 2001(134), 701–746. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)