From actions to events
Communicating through language and gesture
In this paper, I argue that an important component of the language-ready brain is the ability to recognize and conceptualize
events. By ‘event’, I mean any situation or activity in the world or our mental life, that we find salient enough to individuate
as a thought or word. While this may sound either trivial or non-unique to humans, I hope to show that abstracting away events and
their participants from the embodied flow of experience is a characteristic unique to humans. This ability is enabled, I will
argue, by two critical competencies that act as scaffolds for language-ready thought in the prehuman brain. The first, as argued
by Arbib (2006, 2012, 2016) and others, is a sophisticated system of gesture production and understanding in prehumans, which
provided a template for schema-like sequencing and slot-filling of information units. The second involves the integration of
multiple modalities of expression in the communicative act, in particular, the alignment of co-gestural speech and co-speech
gesture. With such computational facilities, action-based gestures can be abstracted away from their associated objects and become
full event representations. This view supports the MSH argument for the emergence of more complex linguistic expressions from
initially holophrastic units. In particular, actions can be thought of as protoverbs, which through this process are abstracted to
full event descriptions, i.e., verbs.
Article outline
- 1.Introduction
- 2.Communication with a common ground
- 3.The structure of actions and events
- 4.The interpretation of gesture
- 5.Gesture sequences
- 6.Multimodal communication: Gesture-speech ensembles
- 6.1Co-gestural demonstratives
- 6.2More expressive co-gestural ensembles
- 7.Meaning in the absence of common groun
- 8.Toward a new road map
-
Acknowledgements
-
References
References (71)
References
Aboitiz, F. (2013) “How did vocal behavior ‘take over’ the gestural communication system?.” Language and Cognition 5, no. 2–3: 167–176.
Abner, N., Cooperrider, K., & Goldin-Meadow, S. (2015). Gesture for linguists: A handy primer. Language and linguistics compass, 9(11), 437–451.
Arbib, M. A. (2002) The Mirror System, Imitation, and the Evolution of Language. Imitation in animals and artifacts, 229.
Arbib, M. A. ed. (2006) Action to language via the mirror neuron system. Cambridge University Press.
Arbib, M. A. (2008). From grasp to language: Embodied concepts and the challenge of abstraction. Journal of Physiology-Paris, 102(1–3), 4–20.
Arbib, M. A. (2012). How the Brain Got Language: The Mirror System Hypothesis. New York & Oxford: Oxford University Press.
Arbib, M. A. (2013). Precis of how the brain got language: the mirror system hypothesis. Language and Cognition, 5(2–3), 107–131.
Arbib, M. A. (2016). Towards a computational comparative neuroprimatology: framing the language-ready brain. Physics of life reviews, 16, 1–54.
Arbib, M. A., Liebal, K. and Pika, S., (2008). “Primate vocalization, gesture, and the evolution of human language.” Current anthropology 49. 6, 1053–1076.
Arbib, M. A. & Rizzolatti, G. (1997). Neural expectations: a possible evolutionary path from manual skills to language. Communication and Cognition, 29, 393–424.
Armstrong, D. F., Stokoe, W. C. and Wilcox, S. E. 1995. Gesture and the nature of language. Cambridge University Press.
Armstrong, D. F., & Wilcox, S. (2007). The gestural origin of language. Oxford University Press.
Asher, N. (1998). Common ground, corrections and coordination. Journal of Semantics.
Asher, N. and Lascarides, A., (2003). Logics of conversation. Cambridge University Press.
Asher, N. and A. Lascarides. (2008). Commitments, beliefs and intentions in dialogue. Proceedings of Londial, 35–42.
Austin, J. L. (1962). How to do things with words. London: Oxford University Press
Bach, K. (2003). Speech acts and pragmatics. Blackwell. Guide to the philosophy of language, 147–167.
Bickerton, D. (1990) Language and Species, University of Chicago Press. Bickerton, Derek (2009). Adam’s Tongue. Hill and Wang.
Bickerton, D. 2009. Adam’s tongue: how humans made language, how language made humans. Macmillan.
Bohn, M., Call, J. and Tomasello, M., (2016). Comprehension of iconic gestures by chimpanzees and human children. Journal of experimental child psychology, 142, pp.1–17.
Cangelosi, A. (2010). Grounding language in action and perception: from cognitive agents to humanoid robots. Physics of life reviews 7(2), 139–151.
Clark, H. H. (1996). Using language. Cambridge University Press.
Clark, H. H., S. E. Brennan, et al. (1991). Grounding in communication. Perspectives on socially shared cognition 13(1991), 127–149.
Clark, H. H., Schreuder, R., & Buttrick, S. (1983). Common ground at the understanding of demonstrative reference. Journal of verbal learning and verbal behavior, 22(2), 245–258.
Cooper, R. and Ginzburg, J. (2015). “Type theory with records for natural language semantics”. Handbook of Contemporary Semantic Theory, pp.375–407.
Corballis, Michael C., (2003) From hand to mouth: The origins of language. Princeton University Press.
Deacon, Terrence W. (1998). The symbolic species: The co-evolution of language and the brain. WW Norton & Company.
Diessel, H. (2013) “Where does language come from? Some reflections on the role of deictic gesture and demonstratives in the evolution
of language.” Language and Cognition 5.2–3: 239–249.
Engle, R.A. and Clark, H.H. 1995. Using composites of speech, gestures, diagrams and demonstrations in explanations of mechanical
devices. In American Association for Applied Linguistics Conference, Long Beach, CA.
Fogassi, L., Coude, G. and Ferrari, P. F., (2013). The extended features of mirror neurons and the voluntary control of vocalization in the pathway to
language. Language and Cognition, 5(2–3), pp.145–155.
Fricke, E. (2013). Towards a unified grammar of gesture and speech: A multimodal approach. Body-language-communication. An international handbook on multimodality in human interaction, 733–754.
Gillespie-Lynch, K., Greenfield, P. M., Lyn, H., & Savage-Rumbaugh, S. (2014). Gestural and symbolic development among apes and humans: support for a multimodal theory of language
evolution. Frontiers in psychology, 5.
Gibson, J. J. (1979). The ecological approach to visual perception: classic edition. Psychology Press.
Gilbert, M. (1992). On social facts. Princeton University Press.
Glenberg, A. M. & Gallese, V. (2012). Action-based language: a theory of language acquisition, comprehension, and production. Cortex, 48(7), 905–22.
Goldin-Meadow, S. and Alibali, M.W. 2013. Gesture’s role in speaking, learning, and creating language. Annual review of psychology, 64, pp. 257-283.
Grice, H. P. (1981). Presupposition and conversational implicature. In C. Peter (ed.), Radical pragmatics. New York: Academic Press, pp. 183–198. Reprinted in Grice, H. P. (1989) Studies in the way of words. Cambridge: Harvard University Press, pp. 269–282.
Hsiao, K. -Y., S. Tellex, S. Vosoughi, R. Kubat, and D. Roy. (2008). Object schemas for grounding language in a responsive robot. Connection Science 20(4), 253–276.
Hauser, Marc D., Noam Chomsky, and W. Tecumseh Fitch. (2002) “The faculty of language: what is it, who has it, and how did it evolve?.” Science 298.5598: 1569–1579.
Hockett, C. F., (1960). The origin of speech. Scientific American, 203(3), pp.88–97.
Iverson, J. M., Capirci, O. and Caselli, M. C., (1994). From communication to language in two modalities. Cognitive development, 9(1), pp.23–43.
Iverson, J. M., & Goldin-Meadow, S. (2005). Gesture paves the way for language development. Psychological science, 16(5), 367–371.
Kamp, H. and Reyle, U. 1993. From discourse to logic: Introduction to model-theoretic semantics of natural language, formal logic and discourse
representation. Studies in Linguistics and Philosophy. Kluwer.
Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In M. R. Key (Ed.), The relationship of verbal and nonverbal communication, 207–228. The Hague: Mouton.
Kendon, A. (1995). Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of pragmatics, 23(3), pp. 247–279.
Lascarides, A. and M. Stone. (2006). Formal semantics for iconic gesture. In Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue (BRANDIAL), pp. 64–71.
Lascarides, A. and M. Stone. (2009). A formal semantic analysis of gesture. Journal of Semantics 26(4), 393–449.
Pika, S. and Mitani, J. C., (2009). The directed scratch: evidence for a referential gesture in chimpanzees. The prehistory of language, 1, pp.166–181.
Pustejovsky, J. (1995). The Generative Lexicon. MIT Press, Cambridge, MA.
Pustejovsky, J. (2013). Where things happen: On the semantics of event localization. In Proceedings of isa-9: International workshop on semantic annotation.
Pustejovsky, J. and N. Krishnaswamy. (2014). Generating simulations of motion events from verbal descriptions. In Lexical and Computational Semantics (*SEM 2014). ACL.
Pustejovsky, J. and J. L. Moszkowicz. (2011). The qualitative spatial dynamics of motion in language. Spatial Cognition & Computation 11(1), 15–44.
Rizzolatti, G., and Arbib, M. (1998). Language within our grasp. Trends in Neurosciences, 21, 188–194.
Schlenker, P. (2015). Visible meaning: Sign language and the foundations of semantics. Manuscript, Institut Jean-Nicod and New York University.
Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Vol. 626). Cambridge university press.
Stalnaker, R. (2002). Common ground. Linguistics and philosophy 25(5), 701–721.
Steels, L. (1999). The talking heads experiment: Vol. I. Words and meaning (Special preedition). Brussels: Vrije Universiteit Brussel.
Steels, L. (2011). Modeling the cultural evolution of language. Physics of Life Reviews, 8(4), 339–356.
Stout, Dietrich, & Hecht, Erin E. (2017). Evolutionary neuroscience of cumulative culture. Proceedings of the National Academy of Sciences, 114(30), 7861–7868.
Strickland, B., Geraci, C., Chemla, E., Schlenker, P., Kelepir, M., and Pfau, R. (2015). Event representations constrain the structure of language: Sign language as a window into universally accessible
linguistic biases. Proceedings of the National Academy of Sciences. doi:
Tomasello, M. and M. Carpenter. (2007). Shared intentionality. Developmental science 10(1), 121–125.
Vendler, Z. (1967). Linguistics and Philosophy. Ithaca: Cornell University Press.
Volterra, V., Caselli, M. C., Capirci, O., & Pizzuto, E. (2005). Gesture and the emergence and development of language. Beyond nature-nurture: Essays in honor of Elizabeth Bates, 3–40.
von Uexkll, J. (1957). A stroll through the worlds of animals and men: A picture book of invisible worlds, in Schiller, C. H. (ed), Instinctive Behavior: The Development of a Modern Concept. New York: International Universities Press, 5–80.
Wang, I., M. Ben Fraj, P. Narayana, D. Patil, G. Mulay, R. Bangar, R. Beveridge, B. Draper, and J. Ruiz. (2017). Eggnog: A continuous, multimodal data set of naturally occurring gestures with ground truth labels. In Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition.
Whitehead, A. N. (1919). An Enquiry concerning the Principles of Natural Knowledge, Cambridge: Cambridge University Press.
Ziemke, T. & Sharkey, N. E. (2001). A stroll through the worlds of robots and animals: Applying Jakob von Uexkll’s theory of meaning to adaptive robots
and artificial life. Semiotica, 2001(134), 701–746.