Gesturing in the wild
Evidence for a flexible mental timeline
Psycholinguistic evidence shows that spatial domains are automatically activated when processing temporal expressions.
Speakers conceptualize time as a straight line deployed along different axes (mostly sagittal, though also vertical). The use of the lateral
axis, which cannot be lexicalized in any language, has nonetheless been attested in temporal tasks in laboratories using a variety of
experiments. This leads to the question of what axes are actually at work when conceptualizating time in oral communication.
The present study examines a great number of temporal expressions, taken from television shows, noting their associated
co-speech gestures. Our results show that (1) speakers overwhelmingly use the lateral axis; (2) they are not performing simple space-to-time
mappings, but are using instead a “timeline”, a material anchor which is a far more complex construct and that can explain some of the
intricacies and contextual variations shown in the pattern of results.
Article outline
- 1.Introduction
- 1.1The lateral axis
- 1.2Multimodality and the NewsScape Library of Television News
- 2.Gesturing the timeline: A multimodal study
- 2.1Research questions and hypotheses
- 2.2Materials and methodology
- 2.2.1Data collection
- 2.2.2Filtering
- 2.2.3Coding
- 3.Results
- 3.1Construal
- 3.2Axis
- 3.3Directionality
- 4.Discussion
- 4.1The lateral timeline: New evidence from spontaneous speech
- 4.2Flexibility
- 5.Conclusion
-
References
References
Alibali, M. W., Bassok, M., Solomon, K. D., Syc, S. E., & Goldin-Meadow, S.
(
1999)
Illuminating mental representations through speech and gesture.
Psychological Science, 101, 327–333.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Barrett, L.
(
2015)
Beyond the brain: how body and environment shape animal and human minds. Princeton, NJ: Princeton University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Boroditsky, L., & Gaby, A.
(
2010)
Remembrances of times east: Absolute spatial representations of time in an aboriginal Australian community.
Psychological Science, 211, 1635–1639. 10.1177/0956797610386621.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Boroditsky, L., Fuhrman, O., & McCormick, K.
(
2011)
Do English and Mandarin speakers think about time differently? Cognition, 118(1), 126–132.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Bressem, J.
(
2013)
A linguistic perspective on the notation of form features in gestures. In
C. Müller,
A. J. Cienki,
E. Fricke,
S. H. Ladewig,
D. McNeill &
S. Teßendorf (Eds.),
Body – language – communication. An international handbook on multimodality in human interaction (pp. 1079–1098). Berlin & New York: Mouton de Gruyter.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brewer, M.
(
2000)
Research design and issues of validity. In
H. Reis &
C. Judd (Eds.),
Handbook of research methods in social and personality psychology (pp. 11–26). Cambridge: Cambridge University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Calbris, G.
(
1990)
The semiotics of French gesture. Bloomington & Indianapolis: Indiana University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Casasanto, D., & Boroditsky, L.
(
2008)
Time in the mind: using space to think about time.
Cognition, 1061, 579–93.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Casasanto, D., & Bottini, R.
(
2014)
Mirror-reading can reverse the flow of time.
Journal of Experimental Psychology: General, 143(2), 473–479.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Casasanto, D., & Jasmin, K.
(
2012)
The hands of time: Temporal gestures in English speakers.
Cognitive Linguistics, 23(4), 643–674.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cienki, A.
(
1998)
Metaphoric gestures and some of their relations to verbal metaphoric expressions. In
J. -P. Koenig (Ed.),
Discourse and cognition: bridging the gap (pp. 189–204). Stanford, CA: CSLI Publications.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cienki, A., & Müller, C.
(
2008)
Metaphor, gesture and thought. In
R. W. Gibbs (Ed),
Cambridge handbook of metaphor and thought (pp. 483–501). Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Clark, A.
(
2008)
Supersizing the mind: Embodiment, action and cognitive extension. Oxford: Oxford University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Clark, H. H.
(
1996)
Using language. Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cooperrider, K., & Nuñez, R.
Coulson, S., & Pagán-Cánovas, C.
(
2013)
Understanding timelines: Conceptual metaphor and conceptual integration.
Cognitive Semiotics, 5(1–2), 198–219.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Dancygier, B.
(
2012)
The language of stories. Cambridge: Cambridge University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Dancygier, B., & Sweetser, E.
(Eds.) (
2012)
Viewpoint in language: A multimodal perspective. Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Dirven, R., & Parings, R.
(Eds.) (
2002)
Metaphor and metonymy in comparison and contrast. Berlin & New York: Mouton de Gruyter.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Engberg-Pedersen, E.
(
1993)
Space in Danish sign language: The semantics and morphosyntax of the use of space in a visual language. Hamburg: SIGNUM-Verlag.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Engberg-Pedersen, E.
(
1999)
Space and time. In
J. Allwood &
P. Gardenfors (Eds.),
Cognitive semantics: Meaning and cognition (pp. 131–152). Amsterdam & Philadelphia: John Benjamins.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Emmorey, K.
(
2001)
Space on hand: The exploitation of signing space to illustrate abstract thought. In
M. Gattis (Ed.),
Spatial schemas and abstract thought (pp. 147–174). Cambridge: The MIT Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fauconnier, G.
(
1997)
Mappings in thought and language. Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fauconnier, G., & Turner, M.
(
1998)
Conceptual integration networks.
Cognitive Science, 22(2), 133–187.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fauconnier, G., & Turner, M.
(
2002)
The way we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fillmore, C. J., Kay, P., & O’Connor, M.
(
1988)
Regularity and idiomaticity in grammatical constructions: The case of let alone.
Language, 64(3), 501–538.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Fuhrman, O., & Boroditsky, L.
(
2010)
Cross-cultural differences in mental representations of time: Evidence from an implicit nonlinguistic task.
Cognitive Science, 341, 1430–1451.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goldberg, A.
(
1995)
Constructions: A Construction Grammar approach to argument structure. Chicago: University of Chicago Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goldberg, A.
(
2006)
Constructions at work. Oxford: Oxford University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goldin-Meadow, S., & Alibali, M. W.
(
2013)
Gesture’s role in speaking, learning, and creating language.
Annual Review of Psychology, 1231, 448–453.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Goldin-Meadow, S.
(
2003)
Gesture.
Encyclopedia of cognitive science, volume 21 (pp. 289–293). London: Macmillan Reference Ltd.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hutchins, E.
(
2005)
Material anchors for conceptual blends.
Journal of Pragmatics, 371, 1555–1577.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ishihara, M., Keller, P. E., Rossetti, Y. & Prinz, W.
(
2008)
Horizontal spatial representations of time: Evidence for the STEARC effect.
Cortex, 44(4), 454–461.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Jewitt, C.
(Ed.) (
2009)
The Routledge handbook of multimodal analysis. London: Routledge.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kendon, A.
(
1995)
Gestures as illocutionary and discourse structure markers in Southern Italian conversation.
Journal of Pragmatics, 23(3), 247–279.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kendon, A.
(
2004)
Gesture: Visible action as utterance. Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kita, S.
(
2009)
Cross-cultural variation of speech-accompanying gesture: a review.
Language and Cognitive Processes, 24(2), 145–167.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kövecses, Z.
(
2005)
Metaphor in culture. Universality and variation. Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lakoff, G.
(
1993)
The contemporary theory of metaphor. In
A. Ortony (Ed.),
Metaphor and thought (pp. 202–251). Cambridge: Cambridge University Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lakoff, G., & Johnson, M.
(
1980)
Metaphors we live by. Chicago: University of Chicago Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lakoff, G., & Johnson, M.
(
1999)
Philosophy in the flesh. New York: Basic Books.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Le Guen, O., & Pool Balam, L. I.
(
2012)
No metaphorical timeline in gesture and cognition among Yucatec Mayas.
Frontiers in Cultural Psychology, 31, Article 271.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Leonard, T., & Cummins, F.
(
2011)
The temporal relation between beat gestures and speech.
Language and Cognitive Processes, 26(10), 1457–1471.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Levinson, S. C.
(
1996)
Frames of reference and Molyneux’s question: Cross-linguistic evidence. In
P. Bloom,
M. Peterson,
L. Nadel &
M. Garrett (Eds.),
Language and space (pp. 109–169). Cambridge, MA: MIT Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McNeill, D.
(
1992)
Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McNeill, D.
(
2005)
Gesture and thought. Chicago: University of Chicago Press.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McNeill, D.
(
2013)
Gesture as a window onto mind and brain, and the relationship to linguistic creativity and ontogenesis. In
C. Müller,
A. J. Cienki,
E. Fricke,
S. H. Ladewig,
D. McNeill &
S. Teßendorf (Eds.),
Body – language – communication. An international handbook on multimodality in human interaction. Volume I (pp. 28–54). Berlin & New York: Mouton de Gruyter.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Müller, C., Cienki, A., Fricke, E., Ladewig, S. H., McNeill, D., & Bressem, J.
(Eds.) (
2013)
Body – language – communication. An international handbook on multimodality in human interaction. Volume II. Berlin & New York: Mouton de Gruyter.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Müller, C., Cienki, A., Fricke, E., Ladewig, S. H., McNeill, D., & Bressem, J.
(Eds.) (
2014)
Body – language – communication. An international handbook on multimodality in human interaction. Volume II. Berlin & New York: Mouton de Gruyter.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Núñez, R., & Sweetser, E.
(
2006)
With the future behind them: Convergent evidence from Aymara language and gesture in the crosslinguistic comparison of spatial construals of time.
Cognitive Science, 301, 401–450.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ouellet, M., Santiago, J., Israeli, Z., & Gabay, S.
(
2010)
Is the future the right time? Experimental Psychology, 571, 308–314.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pagán Cánovas, C., Valenzuela, J., & Santiago, J.
(
2015)
Like the machete the snake: Integration of topic and vehicle in poetry comprehension reveals meaning construction processes.
Psychology of Aesthetics, Creativity, and the Arts, 9(4), 385–393.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pagán-Cánovas, C., & Valenzuela, J.
(
2017)
Timelines and multimodal constructions: facing new challenges.
Linguistics Vanguard, 3(1).
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pagán-Cánovas, C., Valenzuela, J., Alcaraz-Carrión, D., Olza, I., & Ramscar, M.
(
2020)
Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions.
PLoS ONE, 15(6), e0233892.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Radden, G.
(
2004)
The metaphor TIME AS SPACE across languages. In
N. Baumgarten,
C. Böttger,
M. Motz &
J. Probst (Eds.),
Übersetzen, interkulturelle Kommunikation, Spracherwerb und Sprachvermittlung – Das Leben mit Mehreren Sprachen: Festschrift für Juliane House zum 60 Geburtstag (pp. 225–238). Bochum, Germany: Aks-Verlag.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Richter, N.
(
2014)
Structuring discourse: Observations on prosody and gesture in Russian TV-discourse. In
C. Müller,
A. J. Cienki,
E. Fricke,
S. H. Ladewig,
D. McNeill &
S. Teßendorf (Eds.),
Body – language – communication. An international handbook on multimodality in human interaction. Volume II (pp. 1392–1400). Berlin & New York: Mouton de Gruyter.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Santiago, J., Lupiáñez, J., Perez, E., & Funes, M. J.
(
2007)
Time (also) flies from left to right.
Psychonomic Bulletin & Review, 14(3), 512–516.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Santiago, J., Román, A., & Ouellet, M.
(
2011)
Flexible foundations of abstract thought: A review and a theory.
Spatial Dimensions of Social Thought, 41–110.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Scott, A.
(
1989)
The vertical dimension and time in Mandarin.
Australian Journal of Linguistics, 91, 295–331.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sell, A. J., & Kaschak, M. P.
(
2011)
Processing time shifts affects the execution of motor responses.
Brain & Language, 1171, 39–44.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sinha, C., Silva Sinha, V. da, Zinken, J., & Sampaio, W.
(
2011)
When time is not space: The social and linguistic construction of time intervals and temporal event relations in an Amazonian culture.
Language and Cognition, 3(1), 137–169.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Steen, F., & Turner, M.
(
2013)
Multimodal Construction Grammar. In
M. Borkent,
B. Dancygier, &
J. Hinnell (Eds.),
Language and the creative mind (pp. 255–274). Stanford: CSLI Publications.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Stockwell, P.
(
1999)
The inflexibility of invariance.
Language and Literature, 8(2), 125–142.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Torralbo, A., Santiago, J., & Lupiáñez, J.
(
2006)
Flexible conceptual projection of time onto spatial frames of reference.
Cognitive Science, 30(4), 745–757.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Turner, M.
(
1996)
The literary mind. New York: Oxford University Press US.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Turner, M.
(
2014)
The origin of ideas: blending, creativity, and the human spark. New York: Oxford University Press US.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ulrich, R., Eikmeier, V., Vega, I., Ruiz Fernández, S., Alex-Ruf, S., & Maienborn, C.
(
2012)
With the past behind and the future ahead: Back-to-front representation of past and future sentences.
Memory & Cognition, 40(3), 483–495.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ulrich, R., & Maienborn, C.
(
2010)
Left–right coding of past and future in language: The mental timeline during sentence processing.
Cognition, 1171, 126–138.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Valenzuela, J., Hilferty, J., & Garachana, M.
Walker, E., & Cooperrider, K.
(
2015)
The continuity of metaphor: evidence from temporal gestures.
Cognitive Science, 40(2), 481–495.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Weger, U. W., & Jay, P.
(
2008)
Time flies like an arrow: Space–time compatibility effects suggest the use of a mental timeline.
Psychonomic Bulletin and Review, 15(2), 426–430.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Wang, L., & Chu, M.
(
2013)
The role of beat gesture and pitch accent in semantic processing: an ERP study.
Neuropsychologia, 51(13), 2847–55.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Zhang, J., & Patel, V. L.
Zima, E.
(
2014)
Gibt es multimodale Konstruktionen? Eine Studie zu [V(motion) in circles] und [all the way from X PREP Y].
Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion, 151, 1–48.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cited by
Cited by 2 other publications
Alcaraz Carrión, Daniel, Cristóbal Pagán Cánovas & Javier Valenzuela
2020.
Enaction Through Co-speech Gesture: The Rhetorical Handing of the Mental Timeline.
Zeitschrift für Anglistik und Amerikanistik 68:4
► pp. 411 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
Valenzuela, Javier & Daniel Alcaraz Carrión
2020.
Temporal Expressions in English and Spanish: Influence of Typology and Metaphorical Construal.
Frontiers in Psychology 11
![DOI logo](//benjamins.com/logos/doi-logo.svg)
This list is based on CrossRef data as of 27 april 2021. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.