Article published in:
Visually Situated Language ComprehensionEdited by Pia Knoeferle, Pirita Pyykkönen-Klauck and Matthew W. Crocker
[Advances in Consciousness Research 93] 2016
► pp. 323–356
Coordinating action and language
Thomas A. Farmer | Department of Psychology & Delta Center, University of Iowa
Sarah E. Anderson | Department of Psychology, University of Cincinnati
Jonathan B. Freeman | Department of Psychology, New York University
Rick Dale | Cognition and Information Sciences, University of California – Merced
The overarching aim of this chapter is to highlight the strong interfacing between linguistic and motor systems, focusing especially on the degree to which manipulations of a visual context interact with linguistic manipulations in order to modulate the properties of manual movements elicited in response to a linguistic signal. First, we provide a review of recent work in the domain of embodied cognition that highlights the engagement of motor systems during language comprehension, including gestures, eye movements, and postural sway. In the context of this body of work, we then focus on the notion that continuous and non-ballistic computer-mouse movement trajectories can serve as a viable motoric index of various aspects of language processing. The results of studies that have recorded mouse-movement trajectories in visual-world displays will be reviewed, with the goal of highlighting the methodological benefits of data contained in the recorded trajectories, along with their theoretical consequences. New data will be presented from a study in which both eye- and hand-movements were simultaneously recorded, speaking to the relationship between the two movement-related dependent measures. Finally, we will highlight the degree to which mouse-movements around a visual display can be informative for domains other than syntactic processing, focusing on new data from research designed to investigate the processing of verbal aspect.
Published online: 10 March 2016
https://doi.org/10.1075/aicr.93.12far
https://doi.org/10.1075/aicr.93.12far
References
References
Allopenna, P.D., Magnuson, J.S., & Tanenhaus, M.K.
Altmann, G.T.M., Garnham, A., & Dennis, Y.
Altmann, G.T.M., Garnham, A., & Henstra, J.
Altmann, G.T.M., & Kamide, Y.
Altmann G.T.M., & Steedman, M.
Altmann, G.T.M., van Nice, K., Garnham, A., & Henstra, J.A.
Anderson, S.E., Farmer, T.A., Goldstein, M., Schwade, J., & Spivey, M.
(2011) Individual differences in measures of linguistic experience account for variability in the sentence processing skill of five-year-olds. In I. Arnon & E.V. Clark (Eds.), Experience, variation, and generalization: Learning a first language (pp. 203-221). Amsterdam, The Netherlands: John Benjamins. 

Anderson, S.E., Matlock, T., Fausey, C., & Spivey, M.J.
(2008) On the path to understanding on-line processing of grammatical aspect. In
Proceedings of the 30th Annual Conference of the Cognitive Science Society
(pp. 143-148). Mahwah, NJ: Lawrence Erlbaum Associates.
Anderson, S.E., Matlock, T., & Spivey, M.
Bever, T.G.
Boulenger, V., Roy, A.C., Paulignan, Y., Deprez, V., Jeannerod, M., & Nazir, T.A.
Britt, M.A.
Brown, C.M., Van Berkum, J.J.A., & Hagoort, P.
Calvert, S., Rideout, V., Woolard, J., Barr, R., & Strouse, G.
Casasanto, D., & Lozano
(2007) Embodied language production: Evidence from gesture, speech, disfluency, and motor action. Paper presented at
Embodied Sentence Processing Conference: Behavioral, Neuropsychological, and Computational Perspectives
. Saerbruecken, Germany.
Cooper, R.M.
Cisek, P., & Kalaska, J.F.
Dale, R., & Duran, N.D.
Dale, R., Kehoe, C., & Spivey, M.J.
Dale, R., Roche, J.M., Snyder, K., & McCall, R.
Doyle, M., & Walker, R.
Duran, N.D., Dale, R., & McNamara, D.
Engelhardt, P.E., Bailey, K.G., & Ferreira, F.
Farmer, T.A., Anderson, S., & Spivey, M.J.
Farmer, T.A., Cargill, S.E., Hindy, N., Dale, R., & Spivey, M.J.
Farmer, T.A., Christiansen, M.H., & Kemtes, K.A.
(2005) Sentence processing in context: The impact of experience on individual differences. In
Proceedings of the 27th Annual Meeting of the Cognitive Science Society
(pp. 642-647). Mahwah, NJ: Lawrence Erlbaum.
Farmer, T.A., Davidow, J., & McCandliss, B.D.
(2009) Integrating eye- and hand-movements to more accurately index competition-based effects in the visual worldparadigm. Poster presented the
22nd Annual Meeting of the CUNY Conference on Human Sentence Processing
. Davis, CA.
Farmer, T.A., Liu, R., Metha, N., & Zevin, J.
Ferraina, S., Johnson, P.B., Garasto, M.R., Battaglia-Mayer, A., Ercolani, L., Bianchi, L., Lacquaniti, F.., et al.
Ferreira, F., Foucart, A., & Engelhardt, P.E.
Ferreira, F., & Clifton, C.
Ferretti, T.R., Kutas, M., & McRae, K.
Frazier, L., & Fodor, J.
Freeman, J.B., & Ambady, N.
Freeman, J.B., Ambady, N., Rule, N.O., & Johnson, K.L.
Freeman, J.B., Dale, R., & Farmer, T.A.
Gaskell, M.G., & Marslen-Wilson, W.
Glenberg, A.M. & Kashak, M.P.
Gold, J., & Shadlen, M.
Gold, J.I., & Shadlen, M.N.
Goodale, M., Pélisson, D., & Prablanc, C.
Godijn, R., & Theeuwes, J.
Green, M., & Mitchell, D.
Hauk, O., Johnsrude, I., & Pulvermuller, F.
Hauk, O., & Pulvermuller, F.
Havas, D.A., Glenberg, A.M., Gutowski, K.A., Lucarelli, M.J., & Davidson, R.J.
Henis, E.A., & Flash, T.
Hindy, N.C., Hamilton, R., Houghtling, A.S., Coslett, H.B., & Thompson-Schill, S.L.
Huette, S., Kello, C.T., Rhodes, T., & Spivey, M.J.
Johnson, P.B., Ferraina, S., Bianchi, L., & Caminiti, R.
Kalaska, J.F., Scott, S., Cisek, P., & Sergio, L.E.
Knoeferle, P., & Crocker, M.W.
Langacker, R.W.
Levy, J., & Staub, A.
Louwerse, M.M.
Luce, P., Goldinger, S., Auer, E., & Vitevitch, M.
Madden, C.J., & Zwann, R.A.
Magliano, J.P., & Schleich, M.C.
MacDonald, M., Pearlmutter, N., & Seidenberg, M.
Mahon, B.Z., & Caramazza, A.
Magnuson, J.S.
(2005) Moving hand reveals dynamics of thought.
Proceedings of the National Academy of Sciences
, 102, 9995-9996.
Markman, A., & Brendl, C.
Matlock, T., Fausey, C., Cargill, S., & Spivey, M.
McClelland, J.L., & Elman, J.L.
McKinstry, C., Dale, R., & Spivey, M.J.
McMurray, B., Tanenhaus, M.K., & Aslin, R.N.
McRae, K., Spivey-Knowlton, M.J., & Tanenhaus, M.K.
Morrow, D.G.
Nazir, T.A., Boulenger, V., Roy, A., Silber, B., Jeannerod, M., & Paulignan, Y.
Port, N.L., & Wurtz, R.H.
Pulvermüller, F., Hauk, O., Nikulin, V.V., & Ilmoniemi, R.J.
Pylyshyn, Z.
Rosenbaum, D.A.
Schmidt, T.
Sedivy, J.C., K Tanenhaus, M., Chambers, C.G., & Carlson, G.N.
Sheliga, B.M., Riggio, L., & Rizzolatti, G.
Shen, L., & Alexander, G.E.
Smith, J.R., Treat, T.A., McMurray, B., & Farmer, T.A.
(2014) Clothing style and attractiveness influence online processing of women’s sexual interest. Poster presented at the
26th Annual Convention of the Association for Psychological Science
. San Francisco, CA.
in preparation). Men’s online processing of women’s sexual interest: Generalization of dynamic models of decision making.
Snedeker, J., & Trueswell, J.
Song, J.H., & Nakayama, K.
Spivey, M.J., Anderson, S.E., & Farmer, T.A.
Spivey, M.J., Dale, R., Knoblich, G., & Grosjean, M.
Spivey, M., Grosjean, M., & Knoblich, G.
Spivey, M.J., & Tanenhaus, M.K.
Spivey, M., Tanenhaus, M.K., Eberhard, K. & Sedivy, J.
Spivey-Knowlton, M.J., & Sedivy, J.
Tanenhaus, M.K.
Tanenhaus, M.K., Spivey-Knowlton, M., Eberhard, K., & Sedivy, J.
Theeuwes, J., Olivers, C.N.L., & Chizk, C.L.
Thothathiri, M., & Snedeker, J.
Traxler, M., Pickering, M.J., & Clifton, C.
Trueswell, J.C., Sekerina, I., Hill, N.M., & Logrip, M.L.
Trueswell, J.C., Tanenhaus, M.K., & Garnsey, S.M.
van Berkum, J.J.A., Brown, C.M., & Hagoort, P.
van Berkum, J.J.A., Brown, C.M., Hagoort, P., & Zwitserlood, P.
van der Wel, R.P.R.D., Eder, J., Mitchel, A., Walsh, M., & Rosenbaum, D.
van Gompel, R.P.G., Pickering, M.J., Pearson, J., & Liversedge, S.P.
van Gompel, R., Pickering, M.J., & Traxler, M.
Wifall, T., Buss, A.T., Farmer, T.A., Spencer, J.P., & Hazeltine, E.
submitted). Reaching into response selection: Stimulus and response similarity influence central operations.