Part of
Multimodality and Cognitive LinguisticsEdited by María Jesús Pinar Sanz
[Benjamins Current Topics 78] 2015
► pp. 167–180
Edited by María Jesús Pinar Sanz
[Benjamins Current Topics 78] 2015
► pp. 167–180
Intermedial cognitive semiotics
Some examples of multimodal cueing in virtual environments
In this chapter, intermediality is explored from an interdisciplinary perspective that uses neuroscience as well as cognitive-semiotic concerns and insights from online digital communication, presenting it as a process where biophysical, technological, and interpersonal factors interact. Shared attention as well as spatial and temporal cueing – eye contact and the sonic modality – are explored from a task-oriented and social interactive dimension. The spatiotemporal impact of the mediating context is highlighted by moving from the role of visual cueing, in a brief reference to Al Davison’s autobiographical graphic novel The spiral cage, to a more detailed analysis of Annie Abrahams’ (2010) online project A fragmented relation, where cueing is dependent not just on spatial frames but also on the temporal dynamics introduced by the aural dimension recorded in an online environment. The paper tangentially touches upon the role of affect in communication.
References
Abrahams, A
Abry, C., Vilain, A., & Schwartz, J.L
(2004) Vocalize to localize? A call for better crosstalk between auditory and visual communication systems. Interaction Studies, 5, 313–325. 
Bach, D., Schachinger, H., Neuhoff, J., Esposito F., & Di Salle, F
(2008) Rising sound intensity: An intrinsic warning cue activating the amygdala. Cerebral Cortex, 18, 145–150.
Blake, R., Sobel, K.V., & James, T.W
(2004) Neural synergy between kinetic vision and touch. Psychological Science, 15(6), 397–402.
Bouchardon, S., & LópezVarela, A
(2011) Making sense of the digital as embodied experience. CLCWeb: Comparative Literature and Culture, 13(3).
Churchland, P
(2012) Braintrust: What neuroscience tells us about morality. Princeton: Princeton University Press.
Critchley, M
(1977) Ecstatic and synaesthetic experiences during musical perception. In M. Critchley & R.A. Henson (Eds.). Music and the brain: Studies in the neurology of music (pp. 217–230). London: Heinemann.
Davison, A
(2003) The spiral cage. London: Active Images.
Ernst, M., Banks, M., & Bülthoff, H
(2000) Touch can change visual slant perception. Nature Neuroscience, 3(1), 69–73.
Fauconnier, G
(1994) Mental spaces. New York: Cambridge University Press.
(1997) Mappings in thought and language. New York: Cambridge University Press.
Fillmore, C
(1975) Santa Cruz lectures on deixis. Bloomington: Indiana University Linguistics Club.
Forceville, C., & Urios-Aparisi, E
(2009) Multimodal metaphor. Berlin/New York: Mounton de Gruyter.
Gallese, V., & Lakoff, G
(2005) The brain’s concepts: The role of the sensory-motor system in reason and language. Cognitive Neuropsychology, 22, 455–479.
Gallese, V
(2006) Intentional attunement: A neurophysiological perspective on social cognition and its disruption in autism. Brain Res. Cog. Brain Res, 1079, 15–24.
(2008) Mirror neurons and the social nature of language: The neural exploitation hypothesis. Social Neuroscience, 3, 317–333.
George, N., & Conty, L
(2008.) Facing the gaze of others. Clinical Neurophysiology, 38(3), 197–207.
Glickstein, M., & Doron, K
(2008) Cerebellum: Connections and Functions. The Cerebellum, 7(4), 589–594.
Gillespie, A., & Cornish, F
(2010) Intersubjectivity: Towards a dialogical analysis. Journal for the Theory of Social Behaviour, 40, 19–46.
Grishakova, M
(2012) On cognitive and semiotic functions of shifters. Chinese Semiotic Studies, 8(2), 227–238.
Guest, S., Catmur, C., Lloyd, D., & Spence, C
(2002) Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161–171.
Iacoboni, M
(2005) Understanding others: imitation, language, empathy. In S. Hurley & N. Chater (Eds.), Perspectives on imitation: From cognitive neuroscience to social science (pp. 77–99). Cambridge, MA: MIT Press.
Jenson, D., & Iacoboni, M
(2011) Literary biomimesis: Mirror neurons and the ontological priority of representation. California Italian Studies, 2(1) [URL].
Kemmerer, D
(2006) The semantics of space: integrating linguistic typology and cognitive neuroscience. Neuropsychology, 44, 1607–1621.
Kurylowicz, J
(1964) The inflectional categories of Indo-European. Heidelberg: Carl Winter Universitätsverlag.
Lopez-Poveda, E., Palmer, A.R., & Meddis, R
(2010) The neurophysiological bases of auditory perception. New York: Springer Verlag.
López-Varela, A
(2010) Exploring intercultural relations from the intersubjective perspectives offered through creative art in multimodal formats. Lexia, 5–6, 125–147.
(2011) Multimodal metaphor and intersubjective experiences: the importance of eye-contact in Davison’s graphic novel ‘The Spiral Cage’ and in Annie Abrahams Net-Project ‘On Collaboration’. In L. Masucci & G. Di Rosario (Eds.), Lavori del Convegno Palazzo degli Artista Italiani (pp. 307–324). Naples: Oficina di Letterature Electrónica.
Maravita, A., Spence, Ch., Kennett, S., & Driver, J
(2002) Tool-use changes multimodal spatial interactions between vision and touch in normal humans. Cognition, 83, 25–34.
Magosso, E., Serino, A., Pellegrino, G.D., & Ursino, M
(2010) Crossmodal links between vision and touch in spatial attention: A computational modeling study. Computational Intelligence and Neuroscience. [URL]
[URL]Peirce, C.S
(1906) Prolegomena to an apology for pragmaticism, The Monist, 5(4), 492–546. Reprinted in C. Hartshorne & A. W. Weiss (Eds.), Collected papers of C. S. Peirce. Volume V(pp. 530–572). Cambridge, MA: Harvard University Press.
(1932) Division of signs. In C. Hartshorne & A.W. Weiss (Eds.), Collected papers of C. S. Peirce. Volume II. Cambridge, MA: Harvard University Press.
(1991) On the nature of signs. In J. Hoopes (Ed.), Peirce on signs. Chapel Hill: University of North Carolina Press.
Pylyshyn, Z.W
(2000) Situating vision in the world. Trends in Cognitive Sciences, 4(5), 107–207.
Rensink, R
(2000) The dynamic representation of scenes. Visual Cognition, 7, 17–42.
Searle, J.R
(1995) The construction of social reality. London: Allen Lane.
Schmidt, R.A
(1982) Motor control and learning: A behavioural emphasis. Champaign, IL: Human Kinetics Publishers Inc.
(1991) Motor learning and performance. Champaign, IL: Human Kinetics Publishers Inc.
Tresilian, J
(2012) Sensorimotor control and learning: An introduction to the behavioral neuroscience of action. London: Palgrave Macmillan.
Sperber, D., & Wilson, D
(1986) Relevance: Communication and cognition. Oxford: Blackwell.
Stolorow, R.D., & Atwood, G.E
(1992) Contexts of being: The intersubjective foundations of psychological life. Hillsdale, NJ & London: The Analytic Press.
Wade, N.J
Wind, J
Zlatev, J
(2008) The co-evolution of intersubjectivity and bodily mimesis. In J. Zlatev, T. Racine, C. Sinha & E. Itkonen (Eds.), The shared mind: Perspectives on intersubjectivity (pp. 215–244). Amsterdam/Philadelphia: John Benjamins.