Cover not available
Part of
Multimodality and Cognitive Linguistics
Edited by María Jesús Pinar Sanz
[Benjamins Current Topics 78] 2015
► pp. 147–164

Multimodal digital storytelling

Integrating information, emotion and social cognition

Digital stories are a very recent multimedia practice by which ordinary people construct short narratives on personal affairs combining voice, images and sometimes music. This paper contributes to the description of this new emergent genre from both a multimodal and a cognitive point of view, by exploring how diverse semiotic channels in digital storytelling provide different kinds of information (factual, emotional, cultural, etc.) which are finally integrated to construct the global meaning of the narrative. For this purpose, we combine Kress and Van Leeuwen’s (1996) scholarly work related to multimodal representation, with the use of some notions of the Mental Spaces and Conceptual Integration theory (Dancygier, 2008; Fauconnier & Turner, 2002). The results of this study are of interest to those concerned with the representational and communicational modes of semiotic resources in storytelling.

Keywords: conceptual integration, digital storytelling, mental spaces, multimodality, narrative
Published online: 15 October 2015
DOI logo
https://doi.org/10.1075/bct.78.10alo
References
Cameron, L
(1999) Operationalising metaphor for applied linguistic research. In L. Cameron & G. Low (Eds.), Researching and applying metaphor (pp. 3–28). Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Dancygier, B
(2008) The text and the story: levels of blending in fictional narratives. In T. Oakley & A. Hougaard (Eds.), Mental spaces in discourse and interaction (pp. 51–78). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Emmott, C
(1997) Narrative comprehension: A discourse perspective. Oxford: Oxford University Press.Google Scholar
Fauconnier, G
(1997) Mappings in thought and language. Cambridge, MA: Cambridge University Press. DOI logoGoogle Scholar
Fauconnier G., & Turner, M
(2002) The way we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books.Google Scholar
Halliday, M.A.K., & Matthiessen, C.M
(2004) An introduction to functional grammar. Third revised edition. London: Edward Arnold.Google Scholar
Herman, D
(2003) Stories as a tool for thinking. In D. Herman (Ed.), Narrative theory and the cognitive sciences (pp. 163–192). Stanford: CSLI Publications.Google Scholar
(2004) Toward a transmedial narratology. In M.L. Ryan (Ed.), Narrative across media: The languages of storytelling (pp. 47–75). Lincoln: University of Nebraska Press.Google Scholar
(2009) Cognitive approaches to narrative analysis. In G. Brône & J. Vandaele (Eds.), Cognitive poetics: Goals, gains and gaps (pp. 79–118). Berlin: Mouton de Gruyter.Google Scholar
Hodge, R., & Kress, G
(1988) Social semiotics. Cambridge: Polity Press.Google Scholar
Kress, G., & van Leeuwen, T
(1996) (2nd edition 2006). Reading images: The grammar of visual design. London: Routledge.Google Scholar
Labov, W., & Waletzky, J
(1967) Narrative analysis: Oral versions of personal experience. In J. Helm (Ed.), Essays on the verbal and visual arts (pp. 12–44). Seattle, WA: University of Washington Press.Google Scholar
Lakoff, G., & Turner, M
(1989) More than cool reason: A field guide to poetic metaphor. Chicago: University of Chicago Press. DOI logoGoogle Scholar
O’Halloran, K., & Lim Fei, V
(2009) Sequential visual discourse frames. In E. Ventola & A.J. Moya Guijarro (Eds.), The world told and the world shown: Multisemiotic issues (pp. 139–156). London: Palgrave Macmillan.Google Scholar
Oakley, T., & Coulson, S
(2008) Connecting the dots: Mental spaces and metaphoric language in discourse. In T. Oakley & A. Hougaard (Eds.), Mental spaces in discourse and interaction (pp. 27–50). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Porto, D., & Romano, M
(2010) Conceptual integration in natural oral narratives. Actes des journées d’étude Narratology and the new social dimension of narrative (01–02 Février 2010). At [URL] (last accessed 8th March 2013).Google Scholar
Ryan, M.L
(2005) On the theoretical foundations of transmedial narratology. In J.C. Meister (Ed.), Narratology beyond literary criticism: Mediality, disciplinarity (pp. 1–23). Berlin/New York: Walter de Gruyter. DOI logoGoogle Scholar
Semino, E
(2009) Text worlds. In G. Brône & J. Vandaele (Eds.), Cognitive poetics: Goals, gains and gaps (pp. 33–37). Berlin: Mouton de Gruyter.Google Scholar
Turner, M
(2008) Conceptual integration. In D. Geraeerts & H. Cuyckens (Eds.), The Oxford handbook of cognitive linguistics (pp. 264–293). Oxford: Oxford University Press.Google Scholar
Unsworth, L., & Cleirigh, C
(2009) Multimodality and reading: The construction of meaning through image-text interaction. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 151–163). London: RoutledgeGoogle Scholar
Cited by

Cited by 1 other publications

Gkoutsioukosta, Zisoula & Venetia Apostolidou
2023. Building Learning Communities through Digital Storytelling. Social Sciences 12:10 ► pp. 541 ff. DOI logo

This list is based on CrossRef data as of 20 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.