Multimodal coherence research and its applications
Although it is nowadays a commonplace that communicative artefacts and
communicative behaviours almost always employ a variety of ‘modes’ of information
presentation, the issue of how such diverse contributions ‘cohere’ in
order to fulfill common communicative purposes is still an area of open debate.
This chapter presents a selective overview of the development and current
state of the art in approaches to multimodal coherence, focusing particularly
on work taking language as its starting point. Overviews and illustrations of
the approaches are given as well as a summary of some major challenges to be
addressed.
References (89)
Andre, Elisabeth. 1995. Ein planbasierter Ansatz zur Generierung multimedialer Prasentationen, Vol. 108. St. Augustin: Infix.
Asher, Nicholas, and Alex Lascarides. 2003. Logics of Conversation. Cambridge: Cambridge University Press.
Barthes, Roland. 1977 [1964]. “The Rhetoric of the Image.” In Image-Music-Text: 32–51. London: Fontana.
Bateman, John A. 2007. “Towards a Grande Paradigmatique of Film: Christian Metz Reloaded.” Semiotica 167 (1/4): 13–64.
Bateman, John A. 2008. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan.
Bateman, John A. 2013. “Using Multimodal Corpora for Empirical Research.” In The Routledge Handbook of Multimodal Analysis, 2nd ed., ed. by Carey Jewitt, 238–252. London: Routledge.
Bateman, John A. 2014a. “The Integration of Multimodal Resources in Documents: Issues, Approaches and Methods.” In Verbal Communication (Handbook of Communication Science), ed. by Louis de Saussure, and Andrea Rocci. Berlin/New York/Amsterdam: Mouton de Gruyter.
Bateman, John A. 2014b. Text and Image: A Critical Introduction to the Visual/Verbal Divide. London and New York: Routledge.
Bateman, John A., Judy L. Delin, and Renate Henschel. 2007. “Mapping the Multimodal Genres of Traditional and Electronic Newspapers.” In New Directions in the Analysis of Multimodal Discourse, ed. by Terry D. Royce, and Wendy L. Bowcher, 147–172. Lawrence Erlbaum Associates.
Bateman, John A., Thomas Kamps, Jorg Kleinz, and Klaus Reichenberger. 2001. “Constructive Text, Diagram and Layout Generation for Information Presentation: The DArtbio System.” Computational Linguistics 27 (3): 409–449.
Bateman, John A., and Karl-Heinrich Schmidt. 2012. Multimodal Film Analysis: How Films Mean (Routledge Studies in Multimodality). London: Routledge.
Bernhardt, Stephen A. 1985. “Text Structure and Graphic Design: The Visible Design.” In Systemic Perspectives on Discourse, Vol. 1, ed. by James D. Benson, and William S. Greaves, 18–38. Norwood, New Jersey: Ablex.
Bucher, Hans-Jiirgen. 2007. “Textdesign und Multimodalitat. Zur Semantik und Pragmatik medialer Gestaltungsformen.” In Textdesign und Textwirkung in der massenmedialen Kommunikation, ed. by Kersten Sven Roth, and Jürgen Spitzmüller, 49–76. Konstanz: UVK.
Bucher, Hans-Jürgen. 2011. “Multimodales Verstehen oder Rezeption als Interaktion. Theoretische und empirische Grundlagen einer systematischen Analyse der Multimodalitat.” In Bildlinguistik. Theorien – Methoden – Fallbeispiele, ed. by Hans-Joachim Diekmannshenke, Michael Klemm, and Hartmut Stockl, 123–156. Berlin: Erich Schmidt.
Cohn, Neil. 2013. “Visual Narrative Structure.” Cognitive Science 37 (3): 413–452.
Durand, Jacques. 1987. “Rhetorical Figures in the Advertising Image.” In Marketing and Semiotics: New Directions in the Study of Signs for Sale, ed. by Jean Umiker-Sebeok, 295–318. Berlin/New York/Amsterdam: Mouton de Gruyter.
Forceville, Charles J. 1996. Pictorial Metaphor in Advertising. London: Routledge.
Forceville, Charles J. 2002. “The Identification of Target and Source in Pictorial Metaphors.” Journal of Pragmatics 34 (1): 1–14.
Gaede, Werner. 1981. Vom Wort zum Bild: Kreativ-Methoden der Visualisierung. Munich: Langen-Muller/Herbig. 2nd ed., 1992.
Gibson, James J. 1977. “The Theory of Affordances.” In Perceiving, Acting, and Knowing: Toward and Ecological Psychology, ed. by Robert Shaw, and John Bransford, 62–82. Hillsdale, NJ: Erlbaum.
Glenberg, Arthur M., Alex Cherry Wilkinson, and William Epstein. 1982. “The Illusion of Knowing: Failure in the Self-assessment of Comprehension.” Memory and Cognition 10 (6): 597–602.
Grice, H. Paul. 1957. “Meaning.” Philosophical Review 66 (3): 377–388.
Grice, H. Paul. 1969. “Utterer’s Meaning and Intentions.” Philosophical Review 68 (2): 147–177.
Groupe µ. 1992. Traité du signe visuel: pour une rhétorique de l’image. Paris: Editions du Seuil.
Halliday, Michael A. K. 1978. Language as Social Semiotic. London: Edward Arnold.
Halliday, Michael A. K. 1985. An Introduction to Functional Grammar. London: Edward Arnold. (2nd edition 1994; page numbers in the text refer to the second edition).
Halliday, Michael A. K., and Ruqaiya Hasan. 1976. Cohesion in English. London: Longman.
Hill, Charles A., and Marguerite Helmers (eds). 2004. Defining Visual Rhetorics. Mahwah, NJ: Erlbaum.
Hjelmslev, Louis. 1953. Prolegomena to a Theory of Language. Bloomington, Indiana: Indiana University Publications in Anthropology and Linguistics. Translated by Francis J. Whitfield.
Holsanova, Jana, and Andreas Nord. 2010. “Multimedia Design: Media Structures, Media Principles and Users’ Meaning-making in Newspapers and Net Papers.” In Neue Medien – neue Formate. Ausdifferenzierung und Konvergenz in der Medienkommunikation (Interaktiva. Schriftenreihe des Zentrums fur Medien und Interaktivitat (ZMI), GieBen 10), ed. by Hans-Jurgen Bucher, Thomas Gloning, and Kartin Lehnen, 81–103. Frankfurt/New York: Campus Verlag.
Jewitt, Carey (ed.). 2009. The Routledge Handbook of Multimodal Analysis. London: Routledge.
Jones, Daniel. 1918. An Outline of English Phonetics. Leipzig and Berlin: B. G. Teubner.
Kjeldsen, Jens E. 2012. “Pictorial Argumentation in Advertising: Visual Tropes and Figures as a Way of Creating Visual Argumentation.” In Topical Themes in Argumentation Theory: Twenty Exploratory Studies (Argumentation Library 22), ed. by Frans H. van Eemeren, and Bart Garssen, 239–255. Berlin: Springer.
Kloepfer, Rolf. 1977. “Komplementaritat von Sprache und Bild. Am Beispiel von Comic, Karikatur und Reklame.” In Zeichenprozesse. Semiotische Forschung in den Einzelwis-senschaften, ed. by Roland Posner, and Hans-Peter Reinecke, 129–145. Wiesbaden: Athenaum.
Kress, Gunther. 2003. Literacy in the New Media Age. London: Routledge.
Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.
Kress, Gunther, Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis. 2000. Multimodal Teaching and Learning. London: Continuum.
Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
Kress, Gunther, and Theo van Leeuwen. 2006 [1996]. Reading Images: The Grammar of Visual Design. London and New York: Routledge.
Mann, William C., and Sandra A. Thompson. 1988. “Rhetorical Structure Theory: Toward a Functional Theory of Text Organization.” Text 8 (3): 243–281.
Martin, James R. 1983. “Conjunction: The Logic of English Text.” In Micro and Macro Connexity of Discourse (Papers in Textlinguistics 45), ed. by Janos S. Petofi, and Emel Sozer, 1–72. Hamburg: Helmut Buske Verlag.
Martin, James R., and David Rose. 2003. Working with Discourse: Meaning beyond the Clause. London and New York: Continuum.
Martin, James R., and David Rose. 2008. Genre Relations: Mapping Culture. London and New York: Equinox.
Martinec, Radan, and Anthony Salway. 2005. “A System for Image-text Relations in New (and Old) Media.” Visual Communication 4 (3): 337–371.
McQuarrie, Edward F., and David Glen Mick. 2003. “Visual and Verbal Rhetorical Figures Under Directed Processing Versus Incidental Exposure to Advertising.” Journal of Consumer Research 29 (4): 579–587.
Messaris, Paul. 1997. Visual Persuasion: The Role of Images in Advertising. Thousand Oaks, CA: Sage.
Meyer, Bonnie J. 1975. The Organization of Prose and its Effects on Memory. Amsterdam: North-Holland.
Mirzoeff, Nicholas. 1999. An Introduction to Visual Culture. London and New York: Routledge.
Mitchell, W. J. Thomas. 1994. Picture Theory: Essays on Verbal and Visual Representation. Chicago: University of Chicago Press.
Moriarty, Sandra E. 1996. “Abduction: A Theory of Visual Interpretation.” Communication Theory 6 (2): 167–187.
Muckenhaupt, Manfred. 1986. Text und Bild. Grundfragen der Beschreibung von Text-BildKommunikation aus sprachwissenschaftlicher Sicht (Tubinger Beitrage zur Linguistik). Tubingen: Narr.
Muller, Marion G. 2007. “What is Visual Communication? Past and Future of an Emerging Field of Communication Research.” Studies in Communication Sciences 7 (2): 7–34.
Ong, Walter Jackson. 1982. Orality and Literacy: The Technologizing of the Word. London: Methuen.
O’Toole, Michael. 1990. “A Systemic-functional Semiotics of Art.” Semiotica 82 (3/4): 185–209.
O’Toole, Michael. 2011 [1994]. The Language of Displayed Art. Abingdon, Oxon: Routledge.
Phillips, Barbara J., and Edward F. McQuarrie. 2004. “Beyond Visual Metaphor: A New Typology of Visual Rhetoric in Advertising.” Marketing Theory 4 (1/2): 113–136.
Power, Richard, Donia Scott, and Nadjet Bouayad-Agha. 2003. “Document Structure.” Computational Linguistics 29 (2): 211–260.
Rose, Gillian. 2012. Visual Methodologies. An Introduction to Researching with Visual Materials, 3rd ed. London/Thousand Oaks/New Delhi: Sage.
Royce, Terry D. 2002. “Multimodality in the TESOL Classroom: Exploring Visual-verbal Synergy.” TESOL Quarterly 36 (2): 191–205.
Royce, Terry D. 2007. “Intersemiotic Complementarity: A Framework for Multimodal Discourse Analysis.” In New Directions in the Analysis of Multimodal Discourse, ed. by Terry D. Royce, and Wendy L. Bowcher, 63–110. Lawrence Erlbaum Associates.
Sachs-Hombach, Klaus (ed.). 2001. Bildhandeln: interdisziplinäre Forschungen zur Pragmatik bildhafter Darstellungsformen (Reihe Bildwissenschaft 3). Magdeburg: Scriptum-Verlag.
Sachs-Hombach, Klaus, and Jorg R. J. Schirra. 2006. “Bildstil als rhetorische Kategorie.” Image – Zeitschrift fur interdisziplinäre Bildwissenschaft 3: 175–191.
Saussure, Ferdinand de. [1915] 1959. Course in General Linguistics. London: Peter Owen Ltd. Edited by Charles Bally, and Albert Sechehaye; translated by Wade Baskin.
Schnotz, Wolfgang, and Thorsten Rasch. 2005. “Enabling, Facilitating and Inhibiting Effects of Animations in Multimedia Learning: Why Reduction of Cognitive Load Can Have Negative Effects on Learning.” Educational Technology Research and Development 53 (3): 47–58.
Schriver, Karen A. 1997. Dynamics in Document Design: Creating Texts for Readers. New York: John Wiley and Sons.
Scott, Lyn, and Sally Robinson. 1993. Australian Journey – Environments and Communities. Melbourne: Longman Cheshire.
Severin, Werner J., and James W. Tankard. 2009. Communication Theories: Origins, Methods and Uses in the Mass Media, 5th ed. Addison Wesley Longman.
Sperber, Dan, and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Oxford: Blackwell.
Spillner, Bernd. 1982. “Stilanalyse semiotisch komplexer Texte. Zum Verhaltnis von sprach-licher und bildlicher Information in Werbeanzeigen.” Kodikas/Code. Ars Semeiotica 4/5 (1): 91–106.
Spitzmuller, Jiirgen, and Ingo H. Warnke. 2011. “Discourse as a ‘Linguistic Object’: Methodical and Methodological Delimitations.” Critical Discourse Studies 8 (2): 75–94. .
Stockl, Hartmut. 1997. Textstil und Semiotik englischsprachiger Anzeigenwerbung. Frankfurt am Main: Peter Lang.
Stockl, Hartmut. 2004. Die Sprache im Bild – Das Bild in der Sprache: Zur Verkniipfung von Sprache und Bild im massenmedialen Text. Konzepte – Theorien – Analysemethoden. Berlin: Walter de Gruyter.
Taboada, Maite, and Christopher Habel. 2013. “Rhetorical Relations in Multimodal Documents.” Discourse Studies 15 (1): 65–89.
Taverniers, Miriam. 2011. “The Syntax-semantics Interface in Systemic Functional Grammar: Halliday’s Interpretation of the Hjelmslevian Model of Stratification.” Journal of Pragmatics 43 (4): 1100–1126. .
Tseng, Chiaoi. 2013. Cohesion in Film: Tracking Film Elements. Basingstoke: Palgrave Macmillan.
Tufte, Edward R. 1997. Visual Explanations: Images and Quantities, Evidence and Narrative. Cheshire, Connecticut: Graphics Press.
Unsworth, Len. 2006. “Towards a Metalanguage for Multiliteracies Education: Describing the Meaning-making Resources of Language-image Interaction.” English Teaching: Practice and Critique 5 (1): 55–76.
van Leeuwen, Theo. 1991. “Conjunctive Structure in Documentary Film and Television.” Continuum: Journal of Media and Cultural Studies 5 (1): 76–114.
van Leeuwen, Theo. 2005. Introducing Social Semiotics. London: Routledge.
van Leeuwen, Theo, and Carey Jewitt (eds). 2001. Handbook of Visual Analysis. London: Sage.
Wahlster, Wolfgang, Elisabeth Andre, Wolfgang Finkler, Hans-Jürgen Profitlich, and Thomas Rist. 1993. “Plan-based Integration of Natural Language and Graphics Generation.” Artificial Intelligence 63 (1–2): 387–427.
Wildfeuer, Janina. 2012. “Intersemiosis in Film: Towards a New Organisation of Semiotic Resources in Multimodal Filmic Text.” Multimodal Communication 1 (3): 233–304.
Wildfeuer, Janina. 2014. Film Discourse Interpretation. Towards a New Paradigm for Multi-modal Film Analysis. London, New York: Routledge.
Wurman, Richard Saul. 1996. Information Architects. New York, NY: Watson-Guptill Publications.
Cited by (3)
Cited by three other publications
Messner, Monika
2023.
‘Domani a quest’ora potresti essere qui’: multimodal practices for representing temporality in destination advertising.
Visual Communication
Salama, Amir H.Y.
2023.
Towards a Multimodal Hermeneutic Model: The case of Uber-Blog-mediated advertising discourse order of ‘Saudization’.
Russian Journal of Linguistics 27:4
► pp. 886 ff.
Hiippala, Tuomo
2020.
A multimodal perspective on data visualization. In
Data Visualization in Society,
This list is based on CrossRef data as of 20 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.