Cover not available
Part of
The Pragmatics of Discourse Coherence: Theories and applications
Edited by Helmut Gruber and Gisela Redeker
[Pragmatics & Beyond New Series 254] 2014
► pp. 145–177

Multimodal coherence research and its applications

Although it is nowadays a commonplace that communicative artefacts and communicative behaviours almost always employ a variety of ‘modes’ of information presentation, the issue of how such diverse contributions ‘cohere’ in order to fulfill common communicative purposes is still an area of open debate. This chapter presents a selective overview of the development and current state of the art in approaches to multimodal coherence, focusing particularly on work taking language as its starting point. Overviews and illustrations of the approaches are given as well as a summary of some major challenges to be addressed.
Published online: 26 November 2014
DOI logo
https://doi.org/10.1075/pbns.254.06bat
References (89)
Andre, Elisabeth. 1995. Ein planbasierter Ansatz zur Generierung multimedialer Prasentationen, Vol. 108. St. Augustin: Infix.Google Scholar
Asher, Nicholas, and Alex Lascarides. 2003. Logics of Conversation. Cambridge: Cambridge ­University Press.Google Scholar
Barthes, Roland. 1977 [1964]. “The Rhetoric of the Image.” In Image-Music-Text: 32–51. London: Fontana.Google Scholar
Bateman, John A. 2007. “Towards a Grande Paradigmatique of Film: Christian Metz Reloaded.” Semiotica 167 (1/4): 13–64.Google Scholar
. 2008. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan. DOI logoGoogle Scholar
. 2013. “Using Multimodal Corpora for Empirical Research.” In The Routledge Handbook of Multimodal Analysis, 2nd ed., ed. by Carey Jewitt, 238–252. London: Routledge.Google Scholar
. 2014a. “The Integration of Multimodal Resources in Documents: Issues, Approaches and Methods.” In Verbal Communication (Handbook of Communication Science), ed. by Louis de Saussure, and Andrea Rocci. Berlin/New York/Amsterdam: Mouton de Gruyter.Google Scholar
. 2014b. Text and Image: A Critical Introduction to the Visual/Verbal Divide. London and New York: Routledge.Google Scholar
Bateman, John A., Judy L. Delin, and Renate Henschel. 2007. “Mapping the Multimodal Genres of Traditional and Electronic Newspapers.” In New Directions in the Analysis of Multimodal Discourse, ed. by Terry D. Royce, and Wendy L. Bowcher, 147–172. Lawrence Erlbaum Associates.Google Scholar
Bateman, John A., Thomas Kamps, Jorg Kleinz, and Klaus Reichenberger. 2001. “Constructive Text, Diagram and Layout Generation for Information Presentation: The DArtbio System.” Computational Linguistics 27 (3): 409–449. DOI logoGoogle Scholar
Bateman, John A., and Karl-Heinrich Schmidt. 2012. Multimodal Film Analysis: How Films Mean (Routledge Studies in Multimodality). London: Routledge.Google Scholar
Bernhardt, Stephen A. 1985. “Text Structure and Graphic Design: The Visible Design.” In Systemic Perspectives on Discourse, Vol. 1, ed. by James D. Benson, and William S. Greaves, 18–38. Norwood, New Jersey: Ablex.Google Scholar
Bucher, Hans-Jiirgen. 2007. “Textdesign und Multimodalitat. Zur Semantik und Pragmatik medialer Gestaltungsformen.” In Textdesign und Textwirkung in der massenmedialen Kommunikation, ed. by Kersten Sven Roth, and Jürgen Spitzmüller, 49–76. Konstanz: UVK.Google Scholar
Bucher, Hans-Jürgen. 2011. “Multimodales Verstehen oder Rezeption als Interaktion. Theoretische und empirische Grundlagen einer systematischen Analyse der Multimodalitat.” In Bildlinguistik. Theorien – Methoden – Fallbeispiele, ed. by Hans-Joachim Diekmannshenke, Michael Klemm, and Hartmut Stockl, 123–156. Berlin: Erich Schmidt.Google Scholar
Cohn, Neil. 2013. “Visual Narrative Structure.” Cognitive Science 37 (3): 413–452. DOI logoGoogle Scholar
Danlos, Laurence. 2008. “Strong Generative Capacity of RST, SDRT and Discourse Dependency DAGS.” In Constraints in Discourse, ed. by Anton Benz, and Peter Kühnlein, 69–95. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Delin, Judy L., and John A. Bateman. 2002. “Describing and Critiquing Multimodal Documents.” Document Design 3 (2): 140–155. DOI logo.Google Scholar
Durand, Jacques. 1987. “Rhetorical Figures in the Advertising Image.” In Marketing and Semiotics: New Directions in the Study of Signs for Sale, ed. by Jean Umiker-Sebeok, 295–318. Berlin/New York/Amsterdam: Mouton de Gruyter.Google Scholar
Forceville, Charles J. 1996. Pictorial Metaphor in Advertising. London: Routledge. DOI logoGoogle Scholar
. 2002. “The Identification of Target and Source in Pictorial Metaphors.” Journal of Pragmatics 34 (1): 1–14. DOI logoGoogle Scholar
Gaede, Werner. 1981. Vom Wort zum Bild: Kreativ-Methoden der Visualisierung. Munich: ­Langen-Muller/Herbig. 2nd ed., 1992.Google Scholar
Gibson, James J. 1977. “The Theory of Affordances.” In Perceiving, Acting, and Knowing: Toward and Ecological Psychology, ed. by Robert Shaw, and John Bransford, 62–82. Hillsdale, NJ: Erlbaum.Google Scholar
Glenberg, Arthur M., Alex Cherry Wilkinson, and William Epstein. 1982. “The Illusion of Knowing: Failure in the Self-assessment of Comprehension.” Memory and Cognition 10 (6): 597–602. DOI logoGoogle Scholar
Grice, H. Paul. 1957. “Meaning.” Philosophical Review 66 (3): 377–388. DOI logoGoogle Scholar
. 1969. “Utterer’s Meaning and Intentions.” Philosophical Review 68 (2): 147–177. DOI logoGoogle Scholar
Groupe µ. 1992. Traité du signe visuel: pour une rhétorique de l’image. Paris: Editions du Seuil.Google Scholar
Halliday, Michael A. K. 1978. Language as Social Semiotic. London: Edward Arnold.Google Scholar
. 1985. An Introduction to Functional Grammar. London: Edward Arnold. (2nd edition 1994; page numbers in the text refer to the second edition).Google Scholar
Halliday, Michael A. K., and Ruqaiya Hasan. 1976. Cohesion in English. London: Longman.Google Scholar
Hill, Charles A., and Marguerite Helmers (eds). 2004. Defining Visual Rhetorics. Mahwah, NJ: Erlbaum.Google Scholar
Hjelmslev, Louis. 1953. Prolegomena to a Theory of Language. Bloomington, Indiana: Indiana University Publications in Anthropology and Linguistics. Translated by Francis J. Whitfield.Google Scholar
Holsanova, Jana, and Andreas Nord. 2010. “Multimedia Design: Media Structures, Media Principles and Users’ Meaning-making in Newspapers and Net Papers.” In Neue Medien – neue Formate. Ausdifferenzierung und Konvergenz in der Medienkommunikation (Interaktiva. Schriftenreihe des Zentrums fur Medien und Interaktivitat (ZMI), GieBen 10), ed. by Hans-Jurgen Bucher, Thomas Gloning, and Kartin Lehnen, 81–103. Frankfurt/New York: Campus Verlag.Google Scholar
Jewitt, Carey (ed.). 2009. The Routledge Handbook of Multimodal Analysis. London: Routledge.Google Scholar
Jones, Daniel. 1918. An Outline of English Phonetics. Leipzig and Berlin: B. G. Teubner.Google Scholar
Kjeldsen, Jens E. 2012. “Pictorial Argumentation in Advertising: Visual Tropes and Figures as a Way of Creating Visual Argumentation.” In Topical Themes in Argumentation Theory: Twenty Exploratory Studies (Argumentation Library 22), ed. by Frans H. van Eemeren, and Bart Garssen, 239–255. Berlin: Springer. DOI logoGoogle Scholar
Kloepfer, Rolf. 1977. “Komplementaritat von Sprache und Bild. Am Beispiel von Comic, Karikatur und Reklame.” In Zeichenprozesse. Semiotische Forschung in den Einzelwis-senschaften, ed. by Roland Posner, and Hans-Peter Reinecke, 129–145. Wiesbaden: Athenaum.Google Scholar
Kress, Gunther. 2003. Literacy in the New Media Age. London: Routledge. DOI logoGoogle Scholar
. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.Google Scholar
Kress, Gunther, Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis. 2000. Multimodal Teaching and Learning. London: Continuum.Google Scholar
Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar
. 2006 [1996]. Reading Images: The Grammar of Visual Design. London and New York: Routledge.Google Scholar
Mann, William C., and Sandra A. Thompson. 1988. “Rhetorical Structure Theory: Toward a Functional Theory of Text Organization.” Text 8 (3): 243–281.Google Scholar
Martin, James R. 1983. “Conjunction: The Logic of English Text.” In Micro and Macro Connexity of Discourse (Papers in Textlinguistics 45), ed. by Janos S. Petofi, and Emel Sozer, 1–72. Hamburg: Helmut Buske Verlag.Google Scholar
. 1992. English Text: Systems and Structure. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Martin, James R., and David Rose. 2003. Working with Discourse: Meaning beyond the Clause. London and New York: Continuum.Google Scholar
. 2008. Genre Relations: Mapping Culture. London and New York: Equinox.Google Scholar
Martinec, Radan, and Anthony Salway. 2005. “A System for Image-text Relations in New (and Old) Media.” Visual Communication 4 (3): 337–371. DOI logoGoogle Scholar
McQuarrie, Edward F., and David Glen Mick. 2003. “Visual and Verbal Rhetorical Figures Under Directed Processing Versus Incidental Exposure to Advertising.” Journal of Consumer Research 29 (4): 579–587. DOI logoGoogle Scholar
Messaris, Paul. 1997. Visual Persuasion: The Role of Images in Advertising. Thousand Oaks, CA: Sage.Google Scholar
Meyer, Bonnie J. 1975. The Organization of Prose and its Effects on Memory. Amsterdam: North-Holland.Google Scholar
Mirzoeff, Nicholas. 1999. An Introduction to Visual Culture. London and New York: Routledge.Google Scholar
Mitchell, W. J. Thomas. 1994. Picture Theory: Essays on Verbal and Visual Representation. Chicago: University of Chicago Press.Google Scholar
Moriarty, Sandra E. 1996. “Abduction: A Theory of Visual Interpretation.” Communication Theory 6 (2): 167–187. DOI logoGoogle Scholar
Muckenhaupt, Manfred. 1986. Text und Bild. Grundfragen der Beschreibung von Text-BildKommunikation aus sprachwissenschaftlicher Sicht (Tubinger Beitrage zur Linguistik). Tubingen: Narr.Google Scholar
Muller, Marion G. 2007. “What is Visual Communication? Past and Future of an Emerging Field of Communication Research.” Studies in Communication Sciences 7 (2): 7–34.Google Scholar
Ong, Walter Jackson. 1982. Orality and Literacy: The Technologizing of the Word. London: Methuen. DOI logoGoogle Scholar
O’Toole, Michael. 1990. “A Systemic-functional Semiotics of Art.” Semiotica 82 (3/4): 185–209. DOI logoGoogle Scholar
. 2011 [1994]. The Language of Displayed Art. Abingdon, Oxon: Routledge.Google Scholar
Phillips, Barbara J., and Edward F. McQuarrie. 2004. “Beyond Visual Metaphor: A New Typology of Visual Rhetoric in Advertising.” Marketing Theory 4 (1/2): 113–136. DOI logoGoogle Scholar
Power, Richard, Donia Scott, and Nadjet Bouayad-Agha. 2003. “Document Structure.” Computational Linguistics 29 (2): 211–260. DOI logoGoogle Scholar
Rose, Gillian. 2012. Visual Methodologies. An Introduction to Researching with Visual Materials, 3rd ed. London/Thousand Oaks/New Delhi: Sage.Google Scholar
Royce, Terry D. 2002. “Multimodality in the TESOL Classroom: Exploring Visual-verbal Synergy.” TESOL Quarterly 36 (2): 191–205. DOI logoGoogle Scholar
. 2007. “Intersemiotic Complementarity: A Framework for Multimodal Discourse Analysis.” In New Directions in the Analysis of Multimodal Discourse, ed. by Terry D. Royce, and Wendy L. Bowcher, 63–110. Lawrence Erlbaum Associates.Google Scholar
Sachs-Hombach, Klaus (ed.). 2001. Bildhandeln: interdisziplinäre Forschungen zur Pragmatik bildhafter Darstellungsformen (Reihe Bildwissenschaft 3). Magdeburg: Scriptum-Verlag.Google Scholar
Sachs-Hombach, Klaus, and Jorg R. J. Schirra. 2006. “Bildstil als rhetorische Kategorie.” Image – Zeitschrift fur interdisziplinäre Bildwissenschaft 3: 175–191.Google Scholar
Saussure, Ferdinand de. [1915] 1959. Course in General Linguistics. London: Peter Owen Ltd. Edited by Charles Bally, and Albert Sechehaye; translated by Wade Baskin.Google Scholar
Schnotz, Wolfgang, and Thorsten Rasch. 2005. “Enabling, Facilitating and Inhibiting Effects of Animations in Multimedia Learning: Why Reduction of Cognitive Load Can Have Negative Effects on Learning.” Educational Technology Research and Development 53 (3): 47–58. DOI logoGoogle Scholar
Schriver, Karen A. 1997. Dynamics in Document Design: Creating Texts for Readers. New York: John Wiley and Sons.Google Scholar
Scott, Lyn, and Sally Robinson. 1993. Australian Journey – Environments and Communities. Melbourne: Longman Cheshire.Google Scholar
Severin, Werner J., and James W. Tankard. 2009. Communication Theories: Origins, Methods and Uses in the Mass Media, 5th ed. Addison Wesley Longman.Google Scholar
Sperber, Dan, and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Oxford: Blackwell.Google Scholar
Spillner, Bernd. 1982. “Stilanalyse semiotisch komplexer Texte. Zum Verhaltnis von sprach-licher und bildlicher Information in Werbeanzeigen.” Kodikas/Code. Ars Semeiotica 4/5 (1): 91–106.Google Scholar
Spitzmuller, Jiirgen, and Ingo H. Warnke. 2011. “Discourse as a ‘Linguistic Object’: Methodical and Methodological Delimitations.” Critical Discourse Studies 8 (2): 75–94. DOI logo.Google Scholar
Stockl, Hartmut. 1997. Textstil und Semiotik englischsprachiger Anzeigenwerbung. Frankfurt am Main: Peter Lang.Google Scholar
. 2004. Die Sprache im Bild – Das Bild in der Sprache: Zur Verkniipfung von Sprache und Bild im massenmedialen Text. Konzepte – Theorien – Analysemethoden. Berlin: Walter de Gruyter. DOI logoGoogle Scholar
Taboada, Maite, and Christopher Habel. 2013. “Rhetorical Relations in Multimodal Documents.” Discourse Studies 15 (1): 65–89. DOI logoGoogle Scholar
Taverniers, Miriam. 2011. “The Syntax-semantics Interface in Systemic Functional Grammar: Halliday’s Interpretation of the Hjelmslevian Model of Stratification.” Journal of Pragmatics 43 (4): 1100–1126. DOI logo.Google Scholar
Tseng, Chiaoi. 2013. Cohesion in Film: Tracking Film Elements. Basingstoke: Palgrave Macmillan. DOI logoGoogle Scholar
Tufte, Edward R. 1997. Visual Explanations: Images and Quantities, Evidence and Narrative. Cheshire, Connecticut: Graphics Press.Google Scholar
Unsworth, Len. 2006. “Towards a Metalanguage for Multiliteracies Education: Describing the Meaning-making Resources of Language-image Interaction.” English Teaching: Practice and Critique 5 (1): 55–76.Google Scholar
van Enschot, Renske, Hans Hoeken, and Margot van Mulken. 2008. “Rhetoric in Advertising: Attitudes towards Verbo-pictorial Rhetorical Figures.” Information Design Journal 16 (1): 3545. DOI logoGoogle Scholar
van Leeuwen, Theo. 1991. “Conjunctive Structure in Documentary Film and Television.” Continuum: Journal of Media and Cultural Studies 5 (1): 76–114. DOI logoGoogle Scholar
. 2005. Introducing Social Semiotics. London: Routledge.Google Scholar
van Leeuwen, Theo, and Carey Jewitt (eds). 2001. Handbook of Visual Analysis. London: Sage.Google Scholar
van Mulken, Margot. 2003. “Analyzing Rhetorical Devices in Print Advertisements.” Document Design 4 (2): 114–128. DOI logoGoogle Scholar
Wahlster, Wolfgang, Elisabeth Andre, Wolfgang Finkler, Hans-Jürgen Profitlich, and Thomas Rist. 1993. “Plan-based Integration of Natural Language and Graphics Generation.” Artificial Intelligence 63 (1–2): 387–427. DOI logoGoogle Scholar
Wildfeuer, Janina. 2012. “Intersemiosis in Film: Towards a New Organisation of Semiotic Resources in Multimodal Filmic Text.” Multimodal Communication 1 (3): 233–304. DOI logoGoogle Scholar
. 2014. Film Discourse Interpretation. Towards a New Paradigm for Multi-modal Film Analysis. London, New York: Routledge.Google Scholar
Wurman, Richard Saul. 1996. Information Architects. New York, NY: Watson-Guptill Publications.Google Scholar
Cited by (3)

Cited by three other publications

Messner, Monika
2023. ‘Domani a quest’ora potresti essere qui’: multimodal practices for representing temporality in destination advertising. Visual Communication DOI logo
Salama, Amir H.Y.
2023. Towards a Multimodal Hermeneutic Model: The case of Uber-Blog-mediated advertising discourse order of ‘Saudization’. Russian Journal of Linguistics 27:4 ► pp. 886 ff. DOI logo
Hiippala, Tuomo
2020. A multimodal perspective on data visualization. In Data Visualization in Society, DOI logo

This list is based on CrossRef data as of 15 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.