Cover not available
Part of
The Pragmatics of Discourse Coherence: Theories and applications
Edited by Helmut Gruber and Gisela Redeker
[Pragmatics & Beyond New Series 254] 2014
► pp. 145–177

Multimodal coherence research and its applications

Although it is nowadays a commonplace that communicative artefacts and communicative behaviours almost always employ a variety of ‘modes’ of information presentation, the issue of how such diverse contributions ‘cohere’ in order to fulfill common communicative purposes is still an area of open debate. This chapter presents a selective overview of the development and current state of the art in approaches to multimodal coherence, focusing particularly on work taking language as its starting point. Overviews and illustrations of the approaches are given as well as a summary of some major challenges to be addressed.

Published online: 26 November 2014
DOI logo
https://doi.org/10.1075/pbns.254.06bat
References
Andre, Elisabeth
1995 Ein planbasierter Ansatz zur Generierung multimedialer Prasentationen, Vol. 108. St. Augustin: Infix.Google Scholar
Asher, Nicholas, and Alex Lascarides
2003 Logics of Conversation. Cambridge: Cambridge ­University Press.Google Scholar
Barthes, Roland
1977 [1964] “The Rhetoric of the Image.” In Image-Music-Text: 32–51. London: Fontana.Google Scholar
Bateman, John A
2007 “Towards a Grande Paradigmatique of Film: Christian Metz Reloaded.” Semiotica 167 (1/4): 13–64.Google Scholar
2008 Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan. DOI logoGoogle Scholar
2013 “Using Multimodal Corpora for Empirical Research.” In The Routledge Handbook of Multimodal Analysis, 2nd ed., ed. by Carey Jewitt, 238–252. London: Routledge.Google Scholar
2014a “The Integration of Multimodal Resources in Documents: Issues, Approaches and Methods.” In Verbal Communication (Handbook of Communication Science), ed. by Louis de Saussure, and Andrea Rocci. Berlin/New York/Amsterdam: Mouton de Gruyter.Google Scholar
2014b Text and Image: A Critical Introduction to the Visual/Verbal Divide. London and New York: Routledge.Google Scholar
Bateman, John A., Judy L. Delin, and Renate Henschel
2007 “Mapping the Multimodal Genres of Traditional and Electronic Newspapers.” In New Directions in the Analysis of Multimodal Discourse, ed. by Terry D. Royce, and Wendy L. Bowcher, 147–172. Lawrence Erlbaum Associates.Google Scholar
Bateman, John A., Thomas Kamps, Jorg Kleinz, and Klaus Reichenberger
2001 “Constructive Text, Diagram and Layout Generation for Information Presentation: The DArtbio System.” Computational Linguistics 27 (3): 409–449. DOI logoGoogle Scholar
Bateman, John A., and Karl-Heinrich Schmidt
2012 Multimodal Film Analysis: How Films Mean (Routledge Studies in Multimodality). London: Routledge.Google Scholar
Bernhardt, Stephen A
1985 “Text Structure and Graphic Design: The Visible Design.” In Systemic Perspectives on Discourse, Vol. 1, ed. by James D. Benson, and William S. Greaves, 18–38. Norwood, New Jersey: Ablex.Google Scholar
Bucher, Hans-Jiirgen
2007 “Textdesign und Multimodalitat. Zur Semantik und Pragmatik medialer Gestaltungsformen.” In Textdesign und Textwirkung in der massenmedialen Kommunikation, ed. by Kersten Sven Roth, and Jürgen Spitzmüller, 49–76. Konstanz: UVK.Google Scholar
Bucher, Hans-Jürgen
2011 “Multimodales Verstehen oder Rezeption als Interaktion. Theoretische und empirische Grundlagen einer systematischen Analyse der Multimodalitat.” In Bildlinguistik. Theorien – Methoden – Fallbeispiele, ed. by Hans-Joachim Diekmannshenke, Michael Klemm, and Hartmut Stockl, 123–156. Berlin: Erich Schmidt.Google Scholar
Cohn, Neil
2013 “Visual Narrative Structure.” Cognitive Science 37 (3): 413–452. DOI logoGoogle Scholar
Danlos, Laurence
2008 “Strong Generative Capacity of RST, SDRT and Discourse Dependency DAGS.” In Constraints in Discourse, ed. by Anton Benz, and Peter Kühnlein, 69–95. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Delin, Judy L., and John A. Bateman
2002 “Describing and Critiquing Multimodal Documents.” Document Design 3 (2): 140–155. DOI logo.Google Scholar
Durand, Jacques
1987 “Rhetorical Figures in the Advertising Image.” In Marketing and Semiotics: New Directions in the Study of Signs for Sale, ed. by Jean Umiker-Sebeok, 295–318. Berlin/New York/Amsterdam: Mouton de Gruyter.Google Scholar
Forceville, Charles J
1996 Pictorial Metaphor in Advertising. London: Routledge. DOI logoGoogle Scholar
2002 “The Identification of Target and Source in Pictorial Metaphors.” Journal of Pragmatics 34 (1): 1–14. DOI logoGoogle Scholar
Gaede, Werner
1981 Vom Wort zum Bild: Kreativ-Methoden der Visualisierung. Munich: ­Langen-Muller/Herbig. 2nd ed 1992.Google Scholar
Gibson, James J
1977 “The Theory of Affordances.” In Perceiving, Acting, and Knowing: Toward and Ecological Psychology, ed. by Robert Shaw, and John Bransford, 62–82. Hillsdale, NJ: Erlbaum.Google Scholar
Glenberg, Arthur M., Alex Cherry Wilkinson, and William Epstein
1982 “The Illusion of Knowing: Failure in the Self-assessment of Comprehension.” Memory and Cognition 10 (6): 597–602. DOI logoGoogle Scholar
Grice, H. Paul
1957 “Meaning.” Philosophical Review 66 (3): 377–388. DOI logoGoogle Scholar
1969 “Utterer’s Meaning and Intentions.” Philosophical Review 68 (2): 147–177. DOI logoGoogle Scholar
Groupe µ
1992 Traité du signe visuel: pour une rhétorique de l’image. Paris: Editions du Seuil.Google Scholar
Halliday, Michael A. K
1978 Language as Social Semiotic. London: Edward Arnold.Google Scholar
1985 An Introduction to Functional Grammar. London: Edward Arnold. (2nd edition 1994; page numbers in the text refer to the second edition).Google Scholar
Halliday, Michael A. K., and Ruqaiya Hasan
1976 Cohesion in English. London: Longman.Google Scholar
Hill, Charles A., and Marguerite Helmers
(eds) 2004 Defining Visual Rhetorics. Mahwah, NJ: Erlbaum.Google Scholar
Hjelmslev, Louis
1953 Prolegomena to a Theory of Language. Bloomington, Indiana: Indiana University Publications in Anthropology and Linguistics. Translated by Francis J. Whitfield.Google Scholar
Holsanova, Jana, and Andreas Nord
2010 “Multimedia Design: Media Structures, Media Principles and Users’ Meaning-making in Newspapers and Net Papers.” In Neue Medien – neue Formate. Ausdifferenzierung und Konvergenz in der Medienkommunikation (Interaktiva. Schriftenreihe des Zentrums fur Medien und Interaktivitat (ZMI), GieBen 10), ed. by Hans-Jurgen Bucher, Thomas Gloning, and Kartin Lehnen, 81–103. Frankfurt/New York: Campus Verlag.Google Scholar
Jewitt, Carey
(ed.) 2009 The Routledge Handbook of Multimodal Analysis. London: Routledge.Google Scholar
Jones, Daniel
1918 An Outline of English Phonetics. Leipzig and Berlin: B. G. Teubner.Google Scholar
Kjeldsen, Jens E
2012 “Pictorial Argumentation in Advertising: Visual Tropes and Figures as a Way of Creating Visual Argumentation.” In Topical Themes in Argumentation Theory: Twenty Exploratory Studies (Argumentation Library 22), ed. by Frans H. van Eemeren, and Bart Garssen, 239–255. Berlin: Springer. DOI logoGoogle Scholar
Kloepfer, Rolf
1977 “Komplementaritat von Sprache und Bild. Am Beispiel von Comic, Karikatur und Reklame.” In Zeichenprozesse. Semiotische Forschung in den Einzelwis-senschaften, ed. by Roland Posner, and Hans-Peter Reinecke, 129–145. Wiesbaden: Athenaum.Google Scholar
Kress, Gunther
2003 Literacy in the New Media Age. London: Routledge. DOI logoGoogle Scholar
2010 Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.Google Scholar
Kress, Gunther, Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis
2000 Multimodal Teaching and Learning. London: Continuum.Google Scholar
Kress, Gunther, and Theo van Leeuwen
2001 Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar
2006 [1996] Reading Images: The Grammar of Visual Design. London and New York: Routledge.Google Scholar
Mann, William C., and Sandra A. Thompson
1988 “Rhetorical Structure Theory: Toward a Functional Theory of Text Organization.” Text 8 (3): 243–281.Google Scholar
Martin, James R
1983 “Conjunction: The Logic of English Text.” In Micro and Macro Connexity of Discourse (Papers in Textlinguistics 45), ed. by Janos S. Petofi, and Emel Sozer, 1–72. Hamburg: Helmut Buske Verlag.Google Scholar
1992 English Text: Systems and Structure. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Martin, James R., and David Rose
2003 Working with Discourse: Meaning beyond the Clause. London and New York: Continuum.Google Scholar
2008 Genre Relations: Mapping Culture. London and New York: Equinox.Google Scholar
Martinec, Radan, and Anthony Salway
2005 “A System for Image-text Relations in New (and Old) Media.” Visual Communication 4 (3): 337–371. DOI logoGoogle Scholar
McQuarrie, Edward F., and David Glen Mick
2003 “Visual and Verbal Rhetorical Figures Under Directed Processing Versus Incidental Exposure to Advertising.” Journal of Consumer Research 29 (4): 579–587. DOI logoGoogle Scholar
Messaris, Paul
1997 Visual Persuasion: The Role of Images in Advertising. Thousand Oaks, CA: Sage.Google Scholar
Meyer, Bonnie J
1975 The Organization of Prose and its Effects on Memory. Amsterdam: North-Holland.Google Scholar
Mirzoeff, Nicholas
1999 An Introduction to Visual Culture. London and New York: Routledge.Google Scholar
Mitchell, W. J. Thomas
1994 Picture Theory: Essays on Verbal and Visual Representation. Chicago: University of Chicago Press.Google Scholar
Moriarty, Sandra E
1996 “Abduction: A Theory of Visual Interpretation.” Communication Theory 6 (2): 167–187. DOI logoGoogle Scholar
Muckenhaupt, Manfred
1986 Text und Bild. Grundfragen der Beschreibung von Text-BildKommunikation aus sprachwissenschaftlicher Sicht (Tubinger Beitrage zur Linguistik). Tubingen: Narr.Google Scholar
Muller, Marion G
2007 “What is Visual Communication? Past and Future of an Emerging Field of Communication Research.” Studies in Communication Sciences 7 (2): 7–34.Google Scholar
Ong, Walter Jackson
1982 Orality and Literacy: The Technologizing of the Word. London: Methuen. DOI logoGoogle Scholar
O’Toole, Michael
1990 “A Systemic-functional Semiotics of Art.” Semiotica 82 (3/4): 185–209. DOI logoGoogle Scholar
2011 [1994] The Language of Displayed Art. Abingdon, Oxon: Routledge.Google Scholar
Phillips, Barbara J., and Edward F. McQuarrie
2004 “Beyond Visual Metaphor: A New Typology of Visual Rhetoric in Advertising.” Marketing Theory 4 (1/2): 113–136. DOI logoGoogle Scholar
Power, Richard, Donia Scott, and Nadjet Bouayad-Agha
2003 “Document Structure.” Computational Linguistics 29 (2): 211–260. DOI logoGoogle Scholar
Rose, Gillian
2012 Visual Methodologies. An Introduction to Researching with Visual Materials, 3rd ed. London/Thousand Oaks/New Delhi: Sage.Google Scholar
Royce, Terry D
2002 “Multimodality in the TESOL Classroom: Exploring Visual-verbal Synergy.” TESOL Quarterly 36 (2): 191–205. DOI logoGoogle Scholar
2007 “Intersemiotic Complementarity: A Framework for Multimodal Discourse Analysis.” In New Directions in the Analysis of Multimodal Discourse, ed. by Terry D. Royce, and Wendy L. Bowcher, 63–110. Lawrence Erlbaum Associates.Google Scholar
Sachs-Hombach, Klaus
(ed.) 2001 Bildhandeln: interdisziplinäre Forschungen zur Pragmatik bildhafter Darstellungsformen (Reihe Bildwissenschaft 3). Magdeburg: Scriptum-Verlag.Google Scholar
Sachs-Hombach, Klaus, and Jorg R. J. Schirra
2006 “Bildstil als rhetorische Kategorie.” Image – Zeitschrift fur interdisziplinäre Bildwissenschaft 3: 175–191.Google Scholar
Saussure, Ferdinand de
[1915] 1959 Course in General Linguistics. London: Peter Owen Ltd. Edited by Charles Bally, and Albert Sechehaye; translated by Wade Baskin.Google Scholar
Schnotz, Wolfgang, and Thorsten Rasch
2005 “Enabling, Facilitating and Inhibiting Effects of Animations in Multimedia Learning: Why Reduction of Cognitive Load Can Have Negative Effects on Learning.” Educational Technology Research and Development 53 (3): 47–58. DOI logoGoogle Scholar
Schriver, Karen A
1997 Dynamics in Document Design: Creating Texts for Readers. New York: John Wiley and Sons.Google Scholar
Scott, Lyn, and Sally Robinson
1993 Australian Journey – Environments and Communities. Melbourne: Longman Cheshire.Google Scholar
Severin, Werner J., and James W. Tankard
2009 Communication Theories: Origins, Methods and Uses in the Mass Media, 5th ed. Addison Wesley Longman.Google Scholar
Sperber, Dan, and Deirdre Wilson
1986 Relevance: Communication and Cognition. Oxford: Blackwell.Google Scholar
Spillner, Bernd
1982 “Stilanalyse semiotisch komplexer Texte. Zum Verhaltnis von sprach-licher und bildlicher Information in Werbeanzeigen.” Kodikas/Code. Ars Semeiotica 4/5 (1): 91–106.Google Scholar
Spitzmuller, Jiirgen, and Ingo H. Warnke
2011 “Discourse as a ‘Linguistic Object’: Methodical and Methodological Delimitations.” Critical Discourse Studies 8 (2): 75–94. DOI logo.Google Scholar
Stockl, Hartmut
1997 Textstil und Semiotik englischsprachiger Anzeigenwerbung. Frankfurt am Main: Peter Lang.Google Scholar
2004 Die Sprache im Bild – Das Bild in der Sprache: Zur Verkniipfung von Sprache und Bild im massenmedialen Text. Konzepte – Theorien – Analysemethoden. Berlin: Walter de Gruyter. DOI logoGoogle Scholar
Taboada, Maite, and Christopher Habel
2013 “Rhetorical Relations in Multimodal Documents.” Discourse Studies 15 (1): 65–89. DOI logoGoogle Scholar
Taverniers, Miriam
2011 “The Syntax-semantics Interface in Systemic Functional Grammar: Halliday’s Interpretation of the Hjelmslevian Model of Stratification.” Journal of Pragmatics 43 (4): 1100–1126. DOI logo.Google Scholar
Tseng, Chiaoi
2013 Cohesion in Film: Tracking Film Elements. Basingstoke: Palgrave Macmillan. DOI logoGoogle Scholar
Tufte, Edward R
1997 Visual Explanations: Images and Quantities, Evidence and Narrative. Cheshire, Connecticut: Graphics Press.Google Scholar
Unsworth, Len
2006 “Towards a Metalanguage for Multiliteracies Education: Describing the Meaning-making Resources of Language-image Interaction.” English Teaching: Practice and Critique 5 (1): 55–76.Google Scholar
van Enschot, Renske, Hans Hoeken, and Margot van Mulken
2008 “Rhetoric in Advertising: Attitudes towards Verbo-pictorial Rhetorical Figures.” Information Design Journal 16 (1): 3545. DOI logoGoogle Scholar
van Leeuwen, Theo
1991 “Conjunctive Structure in Documentary Film and Television.” Continuum: Journal of Media and Cultural Studies 5 (1): 76–114. DOI logoGoogle Scholar
2005 Introducing Social Semiotics. London: Routledge.Google Scholar
van Leeuwen, Theo, and Carey Jewitt
(eds) 2001 Handbook of Visual Analysis. London: Sage.Google Scholar
van Mulken, Margot
2003 “Analyzing Rhetorical Devices in Print Advertisements.” Document Design 4 (2): 114–128. DOI logoGoogle Scholar
Wahlster, Wolfgang, Elisabeth Andre, Wolfgang Finkler, Hans-Jürgen Profitlich, and Thomas Rist
1993 “Plan-based Integration of Natural Language and Graphics Generation.” Artificial Intelligence 63 (1–2): 387–427. DOI logoGoogle Scholar
Wildfeuer, Janina
2012 “Intersemiosis in Film: Towards a New Organisation of Semiotic Resources in Multimodal Filmic Text.” Multimodal Communication 1 (3): 233–304. DOI logoGoogle Scholar
2014 Film Discourse Interpretation. Towards a New Paradigm for Multi-modal Film Analysis. London, New York: Routledge.Google Scholar
Wurman, Richard Saul
1996 Information Architects. New York, NY: Watson-Guptill Publications.Google Scholar
Cited by

Cited by 3 other publications

Hiippala, Tuomo
2020. A multimodal perspective on data visualization. In Data Visualization in Society, DOI logo
Messner, Monika
2023. ‘Domani a quest’ora potresti essere qui’: multimodal practices for representing temporality in destination advertising. Visual Communication DOI logo
Salama, Amir H.Y.
2023. Towards a Multimodal Hermeneutic Model: The case of Uber-Blog-mediated advertising discourse order of ‘Saudization’. Russian Journal of Linguistics 27:4 ► pp. 886 ff. DOI logo

This list is based on CrossRef data as of 12 april 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.