Article published In:
Information Design JournalVol. 14:3 (2006) ► pp.207–230
A taxonomy of the discourse relations between words and visuals
A document is mainly composed of words and images, but the complex relationship that binds these two completely different semiotic resources is usually taken for granted as transparent. The simple relations between words and images – ‘anchorage’ and ‘relay’, identified by Barthes almost 30 years ago – are unable to deal with the complexity of their bond, made even more complex by current printing and computer technology. This paper aims to identify the potential relations that bind texts and images together by arguing for a multilevel description of their logico-semantic relationships. The multiple, evaluative and metaphorical functions of the relations will also be discussed. The data generated from the proposed framework can form an empirical corpus for quantitative analysis. Examples from a variety of sources will be used as examples to show how the framework can be operationalized.
Cited by
Cited by 20 other publications
Blunden, Jennifer
2020. Adding ‘something more’ to looking: the interaction of artefact, verbiage and visitor in museum exhibitions. Visual Communication 19:1 ► pp. 45 ff. 
Bouko, Catherine
2020. Emotions through texts and images. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) 30:2 ► pp. 222 ff.
Bragg, Rachel Lanier
2018. Mountains of “Green”: Analyzing the Visual Rhetoric of Mountaintop Removal. Journal of Appalachian Studies 24:2 ► pp. 185 ff.
Diani, Giuliana & Annalisa Sezzi
2020. Scientific websites for children: Nurturing children’s scientific literacy through the conflation of multiple semiotic resources. Journal of Visual Literacy 39:3-4 ► pp. 273 ff.
Feng, Debing
2016. Identifying the participants: reference in television news. Visual Communication 15:2 ► pp. 167 ff.
Grange, Huw & Olaug S. Lian
2022. “Doors Started to Appear:” A Methodological Framework for Analyzing Visuo-Verbal Data Drawing on Roland Barthes’s Classification of Text-Image Relations. International Journal of Qualitative Methods 21 ► pp. 160940692210844 ff.
Haapio, Helena & Stefania Passera
2012. 2012 16th International Conference on Information Visualisation, ► pp. 370 ff.
Hiippala, T.
2013. The interface between rhetoric and layout in multimodal artefacts. Literary and Linguistic Computing 28:3 ► pp. 461 ff.
Hiippala, Tuomo
2012. Reading paths and visual perception in multimodal research, psychology and brain sciences. Journal of Pragmatics 44:3 ► pp. 315 ff.
Jerrentrup, Maja Tabea
2022. Unreadable: the pictorial side of text elements. Media Practice and Education 23:3 ► pp. 236 ff.
Lindenberg, Dennis
2023. Logico-semantic relations between spoken text and slides’ visual elements in student presentations conducted online in the English as a Foreign Language context. Visual Communication
Nguyen, Phung Tien
2016. Book Review: Text and Image: A Critical Introduction to the Visual/Verbal Divide by John A Bateman. Visual Communication 15:3 ► pp. 393 ff.
Passera, Stefania
2015. Beyond the Wall of Text: How Information Design Can Make Contracts User-Friendly. In Design, User Experience, and Usability: Users and Interactions [Lecture Notes in Computer Science, 9187], ► pp. 341 ff.
Reinwein, Joachim & Serge Tassé
2022. Modality Effects Examined by Means of an Online Sentence-Picture Comparison Task. Journal of Psycholinguistic Research 51:3 ► pp. 521 ff.
Ruppel, Marc
2009. Narrative Convergence, Cross-Sited Productions and the Archival Dilemma. Convergence: The International Journal of Research into New Media Technologies 15:3 ► pp. 281 ff.
Stöckl, Hartmut & Jana Pflaeging
2022. Multimodal Coherence Revisited: Notes on the Move From Theory to Data in Annotating Print Advertisements. Frontiers in Communication 7
Taboada, Maite & Christopher Habel
2013. Rhetorical relations in multimodal documents. Discourse Studies 15:1 ► pp. 65 ff.
Vorvilas, George, Thanassis Karalis & Konstantinos Ravalis
2010. Applying Multimodal Discourse Analysis to Learning Objects' User Interface. Contemporary Educational Technology 1:3
Yefymenko, Victoria
2022. Multimodality and cross-modal cohesion in manga. Cognition, Communication, Discourse :24 ► pp. 103 ff.
[no author supplied]
2014. Communicative competence in the professional workplace: an identity-based perspective. In Professional Discourse, ► pp. 91 ff.
This list is based on CrossRef data as of 30 march 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.