Word and image
Word and image

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

Free course

Word and image

1 Introduction

In this course you will look at ways in which visual aspects add further layers of meaning to printed literary texts and may be considered to contribute to their poetic function.

As you read, look out for:

  • how multimodal elements can be seen to contribute to the literariness of texts by foregrounding certain aspects through deviation and defamiliarisation;

  • the application of the theory of semiotics as conceived by Saussure and developed by Peirce (see Reading A by Bignell), and how the interpretation of signs and codes depends upon the social and cultural knowledge of individual readers;

  • the use of visual images with verbal texts in the creation of fictional reality in both children’s and adult narratives. (Note that the term ‘high modality’ in this context refers to the feeling of truth and reality conveyed and should not be confused with ‘modes’ of communication or ‘multimodality’);

  • the way words and images are used in some children’s picturebooks to create ‘hybrid’ texts that challenge and subvert established ideological notions of history. In Reading B, Bradford analyses two children’s texts to show how they ideologically disrupt dominant postcolonial perceptions;

  • how children’s picturebooks may combine words and images in distinctly postmodern ways to create strange and uncertain meanings. See Lewis’ discussion of the features of postmodernity in Reading C, and the analysis of examples in his reading and in the main chapter;

  • arguments about how and why we make aesthetic judgements about multimodal texts especially in relation to their material form and context.

The use of different forms of communication in a single text is often known as multimodality. I am primarily concerned here with texts utilising image and words, from the genres of poetry and narrative. Texts from other genres, such as advertisements, and texts using other modes of communication, such as movement, are also included where they seem particularly salient and can be seen as literary, in the Formalist sense. It is important to look both at how individual modes can communicate meaning, and at their interaction in the text – how they can be combined to open up new possibilities for interpretation. Such combinations and juxtapositions of modes can be seen to enhance, reinforce or contradict each other, making meaning unstable and challenging the reader's attempts to make sense of the text.

Multimodality is researched in a range of academic disciplines, among them linguistics, art history, information technology, sociology, cultural studies and media studies. Multimodal texts tell us something about linguistic and artistic creativity, but also about the social and cultural spheres in which we live, and about ourselves as producers and consumers of texts. Close analysis opens up questions about how we make judgements about visual texts, about how we value or downplay their status. Creative multimodality also tells us something about how language works – it may force us to focus on a metaphor or a pun, for example. Or it may reveal something about how relationships between people and institutions are represented, or just cause us to reflect on something we had forgotten to notice for a while.

New technology has brought about a surge in multimodal creativity, as anyone with a reasonably up-to-date computer can now produce texts filled with images, sound and movement. But texts using visuals to make meaning are far from new.

Activity 1: The Mouse's Tale

Timing: 0 hours 10 minutes

Start by considering the well-known example of ‘The Mouse's Tale’ from Lewis Carroll's Alice's Adventures in Wonderland (1865), shown in Figure 1. First, what is this? A poem, a narrative, a picture?

Figure 1
Figure 1 ‘The Mouse's Tale’ (Lewis Carroll, 1865, p. 38).

Think about how the physical shape of the printed text influences your interpretation. It might be helpful to imagine it laid out differently. If the text were set out in the usual, left-aligned layout of prose or some other poems, would you ‘read’ it differently? Why?

Answer

‘The Mouse's Tale’ is an example of concrete poetry, where words are arranged on the page in a significant way. When read aloud, its rhyme and metre are easier to access, but the layout initially challenges our attempts to read it as a poem. We consider concrete poetry in more detail in the next section.

‘The Mouse's Tale’ can be considered as literary not only because it comes from a highly valued and canonical work of literature, but because it can be analysed in terms of defamiliarisation and creative deviation in the discussion of Russian Formalism. The poem is eye-catching (‘made strange’) due to its unusual shape and can thus be seen as deviating from ‘normal’ layout conventions of poetry – even though readers of poetry are familiar, of course, with a range of such conventions. The text is also poetic in Formalist terms because it contains an obvious pun – in this case between semiotic modes. The verbal ‘tale’ being told by the mouse is represented by the visual ‘tail’.

Multimodality forces us as readers to focus on ‘the message for its own sake’ (Jakobson, 1960, p. 356) and pay fresh attention to departures from routinised conventions of literature or poetry. Victor Shklovsky believed that the purpose of art was exactly this – to counter our habitualised perceptions, to force us to notice:

Habitualisation devours works, clothes, furniture, one's wife, and the fear of war. ‘If the whole complex lives of many people go on unconsciously, then such lives are as if they had never been.’ And art exists that one may recover the sensation of life; it exists to make one feel things, to make the stone stony.

(Shklovsky, 1965, p. 12)

In an analogous way, then, Carroll's poem can be seen as poetic because it makes the mouse mousy. I will revisit ‘The Mouse's Tale’ later in the course and consider its value as a poem, as well as looking at other creative texts to draw out their multimodal nature. First however, it's important to set out a definition of ‘text’ that is broad enough for the purposes of this course.

A note about text

It will already be clear that to accept non-linguistic textual elements – such as layout and shape in ‘The Mouse's Tale’ – as meaningful and communicative, we will need to work with a definition of ‘text’ which will admit these as a valid focus for analysis. A simple and useful definition of ‘text’ for this purpose is the one provided by August Rubrecht:

A text is …

  • any artifact

  • produced or modified

  • to communicate meaning.

(Rubrecht, 2001)

This definition allows us to consider how meaning can be conveyed via a range of textual elements, as long as these elements are meaningful. It is important to differentiate things we perceive or see, from things we take meaning from:

A piece of driftwood on the beach is not an artifact, just a random object shaped and placed by natural forces. If a beachcomber takes it home, paints a face on it, and hangs it on a wall, it turns into a text communicating the beachcomber's ideas about what is interesting and beautiful. […] A text is purposeful. A line of footprints taking the left fork at a junction on a snowy trail is not a text. An arrow drawn in the snow and pointing left is.

A beautiful sunset is not a text. A painting or photograph of one is.

(Rubrecht, 2001)

Later in the course I will return to problematise this definition of text, as not all texts which make a claim to be meaningful, or ‘art’, are necessarily interpreted as such. But I turn now to the field of semiotics and consider what it can offer us when we start to analyse multimodal texts.

E301_1

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has 50 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to University-level study, we offer two introductory routes to our qualifications. You could either choose to start with an Access module, or a module which allows you to count your previous learning towards an Open University qualification. Read our guide on Where to take your learning next for more information.

Not ready for formal University study? Then browse over 1000 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus371