Word and image
Word and image

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

Free course

Word and image

3.6 Image, words: which mode for which job?

I've already mentioned the use of maps in books to help the reader ‘see’ where the action ‘happened’ in order to fully enter into the narrative and take part in it. It can be worthwhile looking at which semiotic mode is used for which parts of a narrative, and why this might be the case.

Nikolajeva and Scott (2000) describe a variety of ways in which words and pictures can be combined:

[I]n symmetrical interaction, words and pictures tell the same story, essentially repeating information in different forms of communication. In enhancing interaction, pictures amplify more fully the meaning of the words, or the words expand the picture so that different information in the two modes of communication produces a more complex dynamic. When enhancing interaction becomes very significant, the dynamic becomes truly complementary. Dependent on the degree of different information presented, a counterpointing dynamic may develop where words and images collaborate to communicate meanings beyond the scope of either one alone. An extreme form of counterpointing is contradictory interaction, where words and pictures seem to be in opposition to one another. This ambiguity challenges the reader to mediate between the words and pictures to establish a true understanding of what is being depicted.

(Nikolajeva and Scott, 2000, pp. 225–6)

The next Reading, by Clare Bradford, looks at combinations of word and image in postcolonial literature. Bradford is a researcher in children's literature at Deakin University, Australia. She is concerned here with representations, both linguistic and visual, of racial politics in children's books, and in this Reading shows how these social tensions surface in texts from New Zealand and Canada. She shows how the images, as well as the verbal text, creatively disrupt expectations, taking traditional narratives and playing around with them.

In the first part of this Reading, Bradford looks at Gavin Bishop's reworking of a traditional rhyme, ‘The House that Jack Built’. In case you are not familiar with the rhyme he uses, it makes extensive use of parallelism, and goes like this:

This is the house that Jack built.

 

This is the malt

That lay in the house that Jack built.

 

This is the rat

That ate the malt

That lay in the house that Jack built.

And so on. The last stanza is:

This is the farmer sowing the corn,

That kept the cock that crowed in the morn,

That waked the priest all shaven and shorn,

That married the man all tattered and torn,

That kissed the maiden all forlorn,

That milked the cow with the crumpled horn,

That tossed the dog,

That worried the cat,

That killed the rat,

That ate the malt

That lay in the house that Jack built.

Bradford then moves on to another text, A Coyote Columbus Story. Coyote is a mystical creature, part-human, part-canine, who occurs in many oral folktales around the world, but is particularly associated with indigenous North American Indians. Coyote is a devious trickster, not unlike Anansi from African folk tales.

Activity 6: Picturebook politics (Reading B)

Click on the link below to read Clare Bradford's discussion of postcolonial politics in children's picturebooks, in Reading B.

Clare Bradford's discussion of postcolonial politics in children's picturebooks [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)]

As you read, look out for:

  • what Bradford points to as evidence of the instability of signs;

  • how the interaction of words and images invites you to interpret these stories;

  • the different connotations of visual signs when stories are re-told in new contexts for new audiences.

Answer

Bradford uses terms from semiotics in this Reading, although her work is probably better located within a literary criticism approach rather than a semiotic or linguistic one. And although she picks out for analysis many semiotic signs in the texts, her focus is not strictly a Formalist one. For Bradford, the sociocultural significance of what she sees in these stories is at least as important as the textual elements themselves.

Bradford shows how analysis of visual representation illuminates the meanings in what appear at first to be simple narratives for children. The texts encode, through their words and pictures, conflicting messages and symbolism which convey hybrid, ambiguous or overtly political messages. Oppositional meanings are implied in both modes which destabilise the interpretation of the whole. Such texts are powerful as they call into question cultural narratives – stories we tell ourselves and each other about who we are – and can create unease. There are links here with the notion of ‘hybridity’ (exemplified in relation to Asian Dub Foundation's rap music and Zadie Smith's novel White Teeth, in Alistair Pennycook's Reading). Maybin and Pearce point to new, creative, hybrid cultural practices emerging from migration and the resulting mixing of traditions. Clare Bradford's texts provide further evidence of such texts emerging from postcolonial contexts.

In ‘The House that Jack Built’, signifiers are interpreted by Bradford as having different connotations in New Zealand than they did in the original British rhyme. How convincing did you find her interpretations? Did you agree, for example, that the copybook page in Figure 2 of the Reader represented the imposition of English on the Maori people and the ascendancy of literacy over orality?

The example from Canada, A Coyote Columbus Story, demonstrates how mockery and cartoon-like parody can be used to undermine established narratives of colonial heroism. Although the language clearly pokes fun at Christopher Columbus, it is in the visuals that the real story takes place and readers are invited to take up a questioning and oppositional viewpoint to the verbal narrative.

‘The House that Jack Built’ and A Coyote Columbus Story are examples of texts in which the images carried a large part of the meaning, and introduced an ideological/politicised spin to the story represented in words. In Nikolajeva and Scott's terms then, we could see these texts as examples of a counterpointing dynamic, where additional meanings are generated by the interaction of words and images. Further along this cline is their contradictory interaction, an example of which is shown in Figure 14. It is taken from Satoshi Kitamura's Lily Takes a Walk (1987), widely cited in the literature on picturebooks (e.g. Arizpe and Styles, 2003; Watson and Styles, 1996; Bromley, 2001). Here words and images struggle hard against each other in the text as they tell their contradictory stories.

Figure 14
Figure 14: Lily takes a walk, Satoshi Kitamura

Lily Takes a Walk tells the story of a little girl, Lily, who takes her dog Nicky for a walk. The words tell the story from Lily's point of view and describe where she goes and what she sees on her very pleasant walk. Nicky's experience is entirely different: he sees monsters around every corner and threatening faces in the forms of lamp posts, pillar boxes, and so on. The catalogue of horrors he experiences is represented entirely in the visual mode. Lily, whose experience is represented verbally, is oblivious.

Lily Takes a Walk is not, of course, an ‘ideological’ text in the way that those shown in Reading B were, although it could be interpreted as rejecting any notion that allowing a young child to walk the city streets unaccompanied by an adult is potentially dangerous. Crucial to a semiotic reading of any text is an understanding of the culture, context and prevailing concerns of the society in which it was produced, and the purpose and significance of the text. The importance of this knowledge for interpretation has been demonstrated in the examples so far in this course, but I want to outline the concept of semiotic domains (Gee, 2003). It provides a useful broadening of semiotics, describing the kinds of knowledge that readers need to have in order to engage in all sorts of social practices, including reading. Gee's work has, generally, an educational focus, but his recognition of the importance of being able to ‘read’ further than the literal meaning of words on a page is what makes it useful here.

Semiotic domains

By a semiotic domain I mean any set of practices that recruits one or more modalities (e.g. oral or written language, images, equations, symbols, sounds, gestures, graphs, artifacts, etc.) to communicate distinctive types of meanings. Here are some examples of semiotic domains: cellular biology, postmodern literary criticism, first-person-shooter video games, high-fashion advertisements, Roman Catholic theology, modernist painting, midwifery, rap music, wine connoisseurship. [… ]

[Take a sentence] about basketball – “The guard dribbled down court, held up two fingers, and passed to the open man” – is a sentence from the semiotic domain of basketball. It might seem odd to call basketball a semiotic domain. However, in basketball, particular words, actions, objects, and images take on distinctive meanings. In basketball, ‘dribble’ does not mean drool; a pick (an action where an offensive player positions him or herself so as to block a defensive player guarding one of his or her teammates) means that some defensive player must quickly switch to guard the now-unguarded offensive player; and the wide circle on each end of the court means that players who shoot from beyond it get three points instead of two if they score a basket.

If you don't know these meanings – cannot read these signs – then you can't ‘read’ (understand) basketball. The matter seems fairly inconsequential when we are talking about basketball. However, it quickly seems more consequential when we are talking about the semiotic domain of some type of science being studied in school. [… ]

In the modern world, print literacy is not enough. People need to be literate in a great variety of different semiotic domains. If these domains involve print, people often need the print bits, of course. However, the vast majority of domains involve semiotic (symbolic, representational) resources besides print and some don't involve print as a resource at all.

(Gee, 2003, pp. 18–19)

As readers or viewers, our recognition of semiotic domains will vary according to our social and cultural background. In Reading B we could, for example, identify the semiotic domains of colonial history and postcolonial resistance to that history, as well as canonical English children's literature and nursery rhymes. These domains are particularly salient in the visual mode.

The next section in this course continues the focus on sociocultural aspects of multimodality, and contains the final Reading (Reading C), on the subject of postmodern literature. I look at some possibilities for analysing multimodal texts in terms of the wider literary and social trend of postmodernism, seeing if and how multimodal texts can be seen to fit within this trend.

E301_1

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has 50 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to University-level study, we offer two introductory routes to our qualifications. You could either choose to start with an Access module, or a module which allows you to count your previous learning towards an Open University qualification. Read our guide on Where to take your learning next for more information.

Not ready for formal University study? Then browse over 1000 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus371