4.1 What is multimodality?

Multimodal texts are defined as texts which communicate their message using more than one semiotic mode, or channel of communication. Examples are magazine articles which use words and pictures, or websites which contain audio clips alongside the words, or film which uses words, music, sound effects and moving images. As soon as you start to take this idea seriously, you realise that, in a sense, all human communication is intrinsically multimodal. We rarely read, write, receive or send messages to one other in a single mode. In spoken language, for example, words are often accompanied by facial gestures, hand movements and so on. This paralanguage is communicative, and is hard to separate from words as we engage in the process of interpretation. An email message may be thought of as written text, but it is accessed via a series of visual icons on a computer, is read in the context of a website or desktop screen, and may contain iconic representations of the sender’s mood such as emoticons (or ‘smilies’), or unusual punctuation added by the sender for emphasis, etc. Email communication is often quite ‘speech-like’, too, so can be said to contain elements of spoken language (more on this later).

Even a piece of solid written text with no pictures can be said to convey messages from visual modes. We may be influenced by the typeface of the text: it may seem formal or informal, childlike (such as large lower case letters), or carry other connotations which support or undermine the apparent message of the words. The layout of the page can also be interpreted as conveying meaning: think about your impression of a text set out in columns like a newspaper article, or double spaced like a first draft of a report, or densely packed like a dictionary entry. Advertisements exploit this extra layer of meaning as a matter of routine. Our knowledge and experience of other texts is brought to bear and colours what we take from any new text, even if this process is not a conscious one. Some of the principle communicative components of text are:

• written or spoken language
• intonation
• images (photographs, diagrams, drawings), and aspects of images such as colour, sharpness of focus, spatial composition, etc. Also other visuals such as logos, corporate letterheads, shop or road signs
• gestures
• facial movements
• action (movement in film, for example).

The study of multimodality involves looking at these components and the ways they communicate meaning, both separately and in combination. Components of multimodal texts often take on new meanings, or connotations, when they interact in a complete text. Newspaper headlines, for example, may be placed next to a striking photograph which reinforces the story, or even undermines it. Black and white film may be deliberately used to convey a sense of the past. Teachers may gesture to the class in order to reinforce what they are saying.

In this section you will learn about and try out types of analysis which aim to integrate visual and physical aspects of communication with analysis of spoken and written language. Multimodal approaches to the study of different forms of communication – the visual aspects of communication (in art/cultural studies), the physical (non-verbal communication in psychology) for example – have a long history of course. However, the study of communication within the tradition of Western linguistics has tended to focus predominantly on verbal aspects of communication. A call has come in recent times to integrate visual and physical aspects of communication into analyses of spoken and written language. This arises out of two principal concerns:

• to acknowledge that verbal language always takes place alongside a whole array of other representational (semiotic) resources (the word ‘semiotic’ or ‘semiosis’, meaning ‘the meaning of signs’, is often used in these approaches to signal an interest in language as well as other sign systems)
• that global communication practices at the beginning of the twenty-first century, notably exemplified in internet usage, are increasingly more obviously multimodal, displacing the verbal as the central mode of communication.

It will already be clear that while multimodality is pretty much the norm in most forms of communication, it is not new, nor has it suddenly become relevant to education. Later readings in this section suggest, however, that what is new is an urgent need for a serious consideration of modes other than speaking and writing in the classroom. This need derives from children’s – and adults’– increasing need to engage with new forms and uses of technology.

Computers, then, are rapidly adding new multimodal texts to our daily communicative practices. In some communities, though, multimodal communication is routine and has existed for centuries. The next reading introduces you to an example of this from Brazil.