7 How to build a quiz for open online assessment

Introduction

As a continuation of the previous session on assessment, this session looks more specifically at using quizzes to assess learners on your course.

In this session you will learn about different types of quiz questions and get some practical guidance on how to build quizzes into your course.

Listen to the following audio recording in which Matt Driver introduces the session.

Download this audio clip.Audio player: 7 How to build a quiz for open online assessment

Show transcript|Hide transcript

Transcript: 7 How to build a quiz for open online assessment

Matt Driver

In this session you will discover some of the principles and good practice involved in building quiz questions for online assessment purposes. You will learn about question types and behaviour as well as the value of providing useful feedback to help to embed learning. You will explore how to devise a quiz which engages and retains the interest of learners without asking them to write essay type answers. Essay questions are usually impossible to mark in an open online course which has no teachers.  

You’ll have the chance to try creating a pedagogically robust quiz question which works well in an online setting. There’s also a handy checklist for quiz creation.

End transcript: 7 How to build a quiz for open online assessment

Download

7 How to build a quiz for open online assessment

7.1 Quiz question types and behaviour

Figure 1 Pop quiz.

Show description|Hide description

The image shows a young girl sitting at a school desk. The caption reads: ‘She says this is a “pop quiz”. I’m hoping the answer is root beer’.

Figure 1 Pop quiz.

Before you build a quiz using the online quiz tool, it is important to work out what the quiz questions will be and to test them out with colleagues.

The primary purpose of quiz questions in most open online courses is for learners to check their understanding of the course materials and concepts rather than for formal accreditation. It can be quite difficult working out how to ask robust and pedagogically useful questions which can be answered without asking learners to provide an essay response – it very much depends on the content as some materials are more suited to non-essay type questions than others. It is usually possible to find a way to write a quiz for most topics but not possible to use quizzes for all types of knowledge.

It is a good idea to select one or two question types and get used to how they work before you extend the range of question types you use, especially if you are new to creating quizzes.

7.1.1 Question types

Figure 2 Screenshot of a completed practice quiz for Succeed with Maths 1 (OpenLearn).

Show description|Hide description

This image depicts a completed quiz ‘review and submit’ screen.  On the left there is a ‘back to course’ button, just below it is a Heading labelled ‘Questions’ and below this are numbers 1 to 5 in a row. Below each number is either a tick indicating that all questions have been completed. On the right is a heading of ‘Review and Submit: Week 1 practice quiz’ with some text below it which says ‘You have reached the end of the quiz but you can still go back to any questions that are marked either ‘not completed’ or ‘tries remaining’ by clicking on the question number. If you have finished the quiz and are ready to submit your answers click on the ‘submit’ button to see how well you have done.’ Below this is a table with three headings of Question, Status and Marks. The questions 1 to 5 are listed below ‘question’, the status of ‘correct’ is listed for each question and the mark of ‘3.00’ is listed below ‘marks’. Below the table is are two links, one is labelled ‘Previous: question 5 summary’ and the other is labelled ‘Submit’.

Figure 2 Screenshot of a completed practice quiz for Succeed with Maths 1 (OpenLearn).

You can use quizzes to ask learners to do the following:

• categorise information and ideas
• compare and contrast options
• answer questions which require more than one answer
• analyse texts or scenarios (such as a case study)
• evaluate different options or scenarios
• choose from a selection of potential answers (provide at least 4 choices)

Each of these different assessment options can be supported in questions which do not require the learner to write a paragraph or essay and can be designed to draw quite complex thought processes and responses from learners.

Question types available in Moodle quiz include the following:

• Combined (more than one type of question combined into a question)
• drag and drop into text
• drag and drop onto image
• drag and drop markers
• matching
• multiple choice
• numerical
• OU multiple response
• pattern match
• select missing words
• short answer (this works as a very restricted type of pattern match)
• true/false
• variable numeric
• variable numeric set
• variable numeric set with units

Questions either require a selected response in which all possible answers are available for the learner to choose from, or a constructed response in which the learner must work out their own answer, though the author would ensure there are a very small number of potentially correct answers against which the learner response can be compared.  

Numeric and selected response questions are suitable for using in an open online course for supporting learning and effective assessment. Examples of selected response questions include:

• multiple choice (select one from many)
• multiple response (select several from many)
• true/false (select one from two)
• drag and drop into text, select missing word, drag and drop onto images, matching (select one from many several times)
• drag and drop markers (onto an image or a piece of text)

Please note that drag and drop quiz questions do have accessibility issues for some users – older browsers do not support drag and drop questions so well as new ones (many learners will be using older browsers) and some tablets also have problems with drag and drop. Therefore you need to consider how much you are going to use this question type in your quiz and if there is a pass grade, whether there are sufficient questions of other types to enable a learner to pass if they cannot use the drag and drop questions successfully.

Figure 3 Selecting Moodle quiz question types.

Show description|Hide description

This is a screenshot of the pop-up box for selecting the quiz question type. It is headed ‘Choose a question type to add’. On the left is a list with radio buttons for each question type, one of these ‘OU multiple response’ has been selected. On the right is the definition of this question type, which reads ‘A multiple-choice, multiple-response question type with particular scoring rules’.

Figure 3 Selecting Moodle quiz question types.

Only some constructed response questions are suitable for an open online course where there is no tutor to mark the responses, for example essay questions are impossible to mark without a tutor.

Examples of constructed response questions which can be used (with great care in the case of short answer and pattern match) include:

• Pattern match and short answer (free text matching is tricky, so although the pattern match question has sophisticated response matching functionality, using this requires careful and thorough authors)
• Numeric, variable numeric and variable numeric sets (will match numeric responses)

7.1.2 Question behaviour

In Moodle quiz, you can set the question behaviours to determine whether and when users will receive feedback. You can use quiz question feedback as part of the pedagogy of the course: in practice quizzes it is valuable as a teaching aid to promote deeper thinking about the topic and help learners identify the progress they are making. In graded quizzes it is a bonus to be able to offer feedback to their responses which so often is not available for students taking formal examinations.

Some question behaviours can be combined with Certainty Based Marking (CBM) which asks learners the answer to an objective question and how certain they are that their answer is correct. The mark scheme will reward accurate reporting of certainty and good discrimination between more and less reliable answers to encourage reflection about justification and soundness of relevant knowledge and skills, probing weaknesses more deeply. According to Gardner-Medwin & Curtin, ‘it is easily implemented with existing test material, popular with students, grounded firmly in information theory and proven to enhance the quality of exam data’.

The four types of question behaviours are:

1. deferred feedback with and without Certainty Based Marking (CBM)
2. interactive with multiple tries
3. immediate feedback with and without Certainty Based Marking (CBM)
4. manually graded (not normally suitable for an open online course without a tutor)

Definitions for these question behaviours are:

• deferred feedback – a learner answers all the questions before any feedback is provided. This kind of question behaviour often encourages learners to concentrate on their overall mark than on any feedback specific questions which you might provide.
• interactive with multiple tries – provides instant feedback, hints and multiple tries
• immediate feedback – provides instant feedback with no hints or multiple tries
• manually graded – for essay type questions especially, though can be used for all other question types.

Interactive with multiple tries is the question behaviour most likely to encourage deeper reflective learning.

You also need to think about how accessible the questions are for learners who are using assistive technology or devices which don’t work so well with drag and drop type questions.

Figure 4 A drop-down box in sentence quiz question.

Show description|Hide description

In the grey box on the left is the following text: Question 3, tries remaining 1, marked out of 3.00. Flag question and edit question link (because this is a course creator’s view of the question). In the blue box on the right is the following question text: If you licensed your material with non-derivatives (ND) licence could people add their own examples to it? (The paragraph of text has dropdown boxes showing the word ‘Choose’ in the spaces in the sentence where the words need to be filled in the sentence). Choose, the Choose licence means that Choose can create a new version of your work or remix it.  Under a Choose licence they wouldn’t be able to add their own examples to your materials as this would constitute Choose it.

Figure 4 A drop-down box in sentence quiz question.

The following are some examples of different forms of questions (and the interaction that occur in them) which work well in open online courses:

1. A crossword using drag and drop into text – the learner is asked to drag and drop letters into the correct spaces. This is a useful way to reinforce elements of concepts, though it can be an irritating format of question for people who prefer to type a letter rather than drag it.
2. Crossword using drag and drop onto an image – the crossword is an image and the answers (numbered in a list) can be dragged onto the image.
3. Grid for Sudoku using drag and drop into text – this is a good question which enables learners to gain confidence in manipulating numbers using a familiar puzzle format.
4. Using combined question type to allow input of different numbers to respond to a question – this is a useful question type for reinforcing a teaching point and only requires learners to type their answer into the box provided.
5. Drag and drop onto image to allow for making up whole number and fraction – using the functionality of drag and drop onto an image to rewrite a fraction as a mixed number provides a visual way of enabling learners to see if they have grasped how to convert fractions into a mixed number including a whole number and fraction.
6. Combined question type using select from drop-down list and numeric – for answering a scenario type question which requires several options to be selected (from dropdown lists) and written (in a numeric text box).
7. Combined question type using select from drop-down list and multiple response – can be used to encourage learners to reflect on what they’ve learned and check their understanding.
8. Simple pattern match – the learner types a word into a text-box which is checked against the correct answer in the system. This can be used in a missing word in a sentence question or in answer to a question, to test recollection of something taught in the course or understanding of a concept.
9. Complex pattern match – in which an image or diagram can be included in the question and the learner types the short answer into the text box.
10. Drag and drop using markers – can be used in a question when you want learners to highlight some text to select their answer (such as annotating a poem). It can be quite complicated to set up.
11. Combined question type using multiple response and numeric – the questions are based on an image which is inserted above the questions, some of which require selection from a list and some of which require a number to be typed into a box.
12. Combined question type using select from drop-down list and numeric – allows question types to be combined within one question so learners can provide answers in more than one way, such as selecting a particular word or phrase from a dropdown list and typing a numerical answer. The questions can be based on an image, so this is quite a good question type to use for maps and diagrams.
13. Variable numeric question type – used for generating numerous mathematical questions based on formulae. For example it is possible to create duplicate questions by selecting different values as the basis of the question. The question type can be re-edited to provide variations of the basic question.
14. Drag and drop onto an image – this can be used for mathematical questions as well as to develop pictorial or graphical quizzes on a range of subjects, such as asking learners to complete charts or pie graphs.

Activity 1

This activity will give you some practice of writing a quiz question and the process involved. It does not ask you to try building the quiz question online.

1. Draw up different ways of how to ask a quiz question on a topic of your choice, using the interactive with multiple tries feedback option.
2. Run through the list of question type examples and select at least 2 different ways you could ask your question.  
3. Write out the questions in the question type formats you have chosen. Write out some hint text for interactive with multiple tries.  
4. Test your question by inviting a couple of different colleagues to review it and get their feedback on which version of the question is more effective in asking the learner to demonstrate their understanding of the topic.

Discussion

You will have discovered from doing this activity that what seems to be a straightforward exercise of setting a question for an online quiz can become quite complicated. The process in the activity is a typical example of what you need to do in order to devise a quiz question.

What wasn’t included in this activity was how to build a whole quiz from a series of questions. It is good practice to have a variety of different question types in your quiz, so that the learner does not start to see a pattern in the type of questions being asked and therefore (perhaps mistakenly) believe that they can quickly answer the quiz without much deep thought. For example, filling a quiz with a majority of True/False questions would produce a quiz which becomes boring to answer and could be easily completed without demonstrating understanding on a second attempt (24 hours later, if they didn’t get it right the first time). So although it may take a bit more work and a bit more thought on your part, it is worth providing a variety of different types of question in your quiz, including random variants of the same question, so that in a second or third attempt at a quiz the learner does not encounter exactly the same set of questions each time.

You also need to think about what feedback the learners will receive at the end of the quiz. For quizzes which have a pass grade, different feedback can be provided for the scores above and below the pass grade and it can be used to guide learners to a particular part of the course.

Figure 4 illustrates the Moodle settings for providing different feedback grade boundaries in a quiz. It shows linked text pointing learners to different parts of the course.

Figure 4 Setting the overall feedback and grade boundaries for a quiz.

Show description|Hide description

This is a screenshot of the quiz ‘overall feedback’ setup screen for a Moodle quiz. Grade boundary 100% is followed by a Feedback box in which some text has been typed: ‘Well done, you have achieved your digital badge for this course. Return to the course to find out what you can do next.’ (Some of this text shows there is a hyperlink to the course.) Below this is Grade boundary 40% followed by a Feedback box in which some text has been typed: Unfortunately your score is not high enough to count towards a badge. Please note that you are allowed to try this assessment as many times as you like. If you haven’t achieved enough correct answers you can attempt the quiz again in 24 hours. This will give you time to review the material before your next attempt. Return to Section 1 of the course’ (some of this text shows there is a hyperlink to section 1 of the course).

Figure 4 Setting the overall feedback and grade boundaries for a quiz.

7.2 Quiz creation process and checklist

The following checklist is a guide to creating robust and effective quizzes:

1. Identify the topics you want to cover in the quiz and write them down, you may do this as you are writing the course and the material is fresh in your mind
2. For each topic, draw up possible ways to ask the question, referring to examples of the different question types in use.
3. Decide on the feedback type for each question.
4. Draft questions for each topic in the question formats you have chosen.
5. Write out some hint text for the interactive with multiple try questions.
6. Write out some feedback text for the deferred feedback and immediate feedback questions.
7. Compile all the questions and organise them into a logical order, grouping the random variant versions together under one question number (for example 3a, 3b, 3c each being a variant for question 3).
8. Ask colleagues to review the questions and feedback text. Request that they provide comments on the pedagogical strength of the questions. Tell them about the potential audience for the open online course and the assessment strategy (including if the quiz summative or formative).
9. Review and if necessary revise the questions, feedback text and assessment strategy in light of comments from colleagues.

Once you are happy with the questions, you can set up the quiz in your online course.

7.3 Summary

The session focused on the different types of quizzes and quiz questions you can use on your open online course. We receive constant feedback from learners on our courses on OpenLearn that they really value quizzes with feedback as it gives them the opportunity to test their learning.

The next session looks at the subject of social learning, and how this can impact on your course.

You can now go to Session 8.

Acknowledgements

This session of the course was written by Anna Page on behalf of the Free Learning team at The Open University.

Images

Figure 1: John T. Spencer https://www.flickr.com/ photos/ edrethink/ 8731759135CC-BY-2.0

Every effort has been made to contact copyright owners. If any have been inadvertently overlooked, the publishers will be pleased to make the necessary arrangements at the first opportunity.