Transcript: 7 How to build a quiz for open online assessment
In this session you will discover some of the principles and good practice involved in building quiz questions for online assessment purposes. You will learn about question types and behaviour as well as the value of providing useful feedback to help to embed learning. You will explore how to devise a quiz which engages and retains the interest of learners without asking them to write essay type answers. Essay questions are usually impossible to mark in an open online course which has no teachers.
You’ll have the chance to try creating a pedagogically robust quiz question which works well in an online setting. There’s also a handy checklist for quiz creation.
End transcript: 7 How to build a quiz for open online assessment
Before you build a quiz using the online quiz tool, it is important to work out what the quiz questions will be and to test them out with colleagues.
The primary purpose of quiz questions in most open online courses is for learners to check their understanding of the course materials and concepts rather than for formal accreditation. It can be quite difficult working out how to ask robust and pedagogically useful questions which can be answered without asking learners to provide an essay response – it very much depends on the content as some materials are more suited to non-essay type questions than others. It is usually possible to find a way to write a quiz for most topics but not possible to use quizzes for all types of knowledge.
It is a good idea to select one or two question types and get used to how they work before you extend the range of question types you use, especially if you are new to creating quizzes.
7.1.1 Question types
Figure 2 Screenshot of a completed practice quiz for Succeed with Maths 1 (OpenLearn).
You can use quizzes to ask learners to do the following:
categorise information and ideas
compare and contrast options
answer questions which require more than one answer
analyse texts or scenarios (such as a case study)
evaluate different options or scenarios
choose from a selection of potential answers (provide at least 4 choices)
Each of these different assessment options can be supported in questions which do not require the learner to write a paragraph or essay and can be designed to draw quite complex thought processes and responses from learners.
Question types available in Moodle quiz include the following:
Combined (more than one type of question combined into a question)
drag and drop into text
drag and drop onto image
drag and drop markers
OU multiple response
select missing words
short answer (this works as a very restricted type of pattern match)
variable numeric set
variable numeric set with units
Questions either require a selected response in which all possible answers are available for the learner to choose from, or a constructed response in which the learner must work out their own answer, though the author would ensure there are a very small number of potentially correct answers against which the learner response can be compared.
Numeric and selected response questions are suitable for using in an open online course for supporting learning and effective assessment. Examples of selected response questions include:
multiple choice (select one from many)
multiple response (select several from many)
true/false (select one from two)
drag and drop into text, select missing word, drag and drop onto images, matching (select one from many several times)
drag and drop markers (onto an image or a piece of text)
Please note that drag and drop quiz questions do have accessibility issues for some users – older browsers do not support drag and drop questions so well as new ones (many learners will be using older browsers) and some tablets also have problems with drag and drop. Therefore you need to consider how much you are going to use this question type in your quiz and if there is a pass grade, whether there are sufficient questions of other types to enable a learner to pass if they cannot use the drag and drop questions successfully.
Only some constructed response questions are suitable for an open online course where there is no tutor to mark the responses, for example essay questions are impossible to mark without a tutor.
Examples of constructed response questions which can be used (with great care in the case of short answer and pattern match) include:
Pattern match and short answer (free text matching is tricky, so although the pattern match question has sophisticated response matching functionality, using this requires careful and thorough authors)
Numeric, variable numeric and variable numeric sets (will match numeric responses)
7.1.2 Question behaviour
In Moodle quiz, you can set the question behaviours to determine whether and when users will receive feedback. You can use quiz question feedback as part of the pedagogy of the course: in practice quizzes it is valuable as a teaching aid to promote deeper thinking about the topic and help learners identify the progress they are making. In graded quizzes it is a bonus to be able to offer feedback to their responses which so often is not available for students taking formal examinations.
Some question behaviours can be combined with Certainty Based Marking (CBM) which asks learners the answer to an objective question and how certain they are that their answer is correct. The mark scheme will reward accurate reporting of certainty and good discrimination between more and less reliable answers to encourage reflection about justification and soundness of relevant knowledge and skills, probing weaknesses more deeply. According to Gardner-Medwin & Curtin, ‘it is easily implemented with existing test material, popular with students, grounded firmly in information theory and proven to enhance the quality of exam data’.
The four types of question behaviours are:
deferred feedback with and without Certainty Based Marking (CBM)
interactive with multiple tries
immediate feedback with and without Certainty Based Marking (CBM)
manually graded (not normally suitable for an open online course without a tutor)
Definitions for these question behaviours are:
deferred feedback – a learner answers all the questions before any feedback is provided. This kind of question behaviour often encourages learners to concentrate on their overall mark than on any feedback specific questions which you might provide.
interactive with multiple tries – provides instant feedback, hints and multiple tries
immediate feedback – provides instant feedback with no hints or multiple tries
manually graded – for essay type questions especially, though can be used for all other question types.
Interactive with multiple tries is the question behaviour most likely to encourage deeper reflective learning.
You also need to think about how accessible the questions are for learners who are using assistive technology or devices which don’t work so well with drag and drop type questions.
Figure 4 A drop-down box in sentence quiz question.
The following are some examples of different forms of questions (and the interaction that occur in them) which work well in open online courses:
A crossword using drag and drop into text – the learner is asked to drag and drop letters into the correct spaces. This is a useful way to reinforce elements of concepts, though it can be an irritating format of question for people who prefer to type a letter rather than drag it.
Crossword using drag and drop onto an image – the crossword is an image and the answers (numbered in a list) can be dragged onto the image.
Grid for Sudoku using drag and drop into text – this is a good question which enables learners to gain confidence in manipulating numbers using a familiar puzzle format.
Using combined question type to allow input of different numbers to respond to a question – this is a useful question type for reinforcing a teaching point and only requires learners to type their answer into the box provided.
Drag and drop onto image to allow for making up whole number and fraction – using the functionality of drag and drop onto an image to rewrite a fraction as a mixed number provides a visual way of enabling learners to see if they have grasped how to convert fractions into a mixed number including a whole number and fraction.
Combined question type using select from drop-down list and numeric – for answering a scenario type question which requires several options to be selected (from dropdown lists) and written (in a numeric text box).
Combined question type using select from drop-down list and multiple response – can be used to encourage learners to reflect on what they’ve learned and check their understanding.
Simple pattern match – the learner types a word into a text-box which is checked against the correct answer in the system. This can be used in a missing word in a sentence question or in answer to a question, to test recollection of something taught in the course or understanding of a concept.
Complex pattern match – in which an image or diagram can be included in the question and the learner types the short answer into the text box.
Drag and drop using markers – can be used in a question when you want learners to highlight some text to select their answer (such as annotating a poem). It can be quite complicated to set up.
Combined question type using multiple response and numeric – the questions are based on an image which is inserted above the questions, some of which require selection from a list and some of which require a number to be typed into a box.
Combined question type using select from drop-down list and numeric – allows question types to be combined within one question so learners can provide answers in more than one way, such as selecting a particular word or phrase from a dropdown list and typing a numerical answer. The questions can be based on an image, so this is quite a good question type to use for maps and diagrams.
Variable numeric question type – used for generating numerous mathematical questions based on formulae. For example it is possible to create duplicate questions by selecting different values as the basis of the question. The question type can be re-edited to provide variations of the basic question.
Drag and drop onto an image – this can be used for mathematical questions as well as to develop pictorial or graphical quizzes on a range of subjects, such as asking learners to complete charts or pie graphs.
This activity will give you some practice of writing a quiz question and the process involved. It does not ask you to try building the quiz question online.
Draw up different ways of how to ask a quiz question on a topic of your choice, using the interactive with multiple tries feedback option.
Run through the list of question type examples and select at least 2 different ways you could ask your question.
Write out the questions in the question type formats you have chosen. Write out some hint text for interactive with multiple tries.
Test your question by inviting a couple of different colleagues to review it and get their feedback on which version of the question is more effective in asking the learner to demonstrate their understanding of the topic.
You will have discovered from doing this activity that what seems to be a straightforward exercise of setting a question for an online quiz can become quite complicated. The process in the activity is a typical example of what you need to do in order to devise a quiz question.
What wasn’t included in this activity was how to build a whole quiz from a series of questions. It is good practice to have a variety of different question types in your quiz, so that the learner does not start to see a pattern in the type of questions being asked and therefore (perhaps mistakenly) believe that they can quickly answer the quiz without much deep thought. For example, filling a quiz with a majority of True/False questions would produce a quiz which becomes boring to answer and could be easily completed without demonstrating understanding on a second attempt (24 hours later, if they didn’t get it right the first time). So although it may take a bit more work and a bit more thought on your part, it is worth providing a variety of different types of question in your quiz, including random variants of the same question, so that in a second or third attempt at a quiz the learner does not encounter exactly the same set of questions each time.
You also need to think about what feedback the learners will receive at the end of the quiz. For quizzes which have a pass grade, different feedback can be provided for the scores above and below the pass grade and it can be used to guide learners to a particular part of the course.
Figure 4 illustrates the Moodle settings for providing different feedback grade boundaries in a quiz. It shows linked text pointing learners to different parts of the course.
Figure 4 Setting the overall feedback and grade boundaries for a quiz.
The following checklist is a guide to creating robust and effective quizzes:
Identify the topics you want to cover in the quiz and write them down, you may do this as you are writing the course and the material is fresh in your mind
For each topic, draw up possible ways to ask the question, referring to examples of the different question types in use.
Decide on the feedback type for each question.
Draft questions for each topic in the question formats you have chosen.
Write out some hint text for the interactive with multiple try questions.
Write out some feedback text for the deferred feedback and immediate feedback questions.
Compile all the questions and organise them into a logical order, grouping the random variant versions together under one question number (for example 3a, 3b, 3c each being a variant for question 3).
Ask colleagues to review the questions and feedback text. Request that they provide comments on the pedagogical strength of the questions. Tell them about the potential audience for the open online course and the assessment strategy (including if the quiz summative or formative).
Review and if necessary revise the questions, feedback text and assessment strategy in light of comments from colleagues.
Once you are happy with the questions, you can set up the quiz in your online course.
The session focused on the different types of quizzes and quiz questions you can use on your open online course. We receive constant feedback from learners on our courses on OpenLearn that they really value quizzes with feedback as it gives them the opportunity to test their learning.
The next session looks at the subject of social learning, and how this can impact on your course.