7.1.2 Question behaviour

In Moodle quiz, you can set the question behaviours to determine whether and when users will receive feedback. You can use quiz question feedback as part of the pedagogy of the course: in practice quizzes it is valuable as a teaching aid to promote deeper thinking about the topic and help learners identify the progress they are making. In graded quizzes it is a bonus to be able to offer feedback to their responses which so often is not available for students taking formal examinations.

Some question behaviours can be combined with Certainty Based Marking (CBM) which asks learners the answer to an objective question and how certain they are that their answer is correct. The mark scheme will reward accurate reporting of certainty and good discrimination between more and less reliable answers to encourage reflection about justification and soundness of relevant knowledge and skills, probing weaknesses more deeply. According to Gardner-Medwin & Curtin [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] , ‘it is easily implemented with existing test material, popular with students, grounded firmly in information theory and proven to enhance the quality of exam data’.

The four types of question behaviours are:

1. deferred feedback with and without Certainty Based Marking (CBM)
2. interactive with multiple tries
3. immediate feedback with and without Certainty Based Marking (CBM)
4. manually graded (not normally suitable for an open online course without a tutor)

Definitions for these question behaviours are:

• deferred feedback – a learner answers all the questions before any feedback is provided. This kind of question behaviour often encourages learners to concentrate on their overall mark than on any feedback specific questions which you might provide.
• interactive with multiple tries – provides instant feedback, hints and multiple tries
• immediate feedback – provides instant feedback with no hints or multiple tries
• manually graded – for essay type questions especially, though can be used for all other question types.

Interactive with multiple tries is the question behaviour most likely to encourage deeper reflective learning.

You also need to think about how accessible the questions are for learners who are using assistive technology or devices which don’t work so well with drag and drop type questions.

Figure 4 A drop-down box in sentence quiz question.

Show description|Hide description

In the grey box on the left is the following text: Question 3, tries remaining 1, marked out of 3.00. Flag question and edit question link (because this is a course creator’s view of the question). In the blue box on the right is the following question text: If you licensed your material with non-derivatives (ND) licence could people add their own examples to it? (The paragraph of text has dropdown boxes showing the word ‘Choose’ in the spaces in the sentence where the words need to be filled in the sentence). Choose, the Choose licence means that Choose can create a new version of your work or remix it.  Under a Choose licence they wouldn’t be able to add their own examples to your materials as this would constitute Choose it.

Figure 4 A drop-down box in sentence quiz question.

The following are some examples of different forms of questions (and the interaction that occur in them) which work well in open online courses:

1. A crossword using drag and drop into text – the learner is asked to drag and drop letters into the correct spaces. This is a useful way to reinforce elements of concepts, though it can be an irritating format of question for people who prefer to type a letter rather than drag it.
2. Crossword using drag and drop onto an image – the crossword is an image and the answers (numbered in a list) can be dragged onto the image.
3. Grid for Sudoku using drag and drop into text – this is a good question which enables learners to gain confidence in manipulating numbers using a familiar puzzle format.
4. Using combined question type to allow input of different numbers to respond to a question – this is a useful question type for reinforcing a teaching point and only requires learners to type their answer into the box provided.
5. Drag and drop onto image to allow for making up whole number and fraction – using the functionality of drag and drop onto an image to rewrite a fraction as a mixed number provides a visual way of enabling learners to see if they have grasped how to convert fractions into a mixed number including a whole number and fraction.
6. Combined question type using select from drop-down list and numeric – for answering a scenario type question which requires several options to be selected (from dropdown lists) and written (in a numeric text box).
7. Combined question type using select from drop-down list and multiple response – can be used to encourage learners to reflect on what they’ve learned and check their understanding.
8. Simple pattern match – the learner types a word into a text-box which is checked against the correct answer in the system. This can be used in a missing word in a sentence question or in answer to a question, to test recollection of something taught in the course or understanding of a concept.
9. Complex pattern match – in which an image or diagram can be included in the question and the learner types the short answer into the text box.
10. Drag and drop using markers – can be used in a question when you want learners to highlight some text to select their answer (such as annotating a poem). It can be quite complicated to set up.
11. Combined question type using multiple response and numeric – the questions are based on an image which is inserted above the questions, some of which require selection from a list and some of which require a number to be typed into a box.
12. Combined question type using select from drop-down list and numeric – allows question types to be combined within one question so learners can provide answers in more than one way, such as selecting a particular word or phrase from a dropdown list and typing a numerical answer. The questions can be based on an image, so this is quite a good question type to use for maps and diagrams.
13. Variable numeric question type – used for generating numerous mathematical questions based on formulae. For example it is possible to create duplicate questions by selecting different values as the basis of the question. The question type can be re-edited to provide variations of the basic question.
14. Drag and drop onto an image – this can be used for mathematical questions as well as to develop pictorial or graphical quizzes on a range of subjects, such as asking learners to complete charts or pie graphs.

Activity 1

This activity will give you some practice of writing a quiz question and the process involved. It does not ask you to try building the quiz question online.

1. Draw up different ways of how to ask a quiz question on a topic of your choice, using the interactive with multiple tries feedback option.
2. Run through the list of question type examples and select at least 2 different ways you could ask your question.  
3. Write out the questions in the question type formats you have chosen. Write out some hint text for interactive with multiple tries.  
4. Test your question by inviting a couple of different colleagues to review it and get their feedback on which version of the question is more effective in asking the learner to demonstrate their understanding of the topic.

Discussion

You will have discovered from doing this activity that what seems to be a straightforward exercise of setting a question for an online quiz can become quite complicated. The process in the activity is a typical example of what you need to do in order to devise a quiz question.

What wasn’t included in this activity was how to build a whole quiz from a series of questions. It is good practice to have a variety of different question types in your quiz, so that the learner does not start to see a pattern in the type of questions being asked and therefore (perhaps mistakenly) believe that they can quickly answer the quiz without much deep thought. For example, filling a quiz with a majority of True/False questions would produce a quiz which becomes boring to answer and could be easily completed without demonstrating understanding on a second attempt (24 hours later, if they didn’t get it right the first time). So although it may take a bit more work and a bit more thought on your part, it is worth providing a variety of different types of question in your quiz, including random variants of the same question, so that in a second or third attempt at a quiz the learner does not encounter exactly the same set of questions each time.

You also need to think about what feedback the learners will receive at the end of the quiz. For quizzes which have a pass grade, different feedback can be provided for the scores above and below the pass grade and it can be used to guide learners to a particular part of the course.

Figure 4 illustrates the Moodle settings for providing different feedback grade boundaries in a quiz. It shows linked text pointing learners to different parts of the course.

Figure 4 Setting the overall feedback and grade boundaries for a quiz.

Show description|Hide description

This is a screenshot of the quiz ‘overall feedback’ setup screen for a Moodle quiz. Grade boundary 100% is followed by a Feedback box in which some text has been typed: ‘Well done, you have achieved your digital badge for this course. Return to the course to find out what you can do next.’ (Some of this text shows there is a hyperlink to the course.) Below this is Grade boundary 40% followed by a Feedback box in which some text has been typed: Unfortunately your score is not high enough to count towards a badge. Please note that you are allowed to try this assessment as many times as you like. If you haven’t achieved enough correct answers you can attempt the quiz again in 24 hours. This will give you time to review the material before your next attempt. Return to Section 1 of the course’ (some of this text shows there is a hyperlink to section 1 of the course).

Figure 4 Setting the overall feedback and grade boundaries for a quiz.