onlinecourse_1.0

6 Assessment for open online courses

Copyright © 2016 The Open University

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, transmitted or utilised in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without written permission from the publisher.

Contents

- Introduction

- 6.1 Why assess an open online course?

- 6.2 Formative and summative assessment

- 6.3 Looking at reflection through assessment

- 6.4 Peer review and peer assessment

- 6.5 Moodle tools for online assessment

- 6.5.1 Choice tool

- 6.5.2 Questionnaire tool

- 6.5.3 Quiz tool

- 6.5.4 Workshop tool (for peer review and assessment)

- 6.6 Issuing open badges

- 6.7 Issuing certificates (Statement of Participation) on OpenLearn Works

- 6.8 Summary

- References

- Acknowledgements

Introduction

In an open online course, it is likely that you are going to want to include an element of assessment, whether informal or formal.

This session of the course covers the type of assessment you might want to use on your open online course and gives some practical tips on how to add assessment to your course successfully.

Listen to the following audio recording in which Anna Page introduces the session.

Audio content is not available in this format.

6 Assessment for open online courses

View transcript - 6 Assessment for open online courses

6.1 Why assess an open online course?

Figure 1 From learning to credential.

View description - Figure 1 From learning to credential.

Choosing whether to use some kind of assessment method in your course ideally needs to be considered in the early stages of learning design – it is linked to the purpose of the course, its potential audience, what different learner motivations might be for studying the course and what learners might expect to achieve as a result. It is also linked to your reasons for creating the course and the choice of topic. You might only want to provide the course as a bite sized piece of knowledge about a topic which people may or may not engage with if they come across it in a web search and therefore assessment may not be required. You may want to integrate some informal assessment activities as part of the pedagogy, or you might want to give people a chance to provide evidence to others of what they’ve learned as a result of working through the material. If any kind of evidence is needed then some sort of assessment method is required and the evidence of their learning is more likely to be taken seriously by employers or colleagues, even if the course is informal and not accredited or is not part of a recognised programme of study.

Deciding on type of assessment and the difficulty level is your next decision. You can add or change the assessment components of the course as it grows, however having assessment options in your thinking when you have decided to include it is recommended.

Open online courses which do not have a tutor or moderator helping to guide learners through material or marking essays have some limitations on how assessment can be used, so your learning design will need to take this into account. However, well designed online assessment activities such as polls, quizzes, questionnaires, reflective exercises and self assessment questions can provide really effective consolidation and evidence of learning – providing learners a way of checking their understanding. This can be tracked and recorded by a course hosting platform leading to the award of digital badges or a certificate (on OpenLearn and OpenLearn Works this is called a Statement of Participation).

6.2 Formative and summative assessment

Formative assessment is used as a way of checking understanding during the learning process – it usually involves feedback which concentrates on the content of an answer or how well the learner performed in an activity, and aims to help improve their learning and understanding. In an open online environment where there might be no tutor available to give feedback, formative assessment activities have to be carefully designed to provide some supportive feedback which will help the learner review what they’ve done and if necessary go back and study the materials again or consider the topic from a different perspective.

Well-crafted online course content interspersed with activities which provide feedback can offer learners a degree of responsiveness about their progress even if it isn’t one-to-one personal feedback. Questionnaire and quiz tools can be used successfully to provide this kind of informal feedback even though quizzes are normally associated with ‘grades’.

According to Nicol and Macfarlane-Dick (2006), there are seven principles of good practice feedback: these include clarifying good performance (goals, criteria, expected standards), facilitating development of self-assessment and providing good quality information about learning to the learners. The principles also include encouraging positive motivation and self-esteem and providing learners with opportunities to improve their own performance. For courses involving interaction with teachers and peers, good feedback practice also means encouraging teacher and peer dialogue about learning and provides teachers with information about the learners which they can use to shape subsequent teaching. All of these can be achieved with a well designed open online course and assessment strategy.

Summative assessment is used to measure the outcome of studying a course and provide a grade which can be used to determine whether a person has ‘passed’ a standard. It can be spaced throughout a course at the end of each section or placed at the end of a course.

It is usually difficult, in the informal learning online situation, to be able to verify that the learner is who they say they are – on most platforms there is no identity test to check that they are not just getting another person to complete the online assessment activities for them. So there is an element of trust in providing online assessment activities which count towards some kind of informal recognition of achievement. It is not like an examination with an invigilator, often used in formal face to face education settings where identity can be checked and learners are observed while they answer the questions. If this is an issue for a course you are creating, you may need to consider the feasibility of hosting a face-to-face examination or using some form of online authentication method for learners to prove their identity and to verify they don’t have someone helping them answer the assessment.

In open online courses, summative assessment is often used to count towards or be the main criteria for a digital badge and/or course completion. A well-constructed quiz covering material from the whole course is one method of conducting summative assessment in this situation, a pass grade can be set and the learner can also receive feedback, which often isn’t offered in a formal summative assessment setting.

6.3 Looking at reflection through assessment

Reflective learning enables learners to examine and explore a topic or issue, in view of their own experience and perhaps those of others with whom they can identify. It helps them to clarify and perhaps challenge their understanding of that topic or concept. Developing the ability to reflect on an action or practical and theoretical issues enables the learner to build on their critical thinking skills. It also helps them plan new approaches to a situation in the future. Questions such as ‘what would you do differently in this situation next time’ or ‘what steps might you take next based on what you have learned’ are typical for reflective learning activities.

In tutor supported learning this kind of question can be assessed via essay type questions or a reflective log, however in an open online course when tutor support is not provided, the essay option is not suitable. However encouraging a learner to write down their thoughts in a reflective log is still a useful option (a ‘self-assessment’ activity) in the online setting as it develops the learner’s ability to reflect, feeds the learner’s understanding and ability to answer summative quiz questions, and provides the learner with a personal record of their own thinking and development. In other words the reflective log (whether done as a series of blog pieces, filling in a downloadable log file or just writing thoughts on a piece of paper without sharing it with anyone) is one part of how reflective learning can be encouraged in an open online course.

Other examples of reflective learning in an open course can include providing a case study of a situation with a series of questions which promote reflective thinking and analysis (this can be assessed using a quiz) or asking a series of questions which require the learner to reflect on the concepts explored in the course and choose, based on their thinking, from a selection of answers.

The following is an example of what a case study reflective learning assessment activity might be:

Instructions to learner:

Read the following case study about how this person coped with the effects of their illness. Write down your thoughts on the situation in your reflective log based on the principles discussed in this section and answer the quiz questions in light of your conclusions.

Case study text

Quiz questions

Responses graded based on criteria in the quiz settings

A version of reflective learning is often found in self and peer assessment.

6.4 Peer review and peer assessment

The familiar definition of peer review is about academic peers reviewing an article or book and deciding if a piece of work is worth publishing, however the term ‘peer review’ is sometimes used interchangeably with peer assessment, which is about students reviewing and assessing each other’s work, often as part of a learning activity or towards graded assessment. The assessment criteria for the peer review and assessment activity is set by the teacher. Sometimes self assessment is built into the criteria as well.

An advantage of peer review and assessment is the range of feedback it can bring to learners – there is learning involved in reviewing another learner’s work which can help learners improve their own subsequent work as they gain insights into the topic from different perspectives and may also lead to better understanding of a topic or concept. Learners can gain valuable analytical experience from evaluating the work of others against the defined criteria.

In addition, peer review and assessment can often result in learners receiving feedback faster than if all assessments are done by a teacher, which is useful when the topic is still being discussed and is therefore still fresh in the minds of the learners.

The assessment criteria sometimes includes rubrics on the different elements or objectives for the activity, giving guidance on how much each element is worth when grading. Elements or objectives for grading can include how the learner has organised their content, their subject knowledge, the expression and originality of their ideas, the balance of material used and how well they keep to the curriculum or topic. The rubric might also ask the learners to assess whether the work contained any grammatical, spelling or typographical errors, if this is an appropriate assessment criteria for the subject.

In an online course there are tools which can facilitate peer review and assessment, usually in time-bound courses with a start and end date which are supported by teachers or tutors. Peer assessment can be offered as both formative and summative assessment options. Learner engagement in this type of assessment activity is sometimes reluctant unless the learners know that they will not achieve all the criteria required for recognition of informal learning if they do not participate.

Providing peer review and assessment in an open online course which is perpetually open is a challenge as a tutor or moderator would be an ongoing expense in this situation, so a special tool is needed to enable ongoing peer review and assessment functionality to work.

(Sources: https://en.wikipedia.org/wiki/Peer_assessment; http://www.beds.ac.uk/jpd/volume-5-issue-1-march-2015/so,-you-want-us-to-do-the-marking!-peer-review-and-feedback-to-promote-assessment-as-learning)

6.5 Moodle tools for online assessment

Virtual learning environments (VLEs) have a variety of tools which can be used for online assessment. A very well-known open source VLE is Moodle. A selection of Moodle tools are explored to show the variety of useful options which can be used for formative and summative assessment activities in online courses. Similar tools exist in other VLEs.

6.5.1 Choice tool

The ‘choice’ tool can be used for simple polling on options, with some generic feedback for each question and the results of the poll displayed afterwards. This can show learners what others chose compared to their choice.

For example:

Measure your environmental impact

Answer the questions below and assess your environmental impact compares to others who have taken this course.

Answer the questions (multiple choice questions with generic feedback at the end of each discussing the extent to which your behaviour might have an impact on the environment)

View other responses (percentage responses for each option in each question, along with some generic feedback)

This type of polling activity can be an effective method of encouraging learners to reflect on their views in comparison with those of others, acting as a stepping stone to the next piece of content in the course and opening up their minds to other possibilities or new avenues to explore.

6.5.2 Questionnaire tool

The ‘questionnaire’ tool in Moodle is often used as a ‘survey’ type tool to gather information from learners.  It has a variety of question types (including check boxes, drop down boxes, essay, numeric, radio buttons, rating and True/False).The rating option enables learners to rank their preferences in a question. The questionnaire tool can be set so that learners can view each other’s responses before or after they’ve shared theirs or so they can only see their own response. Therefore it can be used for an activity which encourages learners to reflect on something they’ve learned and read each other’s reflections. In a course which doesn’t have a specific start and end date, when learners might enrol at any time, seeing responses from others might not be so effective, as there might be few entries when the earlier participants contribute their responses and learners are unlikely to return to the activity later to see if more people have responded.

For example, a formative questionnaire activity might be used to provide a learning activity which promotes reflective learning, such as in the following scenarios, which are all open courses with no start and end date or tutor:

1. Learners are invited to write a short paragraph in response to a question. They don’t see responses from any other learners, this is purely constructed as a self-reflection exercise. The question might have some prompts or issues the learner needs to consider as they write down their thoughts.

2. Learners are invited to write a short paragraph responding to the issues raised in the text they have been studying. The question has some prompts or issues the learner must consider as they write their thoughts. Once a learner has submitted their response, they can view responses from other learners and the activity is followed by a short general discussion paragraph which draws together some of the main points raised in the question and what might be raised by the learners.

3. Learners are asked to write a short paragraph about their experience of the topic which has just been discussed in the text. The question rubric has some prompts or issues they might want to reflect upon in relation to their own experience. Learners can see each other’s responses before they respond, so may also choose to write about their reactions to other responses. (Or learners cannot see each other’s responses until after they respond so their responses are not coloured by the experiences of others).

6.5.3 Quiz tool

The quiz tool enables you to build quizzes comprising questions of various different types, such as multiple choice, matching, short-answer and numerical. It also has an essay question type, however this is less useful in an open online course with no tutor because it cannot easily be marked unless there is human intervention for marking unique textual answers. A quiz can be used for both formative and summative type assessment. It can be set up to record that a person has attempted the questions or that the learner has reached an appropriate grade score to pass the quiz.

Figure 2 Screenshot of a practice quiz from Succeed with maths 1 (OpenLearn).

View description - Figure 2 Screenshot of a practice quiz from Succeed with maths 1 (OpenLearn).

A formative quiz can give learners opportunities to test out how much they have absorbed during a course topic, providing stepped feedback for multiple attempts which may direct them to think about different aspects of a question.

For example:

Learners are asked to complete a quiz checking whether they have understood a concept which has been explained in the text. The quiz consists of some multiple choice and some drag and drop questions. The questions give the learner feedback after each attempt which builds upon the previous information provided and after the third attempt directs the learner back to specific course material to review before they attempt the quiz again 24 hours later. None of the questions are graded or carry any penalty and the completion criteria is set to record that the learner attempted and submitted the quiz.

A summative quiz can capture learner responses to the questions which count towards the completion criteria for a digital badge or course certificate.

For example:

Learners are asked to complete the end of course quiz to test how much they have learned when studying the course. The quiz consists of multiple choice, drag and drop, Yes/No, matching and numerical questions. The questions give the learners feedback after each attempt and after the third attempt directs the learner back to the specific course material to review before they attempt the question again 24 hours later. All the questions are graded, some carry a percentage deduction penalty for second and third attempts. The completion criteria is set to to record the grade which the learner achieves on the quiz. If the learner fails to achieve the pass grade they may reattempt the quiz 24 hours later.

It is really important to think about potential quiz questions while writing or compiling the course rather than leaving the quiz question building until the end of authoring, as there are bound to be opportunities to integrate the quiz into the course and it is easier to do this if you think about how you might ask a question and test understanding from the start rather than bolting it on later. For example the author might write a section and put holding text in the content for activities including quizzes, with rough notes on possible questions to ask. This may also help with writing the content and the perspectives included in the text. Building effective quiz questions is a skill which the author can develop especially if collaborating with others who have experience of what makes a good question which can be answered without writing an essay and which really challenges the learner to think carefully about what they’ve learned.

6.5.4 Workshop tool (for peer review and assessment)

In Moodle, the Workshop tool enables the collection, review and peer assessment of learner work. Allocation of submitted work to learner assessors can be done by the teacher or automatically and there will be set periods for the different stages of the process, which is a time-bound activity.

The learner can submit any digital content, for example a digital file such as a word-processed document or spreadsheet and can also type directly into a field using the text editor. Learner submissions are assessed using multi-criteria assessment form defined by the teacher or tutor. It is possible to practice the process of peer assessment and understanding the form with example submissions. Learners are given the opportunity to assess one or more peer submissions and they can all be anonymous if required. Learners obtain two grades in the workshop activity – a grade for their own submission and a grade for their assessment of peer submissions. The grades are recorded in the gradebook.

6.6 Issuing open badges

A digital badge is a means of recognising certain skills and achievements which have been acquired through informal study online. Badges do not carry any formal credit as they are not subject to the same rigour as formal assessment, nor do they prove that a learner has studied a full course (they may only be awarded for completing a section of a course). They are a useful means of demonstrating learner participation and recognising informal learning.

If a course author decides that the assessment criteria should include the option to earn a digital badge, they will need to provide clear information to learners about the badge criteria, for example:

To gain your digital badge you will need to:

1. Visit all the pages of the course.

2. Complete all the quizzes that you will find at the end of each section of the course. These section quizzes are formative. They will help you consolidate your learning and there is no pass mark.

3. Complete the end-of-course quiz that you will find at the end of the final section and achieve at least 40 per cent.

It is usually also good practice to explain how the badge will be issued, depending on the platform being used to host the course and the badge.

Different platforms will have particular processes for issuing badges. Most digital badges need some kind of evidence of completing the activity required for the badge, this is often completed manually and verified by a person (in badge systems such as Credly.com).

Figure 3 OU badged Open Course badges http://www.open.edu/openlearn/get-started/badges-come-openlearn

View description - Figure 3 OU badged Open Course badges http://www.open.edu/openlearn/get-started/ ...

On OpenLearn and OpenLearn Works, The Open University has developed a method of automatically issuing Mozilla compatible badges based on completion tracking of activities completed within the course, with no manual intervention required (though this is an option if an activity requires human intervention for grading or verifying completion). For example, a learner works through the course content and completes a quiz successfully which results in the system recognising that they have achieved the criteria for the badge, so the badge is issued.

There are various options for how badges are used in a course. The course might be divided into several distinct self contained sections which could each have their own badge. An example of this is the National Networks for Collaborative Outreach courses. Alternatively even if the course is in sections, the badge is only awarded at the end as recognition of completing all the sections, examples of this include Understanding Parkinson’s and the OU Badged Open Courses on OpenLearn. Or the badge might be awarded for completing one single activity, such as submitting a URL link for a blog post the learner has written or uploading a file (which may or may not need to be marked and verified before the badge can be awarded).

On OpenLearn and OpenLearn Works there is a process for setting up badges which includes setting activity completion tracking. For OpenLearn Works you can find the guidelines on how to do this on the Moodle tools on OpenLearn Works page.

6.7 Issuing certificates (Statement of Participation) on OpenLearn Works

Another way of recognising informal learning on your course is to provide learners with a certificate at the end of the course to show their participation. Like digital badges, this ‘certificate’ does not carry any formal credit as it is not subject to the same rigour as formal assessment.

On OpenLearn and OpenLearn Works this certificate is called a ‘Statement of Participation’ to distinguish it from OU certificates for formal study. Course completion criteria have to be set up first then the Statement of Participation is set up. On OpenLearn Works the Statement of Participation can include the logo of the organisation or individual who authored the course and also all the badges which the learner may have achieved while studying the course. For OpenLearn Works you can find the guidelines on how to do this on the Moodle tools on OpenLearn Works page.

6.8 Summary

This session covered the different ways you can assess the learners on your open online course.

The following session builds on this by looking more specifically at quizzes.

You can now go to Session 7.

References

Gardner-Medwin, T. and Curtin, N. (2007) ‘Certainty-Based Marking (CBM) for reflective learning and proper knowledge assessment’, from the REAP International Online Conference on Assessment Design for Learner Responsibility, 29–31 May 2007. Available online at http://www.ucl.ac.uk/lapt/REAP_cbm.pdf (accessed 9 June 2016).

Nicol, D.J. and Macfarlane‐Dick, D. (2006) ‘Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice’, Studies in Higher Education, vol. 31, pp. 199–218. Available online at http://www.tandfonline.com/doi/abs/10.1080/03075070600572090 (accessed 9 June 2016).

Acknowledgements

This session of the course was written by Anna Page on behalf of the Free Learning team at The Open Unviversity.

Except for third party materials and otherwise stated (see terms and conditions), this content is made available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 Licence.

Images

Figure 1: Bryan M Mathers http://bryanmmathers.com/learning-to-credential/ CC-BY-ND

Every effort has been made to contact copyright owners. If any have been inadvertently overlooked, the publishers will be pleased to make the necessary arrangements at the first opportunity.

Figure 1 From learning to credential.

Description
A diagram which shows the path through learning to credential. A blue circle contains the text ‘Something worth learning’ is linked to a white rectangle containing the text ‘Curriculum’. This is linked to a white rectangle containing the text ‘scheme of work’. This links to a white rectangle containing the text ‘Learning activities’. This links to a pink circle containing the word ‘Assessment’ which links to a gold star with the word ‘credential’ underneath it.
Back

Figure 2 Screenshot of a practice quiz from Succeed with maths 1 (OpenLearn).

Description
This image depicts the progress through a quiz.  On the left there is a ‘back to course’ button, just below it is a Heading labelled ‘Questions’ and below this are numbers 1 to 5 in a row. Below each number is either a tick or a circle.  Numbers 1 and 2 have ticks, numbers 3, 4 and 5 have circles indicating that those questions have not yet been completed.  On the right is Question 3, which is a mathematics question. Below Question 3 the text says ‘tries remaining: 3’. The question is ‘write 7/100 as a decimal number’. Below the question is a text box for the answer and a button which is labelled ‘check your answers’.
Back

Figure 3 OU badged Open Course badges http://www.open.edu/openlearn/get-started/badges-come-openlearn

Description
A man looks up from his tablet to see OpenLearn badges hovering in the air alongside him.
Back

6 Assessment for open online courses

Transcript

Anna Page

This session is all about assessment. You’ll explore the reasons you might want to provide some sort of assessment activity for your course and the different types of assessment which work well in an open online environment. You’ll also find out which types are best avoided, especially if your course has no tutor or moderator support.

Choosing the right methods of assessment to suit the purposes of the course, and designing activities which are effective in an online environment is a useful skill to develop. Even if you have previous experience of assessment, perhaps drafting complete examination papers in a formal face to face setting or devising informal assessment activities in a classroom, you may not have compiled an interactive quiz or an online activity before.

This session explores formative and summative assessment, reflection in assessment, peer and self assessment. You’ll also find out about the various Moodle tools which can enable you to challenge learners to really absorb the materials and concepts in the course, resulting in deeper learning.

Back