Design: Tips and Checklist
1 Design Tips- General
1. Plan alternative arrangements for students with accessibility issues- use existing college systems and procedures where possible (options will include; paper, screen colours and resolutions etc., scribing, physical assistance and access). Jisc has produced 2 useful guides to help meet the needs of learners with special needs: Meeting The Requirements of Learners With Special Educational Needs and How You Can Make Resources Accessible For Those With Disabilities.
2. Re-sit examinations for summative assessment are important to factor in, as the pressure increases to meet Performance Indicators (PIs). So plan to have enough questions for 3 exam cycles – this will particularly affect your use of Objective Testing / MCQ type assessments
3. Assessment Rubrics in the VLE and Turnitin have proved to be really popular with lecturers as they provide a handy tool for speeding up marking and feedback and making it more consistent. Rubrics can also be used with students as a learning tool that helps to break down and identify the particular knowledge and skills required to meet the outcomes of the unit. This is a good example of a small change making a big impact.
4. Note that the popular plagiarism detection service Turnitin see http://turnitin.com) used by many colleges also supplies powerful online grading tools for essays and reports - these are in general quite a bit slicker to use than college VLE tools, including the rubric function. Use of the grading tools is often overlooked or unknown in Turnitin. There is, however, a downside to this as it is a separate commercial service that colleges have to pay a subscription for to and it has to be set up to work with the local VLE correctly. This means you and your students getting used to two different interfaces.
5. An important consideration when planning for remediation feedback and re-sit examination when using Turnitin is that resubmitting to a Turnitin ‘Assignment Box’ will overwrite any previous submission making it impossible for the lecturers to detect the improvements between the submissions. Best thing to do is to have separate Assignment Boxes, for each attempt and to set up the deadline dates appropriately
6. Do make sure you ‘design in’ activities to introduce / induct your students to the college learning technology systems in general and especially ones that they will be using for assessment activities early on in their academic career and make sure you address any problems early on. A good way to do this is to use a survey / MCQ to find out what devices they use, and what their basic IT skills are like (for using college IT systems) also ask about what internet access they have outside college.
7. If you are expecting your students to be independent learners outside of college teaching contact hours make sure you induct them into what this means and any responsibilities they will have. See the Heart of Worcester College project resources.
8. If you are expecting your students to use their own devices and internet connections to access your online resources and e-assessments, make sure you find out what these are beforehand (see the previous ‘Analyse’ section).
9. Investigate whether your college HR department and managers understand that teaching ‘contact’ hours also include working online.
1.1 Design Tips - Objective Testing / MCQ
1. When creating Objective Testing / MCQ type assessments be aware this can be labour intensive and creating questions for higher order learning can be challenging. But also be aware that the payback can be very high! They can transform marking workloads and of course feedback to students is quick!
2. The Computer Assisted Assessment centre (CAA) has produced a useful introduction to designing Objective / MCQ style tests. This is highly recommended for those new to this area of assessment. The SQA also has a really useful general Guide to Assessment that features a useful section describing question types
3. Remember when developing these assessments, it will be harder to write questions for some outcomes than others (usually the higher order learning tasks from Blooms Learning Taxonomy). A good tip to remember is that most VLE platforms / question authoring tools allow you to create in ‘open text entry’ style questions that can be used for the questions that are more difficult to frame as MCQs. This means you can develop solutions for the harder questions over a longer period of time but still roll out the Objective Testing / MCQ assessment early and include the ‘open text entry’ style questions in them.
4. When designing, developing and introducing Objective Testing / MCQ assessments it makes good sense to do trial runs using them for formative assessments with the target students. This helps to iron out any problems and gets the students and yourself used to the technology and any quirks (there will be some!)
5. If you are new to this allow more time for this work – a good rule of thumb is to double you first estimate.
6. If you can, collaborate with others, – inter-college collaboration makes sense to make this more cost effective. This kind of collaboration can be more difficulty to coordinate than in-house collaboration.
7. Remember some textbooks come with online MCQ tests for teachers to use (downloaded via the web) and publishers often have teacher support websites- these can be used in your VLE. These can be invaluable for providing the basis of a growing question bank and can also be edited (rights permitting) to create new questions more quickly
8. Never use an objective / MCQ type test for summative assessments without making sure your students have done a mock exam in the same conditions / environment as the summative exam
9. When planning a summative objective / MCQ type assessment always make sure you undertake the online test yourself in the same environment to test that it works properly. Make sure you have proper invigilation procedures. Make sure you have a ‘Plan B’ if the online system fails (and yes that may include paper)
1.2 Design Tip - Quality Control / Verification
When planning any e-assessment (and especially for summative objective / MCQ type assessments) make sure you take this through the relevant college procedures. Think about using our design template (or a derivative) to record your design decisions and make sure you create a ‘verification narrative’ that explains to a third party or internal/external verifier how and where they can find the student evidence that demonstrates they have achieved the learning outcomes using your e-assessment.
1.3 Design Tip – E-Portfolio
E-portfolios are, in general, not as mature as VLE systems. In our project we found that student reports etc. that needed to be formatted in certain ways were problematic for students – the formatting was prone to change unexpectedly. So, we would recommend to start with, that where content needs to be formatted to meet certain requirements then it is probably best to do that offline using word processing or presentation software of your choosing (Word, PowerPoint etc.) and then get the students to upload their completed files to the system. This way the students can concentrate on the task at hand to meet that part of the assessment requirements and do not get stressed out dealing with the problems in the e-portfolio formatting tools. This also means the students will find it easier to take their work away with them as well (as files rather than html documents).
Doing this means you (and your students) can concentrate on the main educational features of the e-Portfolio system; to access the student work, get them to share their work, collaborate and comment etc. without getting bogged down in the fine details of online presentation. Later, as you get more used to the system you can explore the formatting options and when you are comfortable with them, provide guidance and support for your students to use them.
3. Checklist of e-Assessment Tools
VLE e-Assessment Tools
Objective / MCQ style tests - interactive multiple-choice, short-answer, jumbled-sentence, crossword, matching/ordering, drag and drop, and gap-fill exercises are some of the options.
Assignment Submission box - typically used for essays and reports (the assignment / report can be authored by individuals or groups) - the classic VLE ‘dropbox’ as it is often called. The assignment activity module enables a teacher to communicate tasks, collect work and provide grades and feedback. Students can submit any digital content (files), such as word-processed documents, spreadsheets, images, or audio and video clips. Alternatively, or in addition, the assignment may require students to type text directly into a text editor. An assignment can also be used to remind students of 'real-world' assignments they need to complete offline, such as practical work, and thus not produce any digital content. One benefit of this is that the teacher can use the Assignment grading tools to mark ‘real’ world practical exercises that involve the creation physical artefacts. Similarly, Assignments are often used to mark the digital contents of student e-Portfolios that exist outside the VLE. Students can submit work individually or as a member of a group. When reviewing assignments, teachers can leave feedback comments and upload files, such as marked-up student submissions, documents with comments or spoken audio feedback. Assignments can be graded using a numerical or custom scale or an advanced grading method such as a rubric. Final grades are recorded in the Gradebook.
Assignment Submission Box (Turnitin) – The online plagiarism detection service Turnitin is widely used in FE and HE to generate ‘similarity’ reports that provide information about what parts of a student’s work are similar to work produced elsewhere. It is usually integrated into a VLE as an option. What is less widely known about the service is that it also includes a powerful grading toolkit that is generally more user friendly that those in VLEs (although that is changing). Drawbacks to using this service include; learning another system and interface, a student cannot make multiple submissions to same assessment – each submission overwrites the previous one.
Rubric– in Moodle / Turnitin / Blackboard etc. rubrics are associated with the Assignment Submission Box. It’s a grading form that uses a table structure, containing a set of criteria (usually down the left hand side) with specified levels of performance to the right of each criterion that form the ‘boxes’ of the table. Clicking on a rubric box for each criterion will automatically create a grade for a student and generate consistent feedback by using the criteria components and their performance level. Personalized feedback can be created as well and added in addition to the auto feedback. When the marks are released the student will see in their view of the rubric the marks and feedback. This has the potential to speed up marking and make it more consistent. Note you can set up a rubric but don’t have to use the associated essay style submission. So, this could be used for other types of assessment activity other than essays - like practice based assessment in order to produce marks and feedback based on the rubric. This may be more useful for graded units in a qualification, although it could be used with ‘pass or fail’ units as well.
Marking Guide – in Moodle – This is a form that that contains the criteria, a space for comments and a space to manually enter the marks for that criterion. It will store frequently used comments as well making feedback easier It is simpler than a rubric and may be more suitable for using when grading ‘Pass or Fail’ SQA units.
Chat – a chat room for real-time text chat by students and staff. Can be good for recording discussions and planning sessions. Contributions can be marked by lecturers and peer assessed by students
Choice- enables a teacher to ask a single question and offer a selection of possible responses. A bit like a single question MCQ. Can be used for formative assessment in order to decide the direction of teaching for individuals and groups
Checklist– in Moodle - of activities for students and staff – can be good for self-assessment and peer assessment and teachers can comment on students work and it can be linked to the marking managing system (Gradebook in Moodle)
Database – in Moodle - The database activity module enables participants to create, maintain and search a collection of entries (i.e. records). The structure of the entries is defined by the teacher as a number of fields. Field types include checkbox, radio buttons, dropdown menu, text area, URL, picture and uploaded file. The teacher can grade student work in this tool and have it recorded in the Gradebook system. This can be a useful alternative to a ‘traditional’ online essay submission box, but make sure you use it for the database features. This can be a very useful tool to create learning resources from students work for future use. The fields the teacher creates for the student to fill in / upload can support and guide the students in their activities. The tool can support peer assessment by allowing students to grade each other’s work.
Feedback – a restricted version of objective / MCQ style test that can be inserted anywhere in a course with multiple choice, yes/no or text input. It is linked to the Gradebook system. It’s really intended for use by teachers to get feedback from students about the course (as the name implies), so is not for peer assessment. It can be used for diagnostic and formative assessment as well
Forum / Discussion Board - The forum activity module enables participants to have asynchronous discussions i.e. discussions that take place over an extended period of time. A teacher can allow files to be attached to forum posts. Attached images are displayed in the forum post. Forum posts can be rated by teachers or students (peer evaluation). Ratings can be aggregated to form a final grade, which is recorded in the Gradebook.
Glossary - The glossary activity module enables participants to create and maintain a list of definitions, like a dictionary, or to collect and organise resources or information. A teacher can allow files to be attached to glossary entries. Attached images are displayed in the entry. Entries can be searched or browsed alphabetically or by category, date or author. Entries can be approved by default or require approval by a teacher before they are viewable by everyone. A teacher can allow comments on entries. Entries can also be rated by teachers or students (peer evaluation). Ratings can be aggregated to form a final grade, which is recorded in the Gradebook.
Lesson - The lesson activity module enables a teacher to deliver content and/or practice activities in interesting and flexible ways. A teacher can use the lesson to create a linear set of content pages or instructional activities that offer a variety of paths or options for the learner. In either case, teachers can choose to increase engagement and ensure understanding by including a variety of questions, such as multiple choice, matching and short answer. Depending on the student's choice of answer and how the teacher develops the lesson, students may progress to the next page, be taken back to a previous page or redirected down a different path entirely. Student performance in a lesson may be graded, with the grade recorded in the Gradebook.
Questionnaire / Survey – This tool construct surveys using a variety of question types, for the purpose of gathering data from users. It is not linked to the Gradebook – typically it is used for end of course evaluations. You can export the response data in the CSV/Excel format and this can be useful to generate report visuals. This can be useful for diagnostic information and formative evaluation that can be shown to the student cohort to give them an overall sense of progress and the range of learning that is being achieved
Quiz / Objective / MCQ style tests – The quiz activity enables a teacher to create quizzes comprising questions of various types, including multiple choice, matching (can be graphical drag and drop), short-answer and numerical interactive multiple-choice, jumbled-sentence, crossword, ordering and gap-fill exercises. The teacher can allow the quiz to be attempted multiple times (or just once), with the questions shuffled or randomly selected from the question bank. A time limit may be set. Each attempt is marked automatically, with the exception of essay free text style questions, and the grade is recorded in the Gradebook. The teacher can choose when and if; hints, feedback and correct answers are shown to students.
SCORM Package – A SCORM package is a collection of files, which are packaged according to an agreed technical standard for ‘learning objects’. Commercial e-learning training providers often produce SCORM packages as these as these will run in different online systems. The SCORM activity module enables SCORM packages to be uploaded as a zip file and added to a course. SCORM content produced by commercial and industrial training providers can be a good way of getting ‘industry standard’ learning resources and assessments into your VLE. Content is usually displayed over several pages, with navigation between the pages. There are various options for displaying content in a pop-up window, with a table of contents, with navigation buttons etc. SCORM activities generally include questions, with grades being recorded in the Gradebook. SCORM activities may be used to present multimedia content and animations and as an assessment tool
Survey – Similar to the Questionnaire, the survey activity module provides a number of verified survey instruments that have been found useful in assessing and stimulating learning in online environments. A teacher can use these to gather data from their students that will help them learn about their class and reflect on their own teaching. Note that these survey tools are pre-populated with questions. Teachers who wish to create their own survey should use the feedback activity module.
Workshop -Workshops are an activity designed to allow peer assessment and potentially a very powerful one. Students submit work into the assessment. The work is allocated to their peers and they are allocated someone else's work to assess. The students grade the work they are allocated. The lecturer can then review the marks and correct any they feel are too far out. The tool has a handy workshop planner that displays all phases of the activity and lists the tasks for each phase. The current phase is highlighted and task completion is indicated with a tick. The workshop activity module enables the collection, review and peer assessment of students' work. Students can submit any digital content (files), such as word-processed documents or spreadsheets and can also type text directly into a field using the text editor. Submissions are assessed using a multi-criteria assessment form defined by the teacher. The process of peer assessment and understanding the assessment form can be practiced in advance with example submissions provided by the teacher, together with a reference assessment. Students are given the opportunity to assess one or more of their peers' submissions. Submissions and reviewers may be anonymous if required.
Checklist for E-Portfolio e-Assessment Tools
Unlike VLEs, which are teacher owned and controlled, the e-portfolio is student owned and controlled, but still usually provided by the college. The student uses the e-Portfolio to collect, organize, present and share digital content they have either created or collected. As well as uploading files they can create web pages and complete journals and blogs detailing their work and reflecting upon it. As we point out in the Tips section above the web formatting tools may be a bit flaky – so check them out first. Students will also create an online profile of themselves that provide their online ‘social media’ identity inside the portfolio system. The teacher uses the e-portfolio system to set up spaces for the students to share their work and to collaborate and use the internal system social media tools. Another important factor to consider and remind your students of at the start is that access to the e-Portfolio usually ends after the student leaves the college – so they need to know how to move their digital materials out of the system if they want to keep it (NB it is best to export in both the formats available – html and LEAP2A). In general, the interest in and use of e-Portfolios is developing rapidly.
e-Portfolio systems usually include ‘internal ‘blogging and social media tools and discussion forums, so that students can comment on and discuss each other’s work – inside the system not on the open web. They include the ability for students to develop and keep their own learning plans and assessment exercises (sometimes these will be pre-created by teachers in the form of various ‘templates’). These plans and templates can then be populated with student-generated content (essays, reports, photos, audio, video, weblinks etc.) that provide evidence of the learning involved.
Teachers can create groups for students to work in and they can also set up spaces into which students can submit their work into for assessment, these groups will usually be associated with a college course. When under assessment in this way the content can be ‘frozen’ until the assessment is complete then released when the assessment is over. The teacher can provide feedback to each student in the e-Portfolio system. In the Mahara system there is a basic 5-star rating system for grading when users (including teachers) add comments on student work, but this is unlikely to be enough for academic marking needs.
For more detailed and complex grading, this can be carried out by using the VLE grading system to mark e-Portfolio content. This is done by setting up an assignment submission in the VLE for the e-portfolio activity and adding the marks and feedback there. The VLE submission can be hidden from students until the marks and feedback are ready to be released. Hiding it in this way will avoid confusion between the two parts of the system. In some institutional systems the VLE and e-Portfolio are more closely integrated and it is possible to set up an assignment in the VLE that is directly linked to the related content in the e-Portfolio
Checklist for Classroom / Lecture Voting Systems
These have been around for some time and can be used in a number of ways. They are popular for rapid diagnostic and formative assessment in a classroom or lecture theatre, with the option to have the results projected onto a video screen – usually anonymised. This can be good to quickly assess where a cohort of students are in their progress towards the course learning outcomes. It can also provide a motivational ‘reality check’ for the students to see where they really are in connection with the learning outcomes of the course.
With a voting system, each student gets a wireless handset and can feedback views, answers or data to a radio receiver and software on a computer. Simple systems are a bit like the audience poll on television quiz shows. More sophisticated versions allow the user to type free text and numbers. The software records the students’ responses and the stored data can produce and format reports, graphs and marking sheets. The teacher creates a quiz or set of questions using the software provided by the system, or in some cases it can be imported. The teacher runs the quiz in class and the students answer using the handset usually choosing from several options. Recent innovations include systems that use ‘apps’ on tablets and mobile phones. These take a bit of setting up and of course the students need to have these on their own devices or be provided with the devices.
Checklist for Social Media e-Assessment tools
When you use social media tools for learning you are stepping out of the closed environment of the college – hence our phrase ‘Leaving the Reservation’. Because most educational use of social media involves using the ‘free’ service options many lecturers and students tend to think of them as ‘natural’ and benign features of the internet environment. But, of course, they are all commercial enterprises and not public services. They make their money by buying and selling information and some of that information is personal – very personal – so it pays to think about the possible statutory and legal implications first. If your college has a policy on this (you should check) then consult it.
None of the participants in our project used social media tools (they concentrated on college systems). But we do know of some uses of these tools, for instance the creation of video ‘blogs’ using YouTube to upload and host videos showing the activity and outputs of student work. Other examples are the use of Google Drive to enable students to easily collaborate on co-authoring a document for assessment, the use of Dropbox for students to upload their videos to for lecturers to access, and Slideshare for students to upload their presentation to for lecturers to access.
The use of social media tools can provide big benefits in utility, speed, usability and convenience compared to college systems. There are a number of legal considerations to take into account when contemplating using these services.
There are a number of important legal and statutory considerations to take into account when contemplating using social media for assessment purposes: Data Protection, Privacy, Inclusion, Discrimination, Defamation, Harassment, and Copyright are some of the more obvious issues. A useful introduction to legal issues has been produced by Jisc in a blog posing entitled ‘Digital skills and values to keep you safe online’ with lots of useful information and links. Jisc also have a handy Social Media for Staff Legal Checklist as well as an online guide to the subject
If you are using social media for your assessment you should record this in IV/EV documentation. You should make sure the SQA EV can easily access the evidence produced by the students. You should take steps to make sure that your feedback and marks to your your individual students remains secure and private.