Design: Tips and Checklist

1 Design Tips- General

1.     Plan alternative arrangements for students with accessibility issues- use existing college systems and procedures where possible (options will include; paper, screen colours and resolutions etc., scribing, physical assistance and access). Jisc has produced 2 useful guides to help meet the needs of learners with special needs: Meeting The Requirements of Learners With Special Educational Needs and How You Can Make Resources Accessible For Those With Disabilities.

2.     Re-sit examinations for summative assessment are important to factor in, as the pressure increases to meet Performance Indicators (PIs). So plan to have enough questions for 3 exam cycles – this will particularly affect your use of Objective Testing / MCQ type assessments

3.     Assessment Rubrics in the VLE and Turnitin have proved to be really popular with lecturers as they provide a handy tool for speeding up marking and feedback and making it more consistent. Rubrics can also be used with students as a learning tool that helps to break down and identify the particular knowledge and skills required to meet the outcomes of the unit. This is a good example of a small change making a big impact.

4.     Note that the popular plagiarism detection service Turnitin see http://turnitin.com) used by many colleges also supplies powerful online grading tools for essays and reports  - these are in general quite a bit slicker to use than college VLE tools, including the rubric function. Use of the grading tools is often overlooked or unknown in Turnitin. There is, however, a downside to this as it is a separate commercial service that colleges have to pay a subscription for to and it has to be set up to work with the local VLE correctly. This means you and your students getting used to two different interfaces.

5.     An important consideration when planning for remediation feedback and re-sit examination when using Turnitin is that resubmitting to a Turnitin ‘Assignment Box’ will overwrite any previous submission making it impossible for the lecturers to detect the improvements between the submissions. Best thing to do is to have separate Assignment Boxes, for each attempt and to set up the deadline dates appropriately

6.     Do make sure you ‘design in’ activities to introduce / induct your students to the college learning technology systems in general and especially ones that they will be using for assessment activities early on in their academic career and make sure you address any problems early on. A good way to do this is to use a survey / MCQ to find out what devices they use, and what their basic IT skills are like (for using college IT systems) also ask about what internet access they have outside college.

7.     If you are expecting your students to be independent learners outside of college teaching contact hours make sure you induct them into what this means and any responsibilities they will have. See the Heart of Worcester College project resources.

8.     If you are expecting your students to use their own devices and internet connections to access your online resources and e-assessments, make sure you find out what these are beforehand (see the previous ‘Analyse’ section).

9.     Investigate whether your college HR department and managers understand that teaching  ‘contact’ hours also include working online.


1.1 Design Tips - Objective Testing / MCQ

1.     When creating Objective Testing / MCQ type assessments be aware this can be labour intensive and creating questions for higher order learning can be challenging. But also be aware that the payback can be very high! They can transform marking workloads and of course feedback to students is quick!

2.     The Computer Assisted Assessment centre (CAA) has produced a useful introduction to designing Objective / MCQ style tests. This is highly recommended for those new to this area of assessment. The SQA also has a really useful general Guide to Assessment that features a useful section describing question types

3.     Remember when developing these assessments, it will be harder to write questions for some outcomes than others (usually the higher order learning tasks from Blooms Learning Taxonomy[1]). A good tip to remember is that most VLE platforms / question authoring tools allow you to create in ‘open text entry’ style questions that can be used for the questions that are more difficult to frame as MCQs. This means you can develop solutions for the harder questions over a longer period of time but still roll out the Objective Testing / MCQ assessment early and include the ‘open text entry’ style questions in them.

4.     When designing, developing and introducing Objective Testing / MCQ assessments it makes good sense to do trial runs using them for formative assessments with the target students. This helps to iron out any problems and gets the students and yourself used to the technology and any quirks (there will be some!)

5.     If you are new to this allow more time for this work – a good rule of thumb is to double you first estimate.

6.     If you can, collaborate with others, – inter-college collaboration makes sense to make this more cost effective. This kind of collaboration can be more difficulty to coordinate than in-house collaboration.

7.     Remember some textbooks come with online MCQ tests for teachers to use (downloaded via the web) and publishers often have teacher support websites- these can be used in your VLE. These can be invaluable for providing the basis of a growing question bank and can also be edited (rights permitting) to create new questions more quickly

8.     Never use an objective / MCQ type test for summative assessments without making sure your students have done a mock exam in the same conditions / environment as the summative exam

9.     When planning a summative objective / MCQ type assessment always make sure you undertake the online test yourself in the same environment to test that it works properly. Make sure you have proper invigilation procedures. Make sure you have a ‘Plan B’ if the online system fails (and yes that may include paper)


1.2 Design Tip - Quality Control / Verification

When planning any e-assessment (and especially for summative objective / MCQ type assessments) make sure you take this through the relevant college procedures. Think about using our design template (or a derivative) to record your design decisions and make sure you create a ‘verification narrative’ that explains to a third party or internal/external verifier how and where they can find the student evidence that demonstrates they have achieved the learning outcomes using your e-assessment.


1.3 Design Tip – E-Portfolio

E-portfolios are, in general, not as mature as VLE systems. In our project we found that student reports etc. that needed to be formatted in certain ways were problematic for students – the formatting was prone to change unexpectedly. So, we would recommend to start with, that where content needs to be formatted to meet certain requirements then it is probably best to do that offline using word processing or presentation software of your choosing (Word, PowerPoint etc.) and then get the students to upload their completed files to the system. This way the students can concentrate on the task at hand to meet that part of the assessment requirements and do not get stressed out dealing with the problems in the e-portfolio formatting tools. This also means the students will find it easier to take their work away with them as well (as files rather than html documents).

 

Doing this means you (and your students) can concentrate on the main educational features of the e-Portfolio system; to access the student work, get them to share their work, collaborate and comment etc. without getting bogged down in the fine details of online presentation. Later, as you get more used to the system you can explore the formatting options and when you are comfortable with them, provide guidance and support for your students to use them.



1. Design Tips

Design Tips- General

  1. Plan alternative arrangements for students with accessibility issues- use existing college systems and procedures where possible (options will include; paper, screen colours and resolutions etc., scribing, physical assistance and access). Jisc has produced 2 useful guides to help meet the needs of learners with special needs: Meeting The Requirements of Learners With Special Educational Needs and How You Can Make Resources Accessible For Those With Disabilities.

  2. Re-sit examinations for summative assessment are important to factor in, as the pressure increases to meet Performance Indicators (PIs). So plan to have enough questions for 3 exam cycles – this will particularly affect your use of Objective Testing / MCQ type assessments

  3. Assessment Rubrics in the VLE and Turnitin have proved to be really popular with lecturers as they provide a handy tool for speeding up marking and feedback and making it more consistent. Rubrics can also be used with students as a learning tool that helps to break down and identify the particular knowledge and skills required to meet the outcomes of the unit. This is a good example of a small change making a big impact.

  4. Note that the popular plagiarism detection service Turnitin see (http://turnitin.com) used by many colleges also supplies powerful online grading tools for essays and reports  - these are in general quite a bit slicker to use than college VLE tools, including the rubric function. Use of the grading tools is often overlooked or unknown in Turnitin. There is, however, a downside to this as it is a separate commercial service that colleges have to pay a subscription for to and it has to be set up to work with the local VLE correctly. This means you and your students getting used to two different interfaces.

  5. An important consideration when planning for remediation feedback and re-sit examination when using Turnitin is that resubmitting to a Turnitin ‘Assignment Box’ will overwrite any previous submission making it impossible for the lecturers to detect the improvements between the submissions. Best thing to do is to have separate Assignment Boxes, for each attempt and to set up the deadline dates appropriately

  6. Do make sure you ‘design in’ activities to introduce / induct your students to the college learning technology systems in general and especially ones that they will be using for assessment activities early on in their academic career and make sure you address any problems early on. A good way to do this is to use a survey / MCQ to find out what devices they use, and what their basic IT skills are like (for using college IT systems) also ask about what internet access they have outside college.

  7. If you are expecting your students to be independent learners outside of college teaching contact hours make sure you induct them into what this means and any responsibilities they will have. See the Heart of Worcester College project resources.

  8. If you are expecting your students to use their own devices and internet connections to access your online resources and e-assessments, make sure you find out what these are beforehand (see the previous ‘Analyse’ section).

  9. Investigate whether your college HR department and managers understand that teaching  ‘contact’ hours also include working online.


Design Tips - Objective Testing / MCQ

"feedback to students is quick!”

  1. When creating Objective Testing / MCQ type assessments be aware this can be labour intensive and creating questions for higher order learning can be challenging. But also be aware that the payback can be very high! They can transform marking workloads and of course feedback to students is quick!

  2. The SQA also has a really useful general Guide to Assessment that features a useful section describing question types

  3. Remember when developing these assessments, it will be harder to write questions for some outcomes than others (usually the higher order learning tasks from Blooms Learning Taxonomy[1]). A good tip to remember is that most VLE platforms / question authoring tools allow you to create in ‘open text entry’ style questions that can be used for the questions that are more difficult to frame as MCQs. This means you can develop solutions for the harder questions over a longer period of time but still roll out the Objective Testing / MCQ assessment early and include the ‘open text entry’ style questions in them.

  4. When designing, developing and introducing Objective Testing / MCQ assessments it makes good sense to do trial runs using them for formative assessments with the target students. This helps to iron out any problems and gets the students and yourself used to the technology and any quirks (there will be some!)

  5. If you are new to this allow more time for this work – a good rule of thumb is to double you first estimate.

  6. If you can, collaborate with others, – inter-college collaboration makes sense to make this more cost effective. This kind of collaboration can be more difficulty to coordinate than in-house collaboration.

  7. Remember some textbooks come with online MCQ tests for teachers to use (downloaded via the web) and publishers often have teacher support websites- these can be used in your VLE. These can be invaluable for providing the basis of a growing question bank and can also be edited (rights permitting) to create new questions more quickly

  8. Never use an objective / MCQ type test for summative assessments without making sure your students have done a mock exam in the same conditions / environment as the summative exam

  9. When planning a summative objective / MCQ type assessment always make sure you undertake the online test yourself in the same environment to test that it works properly. Make sure you have proper invigilation procedures. Make sure you have a ‘Plan B’ if the online system fails (and yes that may include paper).

Design Tip - Quality Control / Verification


"make sure you create a verification narrative”

When planning any e-assessment (and especially for summative objective / MCQ type assessments) make sure you take this through the relevant college procedures. Think about using our design template (or a derivative) to record your design decisions and make sure you create a ‘verification narrative’ that explains to a third party or internal/external verifier how and where they can find the student evidence that demonstrates they have achieved the learning outcomes using your e-assessment.


Design Tip – E-Portfolio

E-portfolios are, in general, not as mature as VLE systems. In our project we found that student reports etc. that needed to be formatted in certain ways were problematic for students – the formatting was prone to change unexpectedly. So, we would recommend to start with, that where content needs to be formatted to meet certain requirements then it is probably best to do that offline using word processing or presentation software of your choosing (Word, PowerPoint etc.) and then get the students to upload their completed files to the system. This way the students can concentrate on the task at hand to meet that part of the assessment requirements and do not get stressed out dealing with the problems in the e-portfolio formatting tools. This also means the students will find it easier to take their work away with them as well (as files rather than html documents).

Doing this means you (and your students) can concentrate on the main educational features of the e-Portfolio system; to access the student work, get them to share their work, collaborate and comment etc. without getting bogged down in the fine details of online presentation. Later, as you get more used to the system you can explore the formatting options and when you are comfortable with them, provide guidance and support for your students to use them.





[1] https://en.wikipedia.org/wiki/Bloom's_taxonomy