Develop: Tips and Checklists

1. Develop Tips - Portability and Manageability

The main reasons for retaining an offline stage in your e-assessment workflow are the benefits of Portability and Manageability that it brings.

1. If you author your e-assessment directly in your college systems, you may lose it if the system is upgraded or is ‘purged’ by your technical support.

2. You may lose you e-assessments if the college system suffers a crash (it does happen).

3. You may lose your e-Assessment materials if you leave your job. Exporting your e-assessment materials from a VLE can be quite difficult due to the technical formats that are used – some are closed and proprietary –designed to lock you into a particular system.

4. Setting up your e-assessment in your college system can be tricky enough without having the worry of editing it there as well.

5. The work involved to create objective / MCQstyle tests and questions is considerable and you are advised to do that first in a word processing programme you are comfortable with – it is also much quicker than authoring using VLE system tools.

6. If you are creating visual questions you will need to create the graphic elements externally anyway, using graphics programmes of your choice (the simple ones that come free with PCs / Macs are probably quite adequate for most purposes). For this you should develop a simple and clear naming convention for your image and word processing files as well as using folders with clear names to organise them – essential if you are collaborating.

7. We strongly advise that you first create your essay and report questions etc. and rubrics and marking scales etc. in a word processing programme.

8. For objective MCQ / style tests use a word processing programme to do your design and development work (some lecturers may prefer a spread sheet is they have the skills). If you are working as part of a team, then using a word processing file format like Word will make collaboration much easier than working in the VLE itself.

9. In your offline workflow try and create a simple reference / coding system that links a question to an SQA unit (use the unit code) and a particular learning outcome (use number) and knowledge / skills (bullet number and evidence (bullet number). This seems quite onerous – so, for individuals the SQA code may be enough but for collaborative work teams the full reference path would be ideal (and possible necessary). If you visualise a scenario where you have to manage hundreds of questions, then these issues become very important

2. Develop Tips - Specialist Tools for Creating Objective / MCQ Style Tests

1. If you are going to design and develop more than just a few objective / MCQ style tests in your college VLE, you should consider investing in software tools that work outside of your standard college VLE systems. This is because of the freedom and flexibility they bring and the ability to manage your assessments independently, using tools like Respondus or Question Mark Perception. It is possible to buy them on a campus-wide licence integration for VLEs like Moodle and Blackboard, or as a single user licences. Respondus also comes as a standalone version for authoring and is useful for converting between different delivery platform formats and electronic question standards. This is particularly useful for importing and editing e-assessment questions supplied with textbooks, which is a great way of quickly developing a question bank (see later section on question banks). A major benefit of a tool like this is that you will be able to take your e-assessment questions with you if you change employers and be able to convert them into any format. This all might seem a bit overwhelming at first, so it makes really good sense to involve your college learning technology / IT support department in this process for help.

2. Hot Potatoes is long standing and popular freeware quiz authoring tool that operates on a computer and generates web-based content and quiz assessments. Hot potatoes can export its content as SCORM packages, which can be imported by most VLEs. There is a Hot Potatoes plugin for Moodle that allows a user to upload a Hot Potatoe test directly into a Moodle course. It does not have the range of format conversions of the other tools but it is free and widely used.

3. Xerte is a free and open source content creation tool that enables non-technical users to produce rich interactive web content, including quizzes and tests. It was originally developed by the University of Nottingham and received additional support from Jisc. It follows the ‘learning object’ philosophy and produces content that can be exported in SCORM format that can be used in VLEs. It can also be integrated with Moodle to provide a local online server installation accessed through Moodle. The Chesterfield College group have produced a useful set of introductory resources about using Xerte in a Moodle college setting.

3. Develop Tips – Commercial Solutions

Increasingly employers are using e-learning and e-assessment to deliver induction, training, meeting legal requirements and on-the-job learning and knowledge exchange and there are a growing number of commercial suppliers who can deliver these kinds of services. You have broadly 2 options; one is obvious the other less so.

1. First option - hire a contractor to develop your e-assessments for you. This is most appropriate for e-assessments featuring rich media, interactivity and simulations etc. to deliver objective / MCQ style tests. NB Make sure your contract with the supplier gives you access to all the editable files and media assets so you are not 'locked in'. Many of these suppliers will be using variants of the ADDIE model, so this toolkit should help when working with them. Cost wise this will not be a cheap option, but you have to set that against the benefits this brings. One obvious consideration is that when contemplating such a move you should target e-assessments that feature high student numbers – in order to get a good return on your investment. Another obvious consideration is to check you have the facilities to deliver such e-assessments before you commission them.

2. Second option – collaborate with employers to use their existing e-assessment materials. They may be willing to licence the content to you or let your students use their systems to take the e-assessments. This option would be very useful for diagnostic / formative assessments and to give students an insight into the kind of online training and assessment that many employers are now using. There are obvious benefits in following this path to develop links with employers and in order to prepare students for future employment.

If commissioning materials from commercial suppliers here are a few thinks to consider:

1. Always have a written contract with a project plan / timeline / schedule of work etc.

2. Make sure the contract specifies that the contractor will assign (give over) to your college the ownership of the materials produced - copyright, database rights, moral rights etc. – often referred collectively to as Intellectual Property Rights (IPR).

3. Make sure that any software code you receive is also delivered in the editable version not just the runtime / compiled version – this is essential to allow your own developers or others to alter the code in the future for maintenance purposes.

4. Make sure that any media elements in the materials are delivered separately in ‘high definition’ original editable file versions (e.g. Photoshop) and not just the final edited and compressed web version – again this is essential for future sustainability and maintenance.

4. Develop Tips - Questions and Question Banks

When you first start to design, develop and implement objective / MCQ style tests you naturally think in terms of the test as being the basic unit of your activity and the questions all being subsidiary to that test. However, as you go along and start developing more of these kinds of assessments you are involved building up a ‘question bank’ this means moving your perspective of your e-assessments from exams / tests to that of individual questions. VLE platform suppliers recognise this and allow you to see and manage your entire collection of objective / MCQ style tests questions in the system, regardless of where the individual questions are embedded. This is an extremely useful function and allows you to reuse, edit and combine questions in different ways – greatly speeding up the design and development process.

As the number of questions in your VLE question bank expands you have a very valuable set of resources that can be used in different scenarios (e.g. diagnostic, formative, summative etc.) and with the all the benefits of providing instant marking and feedback. You can see why some lecturers come to see these resources as ‘gold dust’, this is why in the previous section we described the importance of being able to export / import your questions in different technical formats and manage them in an offline space. Although all your questions might be lodged in your college VLE, with some very useful management and editing tools, you should regard this as a temporary situation – prone to change due to technical and employment factors. So, it is sensible to take steps to manage and update these valuable resources in a space that you can control.

5. Develop Checklist

1. For visual questions such as ‘drag and drop’ and ‘name the parts’ it is generally best to use schematic images, drawings and diagrams rather than photographs. The reason for not using photographs is that they contain too much information, detail and clutter – this may seem rather counter intuitive at first. It might help if you consider how many technical maintenance manuals and textbooks use diagrams rather than photographs.

2. For your visual questions, make sure your students have the required ‘visual literacy’ to understand the question – best to check this early on by formative / diagnostic tests. This is particularly important in some disciplines where understanding charts, graphs and symbols etc. is important

3. This may seem obvious, but do remember to get someone to proof read your assessment materials

4. If you can, get a subject matter colleague to check your assessment materials to see if they make sense and are appropriate for the subject and level

5. As a baseline check – ask yourself how your assessment fits back with the learning outcomes and criteria of the course they are being developed for and map to them explicitly. Sometimes when you are closely involved in tasks like this you can drift off course so it’s always good to check. Record the mapping of the questions to the learning outcomes and criteria in your offline records.