Course

This course steps you through the process of converting one of your own existing paper-based assessments into an e-assessment from beginning to end. It covers typical problems to look out for when introducing e-assessment and provides tips, checklists and design templates for busy lecturers, support staff and managers to use and adapt to their own needs. The course advocates a systematic and collaborative approach to these tasks. The quality management measures and the information needed to support them effectively are identified, together with simple tools and methods to use. Introducing e-assessment is a design intensive process that in turn requires a fair bit of investigation and analysis.

To be successful in this field you will need to be able to identify and overcome problems
in a pragmatic way – an approach associated with engineering disciplines. To support us in this activity we have chosen to adopt the widely ADDIE (Analyse, Design, Develop, Implement & Evaluate) model of instructional design to structure this course. It's very useful method for structuring our activities and places an emphasis on an iterative way of working that allows you to go back one or more steps to change your approach, the image illustrates this

The concept of a problem space is useful when tackling e-assessment – it comes from the disciplines of design and engineering – its a way of recognising that there may be a number interconnected problems that need to be identified and solved to resolve the bigger problem. That is certainly the case with digital assessment, below is a starter set of sub problems to get you thinking:

• Education is a highly sensitive political issue in our society and intimately connected with fundamental ideas, assumptions, and values about how society should work and people should treat each other. Any change here can generate heated debate that involves powerful interest groups inside and outside institutions
  
• The technology we have access to is not as good as we might think in terms of usability
  
• There is considerable commercial and political hype surrounding the topic of technology in education that often leads to entrenched and unrealistic assumptions - this can make realistic assessment and planning difficult
  
• Lack of technical infrastructure in institutions is still a considerable problem
  
• Institutional alignments of service departments (IT, HR, Library etc.), budgets, strategy and management involvement to deliver effective e-assessments is difficult
  
• Organisational silos and internal competition for funds, influence and control are common issues
  
• Commercially driven media myths about the technical abilities of young people are a considerable obstacle – especially in relation to using out-dated institutional systems
  
• Creating e-assessment questions and answers is an expensive labour intensive undertaking and there needs to be much more shared effort, with in and between institutions, rather than individual re-inventions of the wheel to get the full economic benefit
  
• Staff skills (technical and pedagogical) need to be developed and supported effectively
  
• Institutional information management of assessment results and student materials in digital formats remains a considerable challenge – especially maintenance in the longer term
  
• Working culture is a major factor, do lecturers work as a team or is there a tendency to operate in silos? Is the senior management involved and supportive?
  

Finding Your Own Way

Introducing e-assessment is often not a straightforward exercise as there are many factors affecting it that change over time. It has a unique ability to act as an effective ‘lightning rod’ to bring contextual factors into very clear focus – including (perhaps surprisingly) deeply held personal ideas about learning. This can be complicated by the simplistic commercial hype and technically determinist language that sometimes accompanies discussions of e-learning.

Jisc has previously sponsored the publication of an in-depth guide to understanding these factors called Effective Networked Learning^[1] , by the Educational Research Department at Lancaster University. Below is a link to a graphic from the guide that introduces and illustrates how some of these ‘invisible’ contextual factors interact:

Click on this link to display the Organisational Context Diagram

Caption: Contextual factors affecting e-assessment. How pedagogical frameworks, the local educational setting, technology and the organisation interact to produce the context. From the Jisc Guide to Networked Learning

This is expressed another way in the skills pyramid illustration shown below, which brings these contextual actors into focus from the perspective of individual lecturers and they skills they will need. This is derived from several areas of research^[2] that examine how educators can make effective use of technology. The graphic describes the typical ‘skills pyramid’ of an effective teacher with educational technology in the area of e-assessment. You will see that being able to work around the limitations of both the technology and the institution are essential foundations and at the top level; analysis, reflection and creativity are needed in order to develop your own personal style of working in this area.

Click on this link to display the e-Assessment skills pyramid

We argue that finding solutions to the technical and institutional constraints require a much more collaborative approach than exists at present. It is unlikely that you’re working context will be perfect and you will need to ‘find your own way’, collaboration with others will certainly help and this guide with its Readings and Tips and Checklists  will assist you to move quickly forwards.

Analyse - Overview

This section is crucial and represents the foundation of what comes next in the design stage. Here we focus on discovering and analysing the ‘nitty gritty’ of your own situation in relation to e - assessment. This is where you explore your college systems and develop your own abilities to use them, as well as any external tools that you need.

A really good tip is to borrow a method from the world of art and design and start keeping a sketchbook / notebook/ log to jot down ideas and questions. This may be paper or a digital Word document or some other digital record, although many find the immediacy of paper and pen the best.

You need to gather this information and develop practical skills in this section in order to understand your local context and discover any limitations you need to work around. This section is all about developing an enquiring, critical, and systematic approach. By ‘critical’ we especially mean not taking things for granted or believing what you are told at face value – this is especially important in relation to learning technology and popular expectations and stereotypes in relation to students. You really do need to try things out here first before moving onto the design phase.

Design - Overview

In this section we move on from Analysis to the Design phase of development. Remember the ADDIE model is meant to be iterative, so it’s OK to jump back and forwards between the sections. ADDIE just provides a general overall structure to work within. The Design section is where you start to sketch out, in ever-increasing detail, what you plan to do and how. The stage after Design, ‘Develop’ is where you turn those ideas into more fully formed items and the stage after that, ‘Implement’, is where we deploy our e-assessments with students using the various technologies available.

Here’s a good tip taken from the world of Art and Design– keep a ‘scrapbook’ of e-assessments that you like, that you have found useful or inspiring (and indeed more general e-learning designs). Ideally your scrapbook will be digital and it can be as simple or complicated as you like – you might want to use a software tool or a web service like Evernote or even a folder on your PC/Tablet called ‘Scrapbook’ into which you put web links or downloads and use a Word document (or similar) into which you make reference notes about the items you have found. You might also want to adopt another Art and Design habit of keeping a ‘sketchbook’ to jot down ideas, notes and sketches – this can be digital, but many people like the immediacy and simplicity of pen and paper, including software developers!

Another good tip is to design for formative assessments first – this enables you to learn the tools, develop your skills and explore the limitations of the college context and systems that you must deal with. It’s a lot less stressful that going straight for summative e-assessment at the start. It lets you build up to that gradually. It gives you a solid foundation to build on and, of course, formative assessment helps students to learn.

Develop - Overview

In the ‘Develop’ stage we take our ideas from the design stage and create the fully-formed versions ready to use in the next ‘Implement’ stage, where we deploy and maintain our e-assessments using the various technologies and delivery platforms available. As we have already observed the ADDIE model is meant to be iterative and it is OK to jump backwards and forwards between the sections. Another aspect to the model is that the distinctions between the sections need not always be clear-cut, depending on the context and technologies you are working on. In the industrial Computer Based Training (CBT) scenario that ADDIE was developed for, the ‘Develop’ stage would be where the design is converted into instructional learning materials, including relatively static presentational ‘content ‘and more interactive online materials often as  ‘run time versions’ (SCORM etc.). This phase usually involves considerable digital media and software development activity to produce learning resources ready to be loaded into a delivery system for training and assessment in the ‘Implement’ stage.

However, in the Develop stage of your e-assessments, you and your team probably won’t be doing very intensive technical work with sophisticated e-learning authoring tools (such as Flash [yes some organisations are still using Flash!], Articulate and Captivate etc.) to create interactive learning content and assessments. If you are, then there is a wealth of detailed technical guidance about the individual tools and technical standards available. One of our project activities did actually make use of these types of tools and we describe how that worked – please see the tourism case study on the original CITEA project web site under the Case Study Tab.

It is much more likely you will be putting your designs into your college systems (VLE & e-Portfolio) and making them work there. In this scenario, it is common for people to move on from creating their e-assessment designs and go directly to developing / editing them and implementing them directly in their college systems. We would suggest not doing this and suggest you retain a separate offline step in your workflow before you touch the college delivery systems (VLE, e-Portfolio etc.) or any of the specialist tools for creating objective / MCQ style tests. We describe our reasons below in the Tips and Checklist section - there are some real advantages to be gained by doing this - especially for objective / MCQ style tests.

Implement - Overview

In the implement stage we shall be loading our e-assessments into the delivery platform (the College VLE / e-Portfolio in most cases) and running them with students and staff. This is where our previous stages of work using the ADDIE model will pay dividends, at this point you will have already set up and operated test e-assessments using the college systems so you should be familiar with them and their quirks. Setting dates, access conditions, providing feedback and recording and managing marks are all tasks that typically can be tricky at first – so it is essential to practice beforehand.

Evaluate - Overview

Evaluation is both the last and first stage in the ADDIE model. It leads into another cycle of activity and should also occur during each stage as well. Try to get into the habit of jotting down your ideas and evaluation of your progress as you go – don’t leave it all to the end. This is where keeping an e-assessment ‘sketchbook’ or ‘notebook’ can be really useful. Evaluation of e-learning is a research subject in its own right and as demand grows for evidence based practice and strategy in relation to learning technology, then thorough evaluation becomes more important.

A useful resource for planning your evaluations is the ‘Evaluation Cookbook’ produced by Heriot-Watt University, this provides insights and guidance on undertaking different types of evaluation of learning technology. Another really useful set of resources is the work of Don Clark, who comes from the North American Instructional design tradition. He has produced a very accessible online guide to instructional design in general that some readers will find very useful. His guide also contains a usefulsection on evaluation including Kirpatrick’s four levels of evaluation.

The first 3 tips are general good practice ideas for assessment – with and without technology - and provide a solid foundation for development.

1.  Are your assessments aligned [1] with the learning outcomes? This may appear obvious but:

2.  Make sure you explicitly map the assessments to the outcomes and record that mapping in the course documentation – better still - make this clear to your students.

3.  How does this assessment help your students to learn? 

Look at the aims and outcomes for your course then look at these aspects of an assessment:

4.  Design - Record any changes to an existing assessment and store that in the system for Internal Verification (quality) that your college uses:

5.  Prepare a short and clear ‘External Verifier (EV) narrative’ to provide to the EV that describes and explains the changes made and the use of technology. 

6.  Make sure you and other teaching staff are proficient with the tech tools you are using - particularly the VLE grade records and management system and their links into the college student records system

7.  Always test the assessment personally after setting it up:

8.  Students do not like surprises. 

9. Student digital literacy and independent learning:

10.  Collaborate and plan ahead – things work best when the work is shared. 

This is particularly true of using e-learning technologies in education. It will make things easier for everyone. Technology is quite unforgiving if you are not organised! So don’t leave things to the last minute. That approach might work with paper – but not with technology. Remember to use the systems mind set advocated in this toolkit and look ahead to identify potential problems and bottlenecks. Simple techniques include:

[1]The MIT Teaching and Learning Lab has produced a useful guide to this: http://tll.mit.edu/help/assessment-outcome-alignment