Course

Finding Your Own Way

Introducing e-assessment is often not a straightforward exercise as there are many factors affecting it that change over time. It has a unique ability to act as an effective ‘lightning rod’ to bring contextual factors into very clear focus – including (perhaps surprisingly) deeply held personal ideas about learning. This can be complicated by the simplistic commercial hype and technically determinist language that sometimes accompanies discussions of e-learning.

Jisc has previously sponsored the publication of an in-depth guide to understanding these factors called Effective Networked Learning^[1] , by the Educational Research Department at Lancaster University. Below is a link to a graphic from the guide that introduces and illustrates how some of these ‘invisible’ contextual factors interact:

Click on this link to display the Organisational Context Diagram

Caption: Contextual factors affecting e-assessment. How pedagogical frameworks, the local educational setting, technology and the organisation interact to produce the context. From the Jisc Guide to Networked Learning

This is expressed another way in the skills pyramid illustration shown below, which brings these contextual actors into focus from the perspective of individual lecturers and they skills they will need. This is derived from several areas of research^[2] that examine how educators can make effective use of technology. The graphic describes the typical ‘skills pyramid’ of an effective teacher with educational technology in the area of e-assessment. You will see that being able to work around the limitations of both the technology and the institution are essential foundations and at the top level; analysis, reflection and creativity are needed in order to develop your own personal style of working in this area.

Click on this link to display the e-Assessment skills pyramid

We argue that finding solutions to the technical and institutional constraints require a much more collaborative approach than exists at present. It is unlikely that you’re working context will be perfect and you will need to ‘find your own way’, collaboration with others will certainly help and this guide with its Readings and Tips and Checklists  will assist you to move quickly forwards.

Analyse - Overview

This section is crucial and represents the foundation of what comes next in the design stage. Here we focus on discovering and analysing the ‘nitty gritty’ of your own situation in relation to e - assessment. This is where you explore your college systems and develop your own abilities to use them, as well as any external tools that you need.

A really good tip is to borrow a method from the world of art and design and start keeping a sketchbook / notebook/ log to jot down ideas and questions. This may be paper or a digital Word document or some other digital record, although many find the immediacy of paper and pen the best.

You need to gather this information and develop practical skills in this section in order to understand your local context and discover any limitations you need to work around. This section is all about developing an enquiring, critical, and systematic approach. By ‘critical’ we especially mean not taking things for granted or believing what you are told at face value – this is especially important in relation to learning technology and popular expectations and stereotypes in relation to students. You really do need to try things out here first before moving onto the design phase.

Design - Overview

In this section we move on from Analysis to the Design phase of development. Remember the ADDIE model is meant to be iterative, so it’s OK to jump back and forwards between the sections. ADDIE just provides a general overall structure to work within. The Design section is where you start to sketch out, in ever-increasing detail, what you plan to do and how. The stage after Design, ‘Develop’ is where you turn those ideas into more fully formed items and the stage after that, ‘Implement’, is where we deploy our e-assessments with students using the various technologies available.

Here’s a good tip taken from the world of Art and Design– keep a ‘scrapbook’ of e-assessments that you like, that you have found useful or inspiring (and indeed more general e-learning designs). Ideally your scrapbook will be digital and it can be as simple or complicated as you like – you might want to use a software tool or a web service like Evernote or even a folder on your PC/Tablet called ‘Scrapbook’ into which you put web links or downloads and use a Word document (or similar) into which you make reference notes about the items you have found. You might also want to adopt another Art and Design habit of keeping a ‘sketchbook’ to jot down ideas, notes and sketches – this can be digital, but many people like the immediacy and simplicity of pen and paper, including software developers!

Another good tip is to design for formative assessments first – this enables you to learn the tools, develop your skills and explore the limitations of the college context and systems that you must deal with. It’s a lot less stressful that going straight for summative e-assessment at the start. It lets you build up to that gradually. It gives you a solid foundation to build on and, of course, formative assessment helps students to learn.

Develop - Overview

In the ‘Develop’ stage we take our ideas from the design stage and create the fully-formed versions ready to use in the next ‘Implement’ stage, where we deploy and maintain our e-assessments using the various technologies and delivery platforms available. As we have already observed the ADDIE model is meant to be iterative and it is OK to jump backwards and forwards between the sections. Another aspect to the model is that the distinctions between the sections need not always be clear-cut, depending on the context and technologies you are working on. In the industrial Computer Based Training (CBT) scenario that ADDIE was developed for, the ‘Develop’ stage would be where the design is converted into instructional learning materials, including relatively static presentational ‘content ‘and more interactive online materials often as  ‘run time versions’ (SCORM etc.). This phase usually involves considerable digital media and software development activity to produce learning resources ready to be loaded into a delivery system for training and assessment in the ‘Implement’ stage.

However, in the Develop stage of your e-assessments, you and your team probably won’t be doing very intensive technical work with sophisticated e-learning authoring tools (such as Flash [yes some organisations are still using Flash!], Articulate and Captivate etc.) to create interactive learning content and assessments. If you are, then there is a wealth of detailed technical guidance about the individual tools and technical standards available. One of our project activities did actually make use of these types of tools and we describe how that worked – please see the tourism case study on the original CITEA project web site under the Case Study Tab.

It is much more likely you will be putting your designs into your college systems (VLE & e-Portfolio) and making them work there. In this scenario, it is common for people to move on from creating their e-assessment designs and go directly to developing / editing them and implementing them directly in their college systems. We would suggest not doing this and suggest you retain a separate offline step in your workflow before you touch the college delivery systems (VLE, e-Portfolio etc.) or any of the specialist tools for creating objective / MCQ style tests. We describe our reasons below in the Tips and Checklist section - there are some real advantages to be gained by doing this - especially for objective / MCQ style tests.

Implement - Overview

In the implement stage we shall be loading our e-assessments into the delivery platform (the College VLE / e-Portfolio in most cases) and running them with students and staff. This is where our previous stages of work using the ADDIE model will pay dividends, at this point you will have already set up and operated test e-assessments using the college systems so you should be familiar with them and their quirks. Setting dates, access conditions, providing feedback and recording and managing marks are all tasks that typically can be tricky at first – so it is essential to practice beforehand.

Evaluate - Overview

Evaluation is both the last and first stage in the ADDIE model. It leads into another cycle of activity and should also occur during each stage as well. Try to get into the habit of jotting down your ideas and evaluation of your progress as you go – don’t leave it all to the end. This is where keeping an e-assessment ‘sketchbook’ or ‘notebook’ can be really useful. Evaluation of e-learning is a research subject in its own right and as demand grows for evidence based practice and strategy in relation to learning technology, then thorough evaluation becomes more important.

A useful resource for planning your evaluations is the ‘Evaluation Cookbook’ produced by Heriot-Watt University, this provides insights and guidance on undertaking different types of evaluation of learning technology. Another really useful set of resources is the work of Don Clark, who comes from the North American Instructional design tradition. He has produced a very accessible online guide to instructional design in general that some readers will find very useful. His guide also contains a usefulsection on evaluation including Kirpatrick’s four levels of evaluation.