Pain Points

Jisc have produced a really useful review of this area in their Electronic Management of Assessment (EMA): a landscape review and In a section called ‘Pain Points’ they have looked at the way different factors are affecting the take up for e - assessment: 

• Teaching Models

• Technology

• Process (administration etc.)

• Culture (ways of working, reporting, attitudes values etc.)

Their conclusions and observation closely mirror our experience and include :

“The interplay between all of the factors is complex: it is evident that the existing commercial and open source systems do not effectively support all of the existing processes but there are equally some cases where process improvement could clearly be achieved. Similarly,we heard some quite harsh comments about institutional culture but it is clear that experiences with immature or unreliable technologies can turn neutral (or even slightly positive) early adopters into resisters.”

The lack of integration between the VLE and the administration systems is a particularly problematic one and often compounded by the different lines of responsibility and control and resourcing for the VLE, IT and admin systems in many institutions.

Student Skills and Attitudes

Another key barrier are existing student skills and attitudes, together with the preparation and orientation that they may need to undertake e-assessment. We could describe this as the digital literacies needed by students to use the college e-assessment systems effectively.The Heart of Worcester college has produced some award winning development resources to support students. This often comes as a surprise to people new to adopting e-assessment; the assumption is often that the skills problems will be with the lecturers. Despite the considerable commercially biased hype that exaggerates the digital abilities of young people ^[1] the actual research continually paints a very different picture ^[2].

Jisc have produced a guide to developing student literacies that includes the concept of the ‘7 elements of digital literacy’ this is illustrated in the diagram shown below:

[Caption: The Jisc 7 elements of digital literacy Licence CC BY-NC-ND]

As one student observed at City of Glasgow College –‘just because you are good on PlayStation or Facebook does not mean you can use the VLE!’

In our project many lecturers reported problems with students IT skills at a basic level when required to do things like rename files and upload them to a sub mission. This was also identified as problem at the UHI (University of the Highlands and Islands), widely regarded as being at the forefront of e-learning in Scotland, who have produced checklists of the basic skills needed to participate with links to remedial support resource, these are available in the Further Information section of the project website.

Staff skills and attitudes

The Jisc ETNA staff skills surveys of teaching staff in FE in Scotland suggest that there is growing confidence in using computers and common ‘office’ applications. But these surveys showed there is markedly less experience and confidence in using online web - based systems like VLEs and e-Portfolios. This is backed up by the work of the Borders College BOLT project:

(ref: New Media Consortium Horizon Report 2013)

In our project this observation certainly fitted with our experience, we certainly found that lecturers were not confident in using the VLE or e-portfolio systems. This situation is further complicated by the number of online systems lecturers might have to master. The Jisc ElectronicManagement of Assessment (EMA) report observes:

So, we still have a long way to go in terms of system integration and data management, with multiple paper and electronic systems being used, although Jisc are currently researching a feedback hub to improve matters.^[3]

Usability Factors

The actual tools provided in - house by the colleges for lecturers and students to use may be old in terms of technology and design and can suffer from usability issues. To be fair, the actual tasks that the tools are supporting are in some cases quite complex themselves - particularly on the teaching side. However, that said, the usability of the tools is an issue and does affect the take - up of these technologies.Once an institution has adopted these tools there is little appetite for changing them, which in turn reduces the incentives for improvement amongst suppliers, so this situation is unlikely to change quickly. This is why you need to use the tools yourself and experiment with them to get to know them and their limitations. In general the usability issues affect the teachers using the system much more than the students due to the complexity of the tasks in setting up assessments, with date, times and access conditions being particularly problematic - partly due to the terminology used.

It is particularly important to be able to see the systems from a student point of view, in some software products there is a ‘student view’ function (the one in Blackboard is particularly good) but it is also sensible to have some test student accounts to allow you to step through your assessments in the system exactly like a student and to record student data in the system.

---

^{[1] Jisc have produced a useful guide about developing student digital literacy: https://www.jisc.ac.uk/guides/developing-students-digital-literacy↩
[2] http://digitalstudent.jiscinvolve.org/wp/fe-and-skills-digital-student-study/desk-study/↩}

^{[3]http://ema.jiscinvolve.org/wp/2015/05/05/a-brief-typology-of-feedback-hubs/↩}