Analyse: Readings

1. Understanding your own context – prompts for analysis

Your context may just involve you as an individual, a department, a faculty or the whole college. These questions are prompts to help you develop your own picture – it will likely change as you continue working in this area .

Your Students

What are their characteristics –in general? Are they ready for some independent learning? Or do they expect to be closely supported? Will this affect your assessment planning? Are they able to use the college systems effectively? Remember students are one of the most under - utilised resources in education. The REAP project contains useful guidance on involving them in assessment practice – such as peer assessment and peer teaching. Remember to think about assessment for learning not just measuring it.

Subject area

Does your subject area have any characteristics that make it more or less likely for you to imagine using e - assessment? Often ideas come after a period of thinking about it – ideally talk to others. For instance at Glasgow the use of e - portfolios for assessment is growing in areas like construction trades. If your subject requires students to write reports and essays then an online submission is a natural progression from paper essays. The use of objective tests / MCQs has wide potential for application – but does require some thought and experimentation and of course considerable up - front investment of your time. Check out the project case studies for examples of solutions people have developed – what worked and what didn’t.

Are there any existing problem areas in your current assessment practice that you would like to use technology to improve, for instance, a high marking load or late feedback to students? The assessment design template that we introduce in the design section of this toolkit can help you record and share your ideas in a simple and consistent way – and you can customise it to suit your own needs.

Teaching Culture

What are the attitudes and values of the lecturers that you work with? As the Jisc guide to Networked Learning observes, the introduction of technology can highlight personal ideas, values and philosophy about teaching and learning in quite unexpected ways (see the illustration from the Jisc guide in section 1 ‘Getting Stated’ about contextual factors). Here’s a real example from a workshop we attended at one of the partner colleges:

“We took a long hard look at ourselves and our teaching. We realised that we had become stale and that we were teaching on the same programme as each other but in isolation – in our own little silos. We were teaching theory first then doing the practical work so the students had no context for the theory. We weren’t happy and neither were our students.

We decided to change the way we worked. Instead of teaching in this disjointed, way we worked together to redesign the curriculum. We moved from teaching by numbers to teaching through projects. This meant changing everything, especially the assessment as we had now merged 4 units together and were doing the assessment for them through the project work. This was much better, the theory was taught in a practical context and could be applied immediately. The students saw the point of the theory and did not have to wait weeks to use the theory they had been taught previously in an abstract manner. This meant getting the unit re-verified. The result? Students are much happier and are getting excellent results and the staff are happier too.”

Technology

Not surprisingly technology is a major factor in the successful use of - assessment. So, as we say elsewhere, your No.1 priority is to find out the technology you college has and learn how to use it. It is especially important to do this early on in the process and not leave it to the last minute. If you are using a technology that only works well in certain web browsers (as is the case with many of the VLEs) you need to know what those web browsers are and if they are supported in your college. You must make no assumptions in these matters and you must find out for yourself and test the technologies regularly. Obviously this is a lot easier if you are working as part of a collaborative group or team and have some technical support.

Central IT services often have policies restricting what technologies that they will allow on college machines and offer support for, like browsers and plugins and versions of programmes such as Microsoft Office, etc.This is why it is wise to start with a pilot exercise that targets formative assessments in order to find out about your local technology and administrative context. Try to find out when upgrades and changes to the IT systems are planned by the college, if there is no policy of communicating this kind of information routinely ask your IT Dept. Make a point of telling your college IT service when your assessments are scheduled and ask them to alert you to any changes during that period. It obviously makes sense to cultivate good relations with your IT department and find someone you can talk to there. Many central IT departments are still coming to terms with e - learning technology as being part of their support remit and actual arrangements on the ground may still be under negotiation. If things do go wrong due to unannounced systems change etc. having a clear electronic ‘paper trail’ of consultation and notification will help make clear where responsibility lies.

As we indicate elsewhere, it is important to find out what your students skills levels are in relation to using college systems (as are those of your teaching colleagues) and take any remedial measures early on.

Learning Technology Support

Leading on from the previous section if you have access to learning technologists you can ask for their support. This can be especially important during setting up and testing an assessment. Aim to make yourself self - sufficient over time.

Administration systems

Your college administrative systems maybe a mixture of different paper and electronic systems and you need to see how this affects your e - assessments over their whole lifecycle. Systems of paper assessments have well - established processes that do not require much engagement on your part. But with e - assessments you are much more likely to need to need to be able to trace the flow of information and be prepared to take steps to intervene. Again this is a good reason for doing pilot exercises to iron these things out.

Quality Systems

In Scotland this will centre on the internal and external verification procedures in relation to SQA qualifications. Your college will have established internal verification (IV) procedures that are there to manage and account for any changes to teaching and assessment. Often this will primarily take the form of paper - based records, although increasingly colleges are moving to using online systems that include simple shared network drive folders or tools like SharePoint, Drupal or even ‘private’ areas within the VLE system that are used solely for administrative functions.

The crucial thing is to record your reasons for changing an assessment and to indicate where and how verifiers can see and examine the evidence of learning produced by your students. We have produced a simple and adaptable design template that should be able to help with this process. Before we leave this area we suggest that it is really worthwhile to agree a naming convention for common items such as module, unit, programme qualification etc. as well as test, mcq assignment, dropboxes etc. Develop a common structure and layout for online learning resources in the VLE and the location of assessments. It is worth thinking about having an agreed glossary for these terms and to get staff to stick to it and publish it in the VLE for the students to refer to as well. It is also worth thinking about having the assessments for a unit in a VLE located consistently in the same place in the online course (it is confusing for lecturers and students alike when they appear all over the place!). These all seem like small things but together then can make yourstudents (and your own) experience much easier.

Institutional Factors Mini Checklist

These factors are much more general and in some cases intangible but can also be the most important, here are some things to consider:

1. Strategy: is there a clear and agreed strategy for the use of learning technology and e-assessment? Is there an implementation plan? (it is not uncommon to have a strategy but no plan for implementing it). Is progress monitored / audited? Are there resources allocated to support this?

2. Academic leadership: We shall be picking up this theme in our Collaborative Frameworks section later. Is there a clear ownership of pedagogic and educational matters at the college or is it scattered across several units? Is there a unit “indicate where and how verifiers can see and examine the evidence” that deals with this and is it led be a senior teaching academic?

3. Morale: What is the staff morale like? This can have a big bearing on the appetite for change and openness to trying new things.

4. Support: Is there adequate resources to support for staff in e -learning / e -assessment? In terms of IT infrastructure and learning technologists. Is there access to training and development inside and outside the college?

5. Finances: The state of the college finances will have a major impact on the other factors and on planning, especially on staffing levels for teaching, learning technology support and equipment and IT infrastructure.

2. Some Typical obstacles

Pain Points

Jisc have produced a really useful review of this area in their Electronic Management of Assessment (EMA): a landscape review and In a section called ‘Pain Points’ they have looked at the way different factors are affecting the take up for e - assessment: 

• Teaching Models

• Technology

• Process (administration etc.)

• Culture (ways of working, reporting, attitudes values etc.)

Their conclusions and observation closely mirror our experience and include :

“The interplay between all of the factors is complex: it is evident that the existing commercial and open source systems do not effectively support all of the existing processes but there are equally some cases where process improvement could clearly be achieved. Similarly,we heard some quite harsh comments about institutional culture but it is clear that experiences with immature or unreliable technologies can turn neutral (or even slightly positive) early adopters into resisters.”

The lack of integration between the VLE and the administration systems is a particularly problematic one and often compounded by the different lines of responsibility and control and resourcing for the VLE, IT and admin systems in many institutions.

Student Skills and Attitudes

Another key barrier are existing student skills and attitudes, together with the preparation and orientation that they may need to undertake e-assessment. We could describe this as the digital literacies needed by students to use the college e-assessment systems effectively.The Heart of Worcester college has produced some award winning development resources to support students. This often comes as a surprise to people new to adopting e-assessment; the assumption is often that the skills problems will be with the lecturers. Despite the considerable commercially biased hype that exaggerates the digital abilities of young people ^[1] the actual research continually paints a very different picture ^[2].

Jisc have produced a guide to developing student literacies that includes the concept of the ‘7 elements of digital literacy’ this is illustrated in the diagram shown below:

[Caption: The Jisc 7 elements of digital literacy Licence CC BY-NC-ND]

As one student observed at City of Glasgow College –‘just because you are good on PlayStation or Facebook does not mean you can use the VLE!’

In our project many lecturers reported problems with students IT skills at a basic level when required to do things like rename files and upload them to a sub mission. This was also identified as problem at the UHI (University of the Highlands and Islands), widely regarded as being at the forefront of e-learning in Scotland, who have produced checklists of the basic skills needed to participate with links to remedial support resource, these are available in the Further Information section of the project website.

Staff skills and attitudes

The Jisc ETNA staff skills surveys of teaching staff in FE in Scotland suggest that there is growing confidence in using computers and common ‘office’ applications. But these surveys showed there is markedly less experience and confidence in using online web - based systems like VLEs and e-Portfolios. This is backed up by the work of the Borders College BOLT project:

(ref: New Media Consortium Horizon Report 2013)

In our project this observation certainly fitted with our experience, we certainly found that lecturers were not confident in using the VLE or e-portfolio systems. This situation is further complicated by the number of online systems lecturers might have to master. The Jisc ElectronicManagement of Assessment (EMA) report observes:

So, we still have a long way to go in terms of system integration and data management, with multiple paper and electronic systems being used, although Jisc are currently researching a feedback hub to improve matters.^[3]

Usability Factors

The actual tools provided in - house by the colleges for lecturers and students to use may be old in terms of technology and design and can suffer from usability issues. To be fair, the actual tasks that the tools are supporting are in some cases quite complex themselves - particularly on the teaching side. However, that said, the usability of the tools is an issue and does affect the take - up of these technologies.Once an institution has adopted these tools there is little appetite for changing them, which in turn reduces the incentives for improvement amongst suppliers, so this situation is unlikely to change quickly. This is why you need to use the tools yourself and experiment with them to get to know them and their limitations. In general the usability issues affect the teachers using the system much more than the students due to the complexity of the tasks in setting up assessments, with date, times and access conditions being particularly problematic - partly due to the terminology used.

It is particularly important to be able to see the systems from a student point of view, in some software products there is a ‘student view’ function (the one in Blackboard is particularly good) but it is also sensible to have some test student accounts to allow you to step through your assessments in the system exactly like a student and to record student data in the system.

---

^{[1] Jisc have produced a useful guide about developing student digital literacy: https://www.jisc.ac.uk/guides/developing-students-digital-literacy↩
[2] http://digitalstudent.jiscinvolve.org/wp/fe-and-skills-digital-student-study/desk-study/↩}

^{[3]http://ema.jiscinvolve.org/wp/2015/05/05/a-brief-typology-of-feedback-hubs/↩}

3. Beginning to Develop Creative and Systematic Solutions

At this stage you should have a good idea of your own working context, and you should have gathered quite a bit of information and analysed it. Most importantly, you should have a clearer idea of the limitations that you face. This might all seem a bit overwhelming but we reckon identifying these factors early on will provide a solid foundation for progress and reducing wasted time and frustration later on.

At this stage you should be well on the way to developing a critical and systematic approach to working in this area and seeking to interpret your findings in order to develop your own analyses of what may be possible and what may be useful to your students and yourself. So, that covers the systematic component needed for developing effective solutions. In the next section we start to explore the creative dimension of developing effective e-assessments.