Technology Evaluation
Technology Evaluation

This free course is available to start right now. Review the full course description and key learning outcomes and create an account and enrol if you want a free statement of participation.

Free course

Technology Evaluation

4.3 Recognising the complexity of evaluation

In practice the post-implementation evaluation of technology may well be far more complex than I have implied above. There is a range of reasons for this, many unrelated to technology, such as the role that personal beliefs and values and organisational and institutional politics can play in the design, application and use of evaluation. I will detail and discuss these later. It is, however, unsurprising that the ‘messiness’ of complexity may arise, because in many cases technologies are implemented as one element of a wider project, programme or development portfolio.

This is particularly so where ICTs are concerned, as these technologies are now routinely embedded in, and regarded as fundamental to, other organisational and institutional developments, such as restructuring, relocation, re-engineering, up-skilling, downsizing, outsourcing, marketisation, privatisation, and many other ‘solutions’ that are claimed to deliver greater economy, efficiency and effectiveness. Complexity, whether real or perceived, may also explain why ex-post and summative evaluations seem to be the least popular form of technology evaluation, at least in the field of IT/IS, as the research by Brown (2005) demonstrates.

Reactions against complexity, or subconscious or deliberate attempts to ignore it, may also explain why the traditional way of dealing with investment in IT/IS as discrete projects or programmes continues to be the approach favoured by some evaluators. As I previously noted, this approach has not been limited to IT: a point borne out by Barrow’s study of the development of environmental impact assessment (EIA) and social impact assessment (SIA) through the 1960s (Barrow, 2000), as well as being reflected in the changing theories of the relationship between technology and society discussed in Part 2 of this block.

In short, until relatively recently the practice and culture of technology evaluation was to stick to the boundaries of the technology or system when assessing or evaluating technology, rather than to recognise (where appropriate) that the scale and scope of a project or programme, and therefore of the systems and structures in which they operate, impact on the technology/system as much, if not more so, than the technology does on them. This clearly illustrates that as evaluations have become more complex, the straightforward dichotomy between formative and summative, or ex-ante and ex-post evaluations no longer holds. Indeed, I would reiterate that if technology is almost always part of a wider social system then the principles and practices, and methods and approaches, of programme evaluation and technology evaluation become interchangeable.

Activity 15 Post implementation and ex-post evaluation

The research reported in this unit on the reluctance of organisations to undertake post implementation/ex-post evaluation is based on the IT/IS sector.

Use any source(s) of your choice to research whether these claims hold for other technologies in other sectors.

T887_1

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has over 40 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to university level study, find out more about the types of qualifications we offer, including our entry level Access courses and Certificates.

Not ready for University study then browse over 900 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus