4.3 Recognising the complexity of evaluation
In practice the post-implementation evaluation of technology may well be far more complex than I have implied above. There is a range of reasons for this, many unrelated to technology, such as the role that personal beliefs and values and organisational and institutional politics can play in the design, application and use of evaluation. I will detail and discuss these later. It is, however, unsurprising that the ‘messiness’ of complexity may arise, because in many cases technologies are implemented as one element of a wider project, programme or development portfolio.
This is particularly so where ICTs are concerned, as these technologies are now routinely embedded in, and regarded as fundamental to, other organisational and institutional developments, such as restructuring, relocation, re-engineering, up-skilling, downsizing, outsourcing, marketisation, privatisation, and many other ‘solutions’ that are claimed to deliver greater economy, efficiency and effectiveness. Complexity, whether real or perceived, may also explain why ex-post and summative evaluations seem to be the least popular form of technology evaluation, at least in the field of IT/IS, as the research by Brown (2005) demonstrates.
Reactions against complexity, or subconscious or deliberate attempts to ignore it, may also explain why the traditional way of dealing with investment in IT/IS as discrete projects or programmes continues to be the approach favoured by some evaluators. As I previously noted, this approach has not been limited to IT: a point borne out by Barrow’s study of the development of environmental impact assessment (EIA) and social impact assessment (SIA) through the 1960s (Barrow, 2000), as well as being reflected in the changing theories of the relationship between technology and society discussed in Part 2 of this block.
In short, until relatively recently the practice and culture of technology evaluation was to stick to the boundaries of the technology or system when assessing or evaluating technology, rather than to recognise (where appropriate) that the scale and scope of a project or programme, and therefore of the systems and structures in which they operate, impact on the technology/system as much, if not more so, than the technology does on them. This clearly illustrates that as evaluations have become more complex, the straightforward dichotomy between formative and summative, or ex-ante and ex-post evaluations no longer holds. Indeed, I would reiterate that if technology is almost always part of a wider social system then the principles and practices, and methods and approaches, of programme evaluation and technology evaluation become interchangeable.
Activity 15 Post implementation and ex-post evaluation
The research reported in this unit on the reluctance of organisations to undertake post implementation/ex-post evaluation is based on the IT/IS sector.
Use any source(s) of your choice to research whether these claims hold for other technologies in other sectors.