Technology Evaluation
Technology Evaluation

This free course is available to start right now. Review the full course description and key learning outcomes and create an account and enrol if you want a free statement of participation.

Free course

Technology Evaluation

2.2 A reluctance to evaluate

We could continue almost ad infinitum with examples of the level of spending on environmental, medical, transport, business technologies, and so on, from around the world. However, it would add little to the point I’m making here, which is that given the scale of the spend, and the claimed or actual impact of technology on our lives and the planet generally, we should want to know whether this vast investment is delivering some value, worth or benefit.

Evidence suggests that the reality of technology is far removed from what is claimed, at least in an organisational and governmental context. Let’s take IT as an example as it is one of the most significant areas of expenditure on technology, as well as one where the claims for the economic and social benefits are greatest. Despite widespread acceptance of the importance and value of the evaluation of IT/IS to allow, for example, prioritisation of investment and checks on investment and economic returns, Brown (2005) cites numerous examples of studies of evaluation spread over a decade where a persistent conclusion ‘... is the continuing relatively unsophisticated and low level of evaluation activity in all types of organisations.’ As a recent example Brown (2005, p. 169) provides information on a study carried out for the Australian federal government of the value achieved from Information and Communication Technologies (ICTs) by organisations that concluded ‘... less than a third of respondents routinely carried out appraisals or post-implementation exercises for ICT investment.’ ‘... organisational practice seems to support the judgement that it [IS/ICT evaluation] is not worth the effort.’

Unfortunately, a body of research also indicates that even where evaluations are undertaken the findings are often either ignored or misused (e.g. Weiss, 1998; Brown, 2005). Amongst the reasons for this can be bad timing, weak methodology and cost. Credibility can also be an issue, with Bamberger et al. (2006) noting that a former Director General of the World Bank’s Operations Evaluation Department had observed that the ‘prerequisite of credibility is missing in the evaluation systems used by most governments, companies and development agencies.’ (Picciotto, 2002, p. 14, quoted in Bamberger, 2006, p. 158, original emphasis). Even more significant, perhaps, is that:

Lack of ownership or investment in evaluation may be an inevitable result when, as is often the case, many stakeholders are never consulted about the objectives or design of evaluation, are not involved in the implementation, and have no opportunity to comment on the findings ... Many stakeholders never even get to see the report.

(Bamberger et al., 2006, p. 158)

I’ll return to a fuller investigation of credibility, ethics and trust in Section 9.

Activity 6 Survey of examples of technology evaluation

This activity is designed to test out some of the points made in this section of the unit. Take some time to recall whether you have witnessed the evaluation of a technology or a technology-based project/programme during the past five years. This could be in an organisation or elsewhere (e.g. as a resident in an area where a technology – such as a wind farm or transmission mast for mobile telephony – has been installed).

If you have an example add brief details to your notes detailing:

a) what was evaluated (e.g. a new piece of software, a vehicle, workstation, etc);

b) how it was used (e.g. to justify further spending on technology);

c) which individuals or groups within the context/organisation were the primary participants in the evaluation (e.g. departmental heads, senior management, politicians, etc).

Whether or not the lack of willingness to undertake post-implementation evaluation of IT/IS applies to other families of technologies, such as industrial plant, transport systems and so on, was not something I’d been able to ascertain when I came to the end of writing this unit. Nor could I ascertain the extent to which the other shortcomings noted above applied to technology evaluation specifically, rather than to evaluation in general. However, it should be possible to address this lack of information as we gather data through Activity 6. I have to admit that given there is no shortage of literature on methods for technology assessment, and that it would appear from published and anecdotal evidence that this form of evaluative activity is commonplace for technology investments, it does seem remiss of management that these ‘findings’, or projections of the benefits to be delivered, are not ‘tested’ by carrying out post-implementation or summative evaluation.

If you have a background in IT you probably have your own views and opinions about this mismatch. From my own experience of technology evaluation and assessment in the public sector I know I do. Many of the possible reasons will become apparent – although often by implication rather than directly – as you work your way through the remainder of this unit. Suffice to note again that many will be contextual rather than anything much to do with a technology, while others will stem from the nature of evaluation itself. Consequently, this is a good point at which to shift our attention to a range of topics and issues that are relevant to evaluation in general. First, however, I want to briefly introduce a case study that draws out a range of important points from the discussion above.

Box 2 Introducing the ID card case study

The UK ID card – the central feature of the National Identity Scheme that was dropped by the coalition government in May 2010 – makes an excellent case study for this unit for the following reasons:

  • it encompassed both technology and policy dimensions;
  • the technologies it would have incorporated had both public and commercial usages;
  • the technology and policy developed, allowing both ex-ante and ex-post evaluation;
  • both the technology and policy, or elements of them, had international significance.

In short, whether you are UK-based or not, and whether you have a background in the public or commercial sector or are just a plain citizen, at least some elements of the case should be of relevance.

That said, it is important to acknowledge that for some non-UK students the debate that was generated in the UK media by the ID card scheme may seem overblown, particularly if you come from a country where ID cards are the norm. The attitude to a compulsory ID card in the UK was and is heavily influenced by deep-seated cultural and ideological biases that regard measures such as compulsory ID as a threat to, and infringement of, individual freedom. To this has to be added a deep distrust amongst many people of ‘big’ government: a situation that was aggravated considerably in 2007 by several high-profile losses of large quantities of citizens’ personal data by government departments and agencies.

Activity 7 Contextual background to the UK ID card

Technology evaluation and assessment in complex, multi-layered contexts such as defence, biotechnology, pharmaceuticals, social policy and energy generation, are particularly challenging. The objective of this activity is to build up a picture of just such a context – the UK ID card and identity scheme.

There is a wealth of information about the ID card and identity scheme in the public domain. For background information and the Government’s position visit the Identity and Passport Service [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)]  and review the material available. For a thoroughly researched assessment, which contains a wealth of information on context, refer to the report by the London School of Economics - The Identity Project: an assessment of the UK Identity Cards Bill and its implications. Other important documents include the Crosby Report and the 2007 Annual Report of the Independent Scheme Assurance Panel (ISAP).

You may also use any other source of your choice.

Pay particular attention to the former government’s stated rationale for the card and scheme, the counter claims of other organisations (e.g. http://www.no2id.net), where appropriate, and the types of technologies that were to be utilised (you may have to infer these, as specific examples may be hard to come by). It would be advisable to download and store any documents you view as it is likely you will need these for further work with the case study.

T887_1

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has over 40 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to university level study, find out more about the types of qualifications we offer, including our entry level Access courses and Certificates.

Not ready for University study then browse over 900 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus