3 Understanding success in IT systems
At the beginning of this course, we briefly defined success and failure before looking in greater detail at the sociotechnical and evolutionary nature of IT systems, and at tools from systems thinking to help you analyse IT systems. It is now time to return to a key question introduced at the start of this course: if this is a course about the success of IT systems, what do we mean by success? What evidence is there about IT success and what models exist to better understand successful systems?
A consultancy firm, The Standish Group, has used surveys to monitor IT project performance since 1994. Most of their data is not openly available but using figures published by a number of different authors in various journals and white papers Nasir and Sahibuddin (2011) have been able to summarise the Standish findings for 1994 to 2008. These are shown in Table 1, where figures for 2010 and 2012, taken directly from Standish (2013), have also been added. Succeeded is defined as ‘delivered on-time, on-budget, with required features and functions’; challenged as ‘late, over budget, and/or with less [sic] than the required features and functions’; and failed as ‘cancelled prior to completion or delivered and never used’ (Standish, 2013, p. 1).
| Year | 1994 | 1996 | 1998 | 2000 | 2003 | 2004 | 2006 | 2008 | 2010 | 2012 |
| Succeeded (%) | 16 | 27 | 26 | 28 | 34 | 29 | 35 | 32 | 37 | 39 |
| Challenged (%) | 53 | 33 | 46 | 49 | 51 | 53 | 46 | 44 | 42 | 43 |
| Failed (%) | 31 | 40 | 28 | 23 | 15 | 18 | 19 | 24 | 21 | 18 |
As you can see, there are some grounds for optimism in Table 1 in that the proportion succeeding was increasing during the period represented but nevertheless the picture painted is not rosy.
A similar impression is created by the work of Flyvberg and Budzier (2011). They looked at 1471 ‘large IT projects … that ran the gamut from enterprise resource planning to management information and customer relationship management systems’ (p. 24) and compared their budgets and estimated performance benefits with the actual costs and results. Their conclusion was that the average cost overrun was 27%.
It is not only survey data that shows the difficulties of building successful systems. It is true that the media prefers reporting bad news – it is claimed (see, for example, Williams, 2010) that the ratio of bad news stories to good news stories is around 17 to 1 – but it is also the case that where systems failure is concerned there is plenty to report. As discussed earlier, in 2008 the BBC launched its Digital Media Initiative to build a system that would allow all its audio and visual material (new and archived) to be made available to staff making programmes via a desktop. Between 2010 and 2012 £98.4 million was spent on the project and in May 2013 the initiative was cancelled. At the very same time the media were reporting problems being experienced by the Co-operative Bank, one of which related to its IT system:
The Manchester-based organisation’s [i.e. the Co-operative Bank’s] finances may face extra strain in the next few months from more writedowns on a defunct IT system, according to sources. The Co-op has already written off £150 million on a system that it did not use. Some have warned that figure could get much bigger.
In September 2013 the National Audit Office (NAO) reported an investigation into plans by the Department for Work and Pensions to roll-out universal credit across the UK whereby benefits and tax credits to people who are unemployed or on low incomes are merged together into a single monthly payment. This is one aspect of what they found:
Over 70 per cent of the £425 million spent to date has been on IT systems. The Department, however, has already written off £34 million of its new IT systems and does not yet know if they will support national roll-out. The existing systems offer limited functionality. For instance, the current IT system lacks a component to identify potentially fraudulent claims so that the Department has to rely on multiple manual checks on claims and payments. Such checks will not be feasible or adequate once the system is running nationally. Problems with the IT system have delayed national roll-out of the programme.
Of course, a lot of systems do work. Admittedly some need tweaking and others need more major remedial work before they are judged to be successful but there are those that meet the goal of ‘right first time’. Overall though, evidence suggests that ‘system design and building’ ought to be placed in the high-risk category of endeavour. The next topic we are going to look at lies right at the heart of this question and, indeed, the course’s title: what do we mean by ‘a successful system’?