Technology Evaluation
Technology Evaluation

This free course is available to start right now. Review the full course description and key learning outcomes and create an account and enrol if you want a free statement of participation.

Free course

Technology Evaluation

4.4 Beyond the formative–summative divide

4.4.1 Process evaluation

The argument that there is a fundamental distinction between formative and summative evaluation – which is most often attributed to Scriven in 1967 (Clarke, 1999) – and of the usefulness of maintaining a dichotomy between them, has been challenged as our knowledge and experience of the actual practice of evaluation has increased. Recognising this, most of the leading scholars of traditional programme evaluation have adopted a more nuanced approach to defining different types of evaluation. Weiss, for example, concludes that a distinction between the study of process(es) and the study of outcome(s) ‘... is a more useful distinction for many purposes’ (1998, p. 33) than the summative–formative dichotomy.

Weiss suggests that there are at least three situations where process evaluation is relevant. Most obvious is where our key questions concern process, such as where an evaluation’s sponsors want to know what is going on. Another is when we ‘want to be sure what the outcomes were outcomes of’. This may sound a bit obvious, but given the contested nature of the findings of many evaluations it pays to be doubly sure that there is a clear and transparent relationship between claims for the relationship between process(es) and outcome(s), particularly when drawing on findings from an evaluation by a third party. The fundamental point here is that credibility of, and trust in, the evaluation (and those carrying it out) are at stake, both of which are features of evaluation that I discuss more fully below. Third is where our aim is to find out which specific features of a particular technology, project or policy are associated with particular outcomes. For example, were certain features of a technology more clearly associated with successful outcomes of its implementation than others?

Here I follow Clarke’s example and briefly summarise Chen’s (1996) suggestion that there are four basic types of process evaluation.

Process-improvement evaluation

Here the emphasis is on identifying the strengths and weaknesses of the processes of a programme or project, and making recommendations for adjusting the structure and/or implementation of it. An important point to note here is that the information generated by an evaluation of this kind can be used either instrumentally or conceptually (Clarke, 1999). That is, the information can be used to make changes to the specific programme or project that is under review, and may also be useful in wider debates about the feasibility of a technology project or programme.

Process-assessment evaluation

The aim here is to study whether a programme or project has been implemented successfully. This type of evaluation is also referred to as an implementation study.

Outcome-improvement evaluation

The emphasis here is on how the various components of a programme or project affect its outcomes. In other words, ‘this type of evaluation attempts to identify which elements are more successful at producing desired outcomes’ (Clarke, 1999, p. 13). Because no analysis is undertaken of the overall effectiveness or impact of the project or programme, this approach does not constitute summative evaluation.

Outcome-assessment evaluation

Providing an overall judgment of the outcomes of a project or programme in terms of its benefits, merits and worth is the primary purpose here.

T887_1

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has over 40 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to university level study, find out more about the types of qualifications we offer, including our entry level Access courses and Certificates.

Not ready for University study then browse over 900 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus