4.4 Beyond the formative–summative divide
4.4.1 Process evaluation
The argument that there is a fundamental distinction between formative and summative evaluation – which is most often attributed to Scriven in 1967 (Clarke, 1999) – and of the usefulness of maintaining a dichotomy between them, has been challenged as our knowledge and experience of the actual practice of evaluation has increased. Recognising this, most of the leading scholars of traditional programme evaluation have adopted a more nuanced approach to defining different types of evaluation. Weiss, for example, concludes that a distinction between the study of process(es) and the study of outcome(s) ‘... is a more useful distinction for many purposes’ (1998, p. 33) than the summative–formative dichotomy.
Weiss suggests that there are at least three situations where process evaluation is relevant. Most obvious is where our key questions concern process, such as where an evaluation’s sponsors want to know what is going on. Another is when we ‘want to be sure what the outcomes were outcomes of’. This may sound a bit obvious, but given the contested nature of the findings of many evaluations it pays to be doubly sure that there is a clear and transparent relationship between claims for the relationship between process(es) and outcome(s), particularly when drawing on findings from an evaluation by a third party. The fundamental point here is that credibility of, and trust in, the evaluation (and those carrying it out) are at stake, both of which are features of evaluation that I discuss more fully below. Third is where our aim is to find out which specific features of a particular technology, project or policy are associated with particular outcomes. For example, were certain features of a technology more clearly associated with successful outcomes of its implementation than others?
Here I follow Clarke’s example and briefly summarise Chen’s (1996) suggestion that there are four basic types of process evaluation.
Here the emphasis is on identifying the strengths and weaknesses of the processes of a programme or project, and making recommendations for adjusting the structure and/or implementation of it. An important point to note here is that the information generated by an evaluation of this kind can be used either instrumentally or conceptually (Clarke, 1999). That is, the information can be used to make changes to the specific programme or project that is under review, and may also be useful in wider debates about the feasibility of a technology project or programme.
The aim here is to study whether a programme or project has been implemented successfully. This type of evaluation is also referred to as an implementation study.
The emphasis here is on how the various components of a programme or project affect its outcomes. In other words, ‘this type of evaluation attempts to identify which elements are more successful at producing desired outcomes’ (Clarke, 1999, p. 13). Because no analysis is undertaken of the overall effectiveness or impact of the project or programme, this approach does not constitute summative evaluation.
Providing an overall judgment of the outcomes of a project or programme in terms of its benefits, merits and worth is the primary purpose here.