1.3 Assessment or evaluation?
Somewhat confusingly, technology evaluation is often referred to as technology assessment. This is confusing because the purpose of the two is different, as the discussion in Section 4.1, below, demonstrates. While in practice the two activities may overlap and inform each other, they need not do so and can be two discrete activities. Consequently, in this unit we maintain a distinction between the two where this is appropriate, although the term technology evaluation will be used where the material under discussion is applicable to both technology evaluation and assessment.
Depending on whether an evaluation is focused on the production of a technology, its operation or its impact/output, the first evaluative activity is likely to be technology assessment or ex-ante evaluation. This is making judgements about the potential value of developing and implementing technologies or systems of technologies. The aim in almost all cases will be to choose the ‘best’ technology/system to invest in to deliver ‘benefits’ (e.g. efficiency or effectiveness) to an individual, group, organisation or, as with the example of the Roman army’s short sword, a whole country, state or empire. In other words, and as Braun (1998, p. 1) succinctly puts it: ‘The purpose of technology assessment is to look beyond the immediately obvious and analyse the ramifications of a given technology in as wide-ranging and far-sighted a manner as possible.’
Return to the material you produced for Activities 1 and 2. Now think through the extent to which you undertook an assessment or ex-ante evaluation of your chosen technology before purchasing it. That is, what assumptions and judgements of the future benefits (or costs) of the technology did you make and on what did you base these?
You may, of course, have carried out this process implicitly (without really thinking about it), in which case you may now have no memory that you did it. Alternatively you may have done it explicitly (making a conscious decision to look for information) and either informally (by listening to the views and opinions of other people over time) or formally (by collecting published reviews and information). Note down which of these it was and related information.
Try also to recall how influential these processes were when you actually came to purchase the product. For example, did you stick to your assessment or did some other factor – such as the sales pitch of a salesperson – change your assessment?
Keep your notes from this activity as you will need them later.
Technology evaluation may take several forms and employ a variety of methods, such as testing and monitoring specific components, technologies or systems against either expected or defined norms, such as speed, accuracy and portability. This data can then be used in any process of further refinement and development of a technology or system. This process is often labelled formative evaluation. The aim of summative evaluation, and its closely related cousin, ex-post evaluation, meanwhile, is to assess what the impact or outcome has been of the application or implementation of a technology or technological system for users, organisations and so on. As a commonly used adage states, formative evaluation is what cooks do when they taste a dish they are preparing, whereas a summative evaluation is what the diners do when they taste the final dish.
Table 1, below, sets out the types and stages at which the forms of assessment and evaluation are most usually deployed. The spare cells in the table are deliberate as we shall return to this table later in the unit when I discuss additional forms of evaluation.
Table 1 Types and applications of evaluation and assessment
|Types of evaluation and assessment|
Return to the product you used as the basis for Activities 1–3. Using the definitions above, note down how many forms of evaluation you have applied to that product.
- Have you, for example, carried out some form of ex-post or summative evaluation of the product? If not, or if you have done this implicitly, do it explicitly now. What is the outcome?
- Has the product ‘delivered’ the benefits you identified for Activity 1?
- Which contextual or technical factors identified for Activity 2 have proved most valuable (i.e. of benefit)?
- Did these confirm your assessment in Activity 3? If not, why not?
The four activities undertaken so far had two primary purposes. First, to get you into an evaluative frame of mind. Second, to get you thinking about the extent to which you might naturally be an implicit or explicit evaluator.
A further purpose was to flag that different forms of evaluation and assessment exist, with differing functions. A comprehensive discussion that elaborates fully on the brief overview above is included in Section 4.
My first task is, however, to introduce a dimension of Table 1 that is fundamental to any form of evaluation and assessment – context.