4 The challenges of choosing indicators
However extensive your explorations and rigorous your methodology, ultimately you need to ask if what you are measuring reflects the reality of lived experience. In other words, does observed/measured data equal evidence of impact and change? Secondly, why and to whom does it matter to be able to measure in the first place?
Having clarity as to the sources and meaning of your data is especially pertinent when working with numerical data of any sort. There is something very seductive and attractive about numbers – you can play with them, add them up, create logical and pleasing calculations, and do all sorts of statistical manipulations with them. This is not to deny the power of numbers but to draw attention to our human propensity to regard numerical data as having greater validity and rigour than non-numerical, messy and difficult-to-categorise data. For example, spreadsheets, tables and databases can be derived using records of the number of students attending school, their individual attendance records and their exam grades. This is much more straightforward than gathering verbal accounts from students and teachers on their learning experiences through workshops or other means, and then collating and interpreting them!
In collecting monitoring data, you are likely to use a mix of primary and secondary data, depending on your position in the process and the type of intervention undertaken. As a researcher or project manager of a new health facility, you could be generating the data at first hand through surveys, observations and so on, or have access to someone else’s primary data. This is time-consuming and expensive but has the advantage that the data is tailored to your specific purpose and you know the full details of its sources and means of collection. However, even in this case you may need to draw on secondary sources of information such as government statistics, censuses and data collected by other agencies. With today’s technology, linking of datasets is both feasible and desirable but care is needed to ensure quality and robustness – reliability, soundness of methodology, completeness. You may rely on figures giving you population statistics for your area but how exactly were those numbers derived?
Whether working with primary or secondary data, it is often the case that you cannot easily measure what you are really interested in. This includes the many drivers to human behaviour such as levels of tolerance, capacity for dealing with adversity, levels of dissatisfaction or anxiety. A great deal of effort goes into finding ways of categorising and quantifying such non-numerical data through the use of proxy indicators. In other words, when an activity is not directly measurable, something that is measurable is used to ‘approximate’ to it.
An example of the use of proxies is of those used by Collier (2000) in developing his economic ‘Greed versus Grievance’ model for identifying drivers of conflict. One example illustrates issues that may arise when using proxies.
In Collier’s model:
- average years of schooling (available data) equated with unemployment (data sparse in the contexts under study)
A range of issues arise with use of this indicator. It assumes:
- that a relationship exists between unemployment and propensity to engage in conflict and
- that data on years of schooling is an accurate reflection of lack of economic opportunity or of unemployment.
Furthermore, being educated does not guarantee employment in contexts where opportunities are scarce. Conversely, being uneducated does not equate with unemployment where an informal or seasonal economy operates. Thus, even if unemployment as measured by years of schooling is a valid indicator in this model, its value is undermined by the complexities of social life and the ways in which people adapt to the situations in which they find themselves. Such diversity and creativity is often missed in official records as it may be ‘under the radar’ (invisible to the official eye).