Skip to main content
Printable page generated Wednesday, 25 May 2022, 08:45
Use 'Print preview' to check the number of pages and printer settings.
Print functionality varies between browsers.
Printable page generated Wednesday, 25 May 2022, 08:45

T890: Research methods

Unit 4: Research methods

This unit looks at research methods pertinent to technology innovation research. It has not been possible to cover all methods that might be applied to problems in technology innovation research. Instead the unit sets out to introduce and interrogate the application of research methods germane to technology research. As with previous units, a selection of readings will drive the teaching, but in contrast to the first three units where the focus was on understanding technology policy and innovation, we now focus on research designs that facilitate exploration of technology policy and innovation.

A common distinction that is applied to research methods is between quantitative and qualitative methods, although not all researchers think it a consequential one. We have not employed this distinction, but instead chosen papers that explicitly demonstrate their research design and how it meets researcher’s knowledge needs. These papers invariably employ both quantitative and qualitative methods. The research methods chosen are; literature study, case study, structured survey, interview, focus group, participatory approaches, narrative analysis, cost-benefit analysis, scenario methodology and technology foresight.

Learning outcomes

By the end of this unit, you should be able to:

  • assess critically the following methods: literature study, case study, structured surveys, interviews, focus groups, participatory approaches, narrative analysis, cost-benefit analysis, scenario methodology and technology foresight.

  • critically assess research methods pertinent to technology innovation research.

1 Case study

Technology policy analysis and research has historically focused either on issues of promotion or on issues of regulation. Technological promotion can take the form of initiatives to promote a whole technology (like IT or biotechnology, aerospace or renewable energy technologies), an industrial sector (like electrical engineering or construction), a section of industry (like small and medium enterprises), or even a specific firm. It can also take a regional perspective, such as when measures are proposed to address the uneven technological development across a country, or say the European Union.

We have seen in units 2 and 3 that the classic notion of technology policy meaning state policy for technology has broadened considerably – both to encompass the role of different interest groups in shaping public policy, and to take up the increasing use of initiatives that are mixed public/private.

The paper you will study shortly takes up some of these newer and broader issues of promotion of technology in quite an unusual way. The paper suggests, with research evidence, that a seemingly simple concept that academia and industry together can increase innovation, turns out to be much more complex. As an example of promotion, it involves national, regional and local government. It also involves players that are not obviously either public or private but could be either or both at the same time, like universities, as well as a range of companies. Although the study was of UK Science Parks, similar claims have been made for such initiatives all over the world.

The paper has two major parts: one empirical, one concerning re-conceptualization. The part ‘science parks in their own terms’ illustrates the popular conception of UK science parks, and then reports on the empirical research conducted to evaluate the popular conception. A multi-method methodology was adopted. The empirical research involved a comprehensive literature study, including unpublished policy documents (what are called ‘grey’ literature). It also involved a structured questionnaire survey of firms on all UK science parks. Before reading the paper, it is helpful to have an understanding of these methods; a literature study and structured survey.

Activity 25

Please read the following extract from ‘How to do a literature study’ by Stephanie Barrientos in ‘Research skills for policy and development: How to find out fast’ and answer the following question.

Question 17

What is the difference between a literature survey and a literature study?

Comment

A literature survey is a broad sweep through the relevant literature to learn what is available and to begin to set boundaries. After completing the literature survey, a literature study involves a more critical engagement with carefully selected literature from the broader literature survey. This includes examining the arguments in the selected literature and seeking answers to the research question(s) in a methodical way.

Activity 26

The next reading gives a brief description of structured surveys. Read it now and note the different applications of structured surveys and semi-structured interviews, both of which are used in the science park research. The results of the structured survey research (discussed on pp.412-414) cast doubt on the popular conceptualization of science parks and what science parks might achieve. An in-depth study was also undertaken of three different science parks as case studies. The parks studied were chosen because they appeared to be very different. This led to the second part of the paper ‘science parks in our terms’.

Comment

Structured surveys are used to generate descriptive detail or answers to ‘what’ questions. Questions are standardised and normally of the ‘tick-box’ variety for ease of analysis.

Semi-structured interviews are used to generate detailed, complex and varied data or answers to ‘why’ and ‘how’ questions. There is the flexibility to ask supplementary questions to follow up interesting lines of enquiry for the research allowing the development of theories or hypotheses.

Activity 27

This third reading considers the use of case studies as a research tool. Read section 13.1 to get a sense of the advantages of the approach as an introduction to looking in more detail at the science park case studies. Thomas makes the point that choosing case studies is never like sampling, in the sense that it cannot be ‘representative’. Do you agree that the concept of ‘representativeness’ would not be useful if you were choosing case studies to investigate your research area?

The first case study park, Cambridge Science Park, had been established by an elite university college, in a ‘sunrise’ (well considered) location in South East England. The second, Aston, was set up by a technological university on derelict land in an inner-city urban manufacturing heartland – Birmingham. The last, Heriot-Watt, was set up in a vibrant Scottish city, Edinburgh. The in-depth case studies not only looked at the park firms and their characteristics: links with the university – academic/industry links; with other park firms – synergy; links space and position in the regional, national and international economy; industrial sector; whether the firms were new, relocations, independent or parts of larger companies; and so on. The case studies also looked at the rationale for the science parks, their histories, relationship with universities, and so on.

Activity 28

Read the paper ‘Science parks: a concept in science, society and “space” (a realist tale)’. The first part is on pages 412-414. The second part is on pages 414-420.

Question 18

What does it have to say about conceptualization? i.e. what was the popular conceptualization and what did their research reveal?

Question 19

How do the different methods assist the reconceptualization ‘in our terms’?

Question 20

What differences emerge in this specific research project between causation in the initial conceptualization and in ‘our’ reconceptualization?

Comment

Conceptualization is often seen as belonging either to the beginning of the research process (leading to a research design) or to the end (after data are collected). This paper argues that conceptualization and reconceptualization is possible (and required) throughout research. The paper first lays out the popular conception of science parks ‘in their own terms’ by articulating the arguments of science park advocates and practitioners and drawing on the literature. As the paper says,

‘...it is difficult to derive from the policy literature exactly what is expected to occur as a result of this spatial juxtaposition {science park by HEI} but a large amount of textual analysis and a range of interviews enabled us to establish four processes as being the most important. These are: first, the creation of employment...,second, the establishment of new firms, third, the facilitation of links between industry and academe, ...and fourth, the generation of firms operating at very high levels of technological sophistication.’

The questionnaire survey of park firms and a range of interviews (both research tools were discussed in Activity 2), together with documentary evidence (a literature survey including ‘grey literature’ is discussed in Activity 1) cast some serious doubts about this conceptualization. The three case studies allowed further reconceptualization. Each case had been chosen to be spatially, technologically, academically, and economically different. This allowed a reconceptualization of the process of causation of technological development.

The causation in the initial conceptualization concerns what will happen if science parks are set up, namely employment, high technology firm creation and academic-industrial links. The causation and powers suggested in the re-conceptualization are altogether different, more complex and less uniformly positive.

More generally, the idea of reconceptualization can be used to think about the ways that social shaping of technology perspectives have allowed a reconceptualization of the study of technology policy and innovation – bringing in a more nuanced approach than that of the state/market dualism and the linear model of innovation as discussed in previous blocks and particularly in “Mode 2” approaches. The process aspect of policy making and application is brought out clearly in this paper and is a direct critique of prescriptive approaches to policy – that science parks will automatically and necessarily bring high tech employment and force academic industrial synergy. The paper and research is strongly suggestive that innovation requires systems and that successful regional and national systems of innovation require more than science parks. These issues of spatial innovation clustering and innovation systems were discussed in unit 3. Thus, the paper finishes by having articulated some elements of a new conceptual framework for researching science parks.

I want to say a little more about case studies. You may recall the Stephen Potter article ‘Managing the design of an innovative green transport project’ that you studied in unit 1. This also demonstrated the utility of a case study approach. In the science park paper multiple case studies are used. These are chosen carefully to obtain replication and contrast to ensure that the case study is not simply reduced to comparing outcomes. The point is to ‘get inside’ the cases, to see how they ‘work’ and hence to investigate in detail whether the mechanisms at work are the ones envisaged in the theoretical ideas originally put forward.

The next four sections concentrate on getting information from people or the generation of primary data.

2 Interview

The interview is probably the best known way of using people as sources of information, but it is also one of the most difficult. If your knowledge needs are quite straightforward and respondents will not have to think deeply about their answers, then perhaps a questionnaire is the more appropriate tool. However, if your knowledge needs require the generation of complex, nuanced data, requiring some reflection on the part of the respondent, or is likely to include technical terms which may need clarification, then the interview is probably the better tool.

Interviews are either structured or semi-structured, with the latter format providing a more open structure to encourage the respondent to fully develop their answers and scope to pursue interesting tangents.

Activity 29

Download and read section 9.3 of the following document which is an account of the process of conducting interviews with women khannawallis, who make meals for migrant workers, in Bombay, India. The acronym AMM referred to in the reading is the Annapurna Mahila Mandal, a trade union like organisation for the women. Answer the following questions.

Question 21

What factors influenced the selection of women for interview and the number of interviews done with particular women?

Question 22

In what ways did the researcher have to change the format and recording of the interviews in response to the contextual environment?

Question 23

How did the female researcher overcome the difficulty of entry to the male domain of the kholis?

Question 24

What strengths and weaknesses does the researcher see in her interview sample?

Comment

Question 21

The researcher wanted to interview khannawallis from different caste/religion/marital status. She gained access to the largest sample of women through her contact with the AMM. Access to non-AMM women was obtained through a union representative who acted as ‘gate-keeper’ and clients working at the local textile mill who bought meals from the women. The researcher was then able to find other contacts by ‘snowballing’ (initial contacts provide further contacts). Return interviews were conducted with women with whom the researcher had a rapport or who were particularly interesting from a research perspective, for example, the household with three generations of khannawallis.

Question 22

The researcher had to abandon using a small tape recorder to record interviews because it attracted an audience among the khannawallis friends and relatives who ‘helped’ her answer the questions. Not wanting to use a notebook, together with translation difficulties, meant the researcher opted for a semi-structured interview in order to aid her memorising of the responses.

Question 23

The researcher met a young man called Vilas, whose mother is a khannawalli, who was interested in her research and instrumental in helping her gain access to the clients in their kholis. He also acted as her escort.

Question 24

The researcher feels the strengths of the interviews lay in the mix of results that provide unique insights into the relationship between the client-khannawalli relationship. The weaknesses lay in the non-neat client sample which relied on an opportunistic approach to obtaining interviews. But it should be noted that research will always have an element of pragmatic opportunism, where the researcher will need to adapt from what is desirable to what is manageable in the research process.

3 Focus groups

An alternative research approach to the interview is the use of Focus Groups. These are group interviews that last 1-2 hrs. The technique has a long history although mainly in market research. Consequently, its status in academic research has not been high, but it is becoming much more accepted both as an efficient exploratory approach and as a source of primary data for testing hypotheses. The typical size of such groups is 6-10. Their composition is not based on representativeness but on finding segments of the population of interest, likely to provide the most meaningful information. Thus research design will often be based on selected comparisons e.g. if class is an issue, groups would be class based, if gender is an issue, then groups would be all-women and all-men. The number of focus groups you use will be influenced by the degree of difference being explored in the population being studied, and by the need to ensure there is some sense that the group is not atypical of its population subgroup. The focus group process involves non-directive moderators (or facilitators) who concentrate on ensuring the area of interest is covered in the depth required, allowing a self-managing process to develop. Depending on the research goals the moderator may be more or less involved in directing and prompting the discussion to ensure the research areas are addressed.

Continuing to look to people as sources of information, we now consider participatory approaches to research including participant observation and participatory action research.

4 Participation, participant observation and participatory action research

Including people in researching and decision making on issues that affect them is becoming more prevalent at many different scales, from the management of multinational corporations to small-scale development initiatives, and from local government to national and international policy formulation. As a result, research on participation or which takes a participatory approach is becoming increasingly popular amongst researchers and policy makers.

Research often requires thinking about how different groups of people can participate in a process for the best outcome. Participation means the involvement of relevant stakeholders in the decisions and activities that affect them. Stakeholder analysis for a particular process involves making an assessment of who it is likely to effect, and in what way. The advantages of a participatory approach can essentially be seen in terms of either efficiency or ethics. The two views are not exclusive, and it is perfectly possible to hold that participation is both effective and the right thing to do.

The pragmatic argument for participation is that processes which are participatory work better than ones which are not. In other words, because participatory approaches include stakeholders in decision-making or implementation, they offer a better match with the facts-on-the-ground, as well as a sense of ownership on the part of stakeholders, and these results in this view are associated with success. The ethical argument is value-driven and to do with respecting the rights of stakeholders to influence decisions which affect their lives. In this way of thinking, stakeholders must be made partners in the processes that affect them because it is their right to be recognised and respected as autonomous human beings. An ethical view of participation is often associated with an interest in issues of power and empowerment, and a shift is sought from ‘experts on top to experts on tap’ (Gibson, 1996, p. 139).

Taking a participatory approach to research or action requires an appreciation of who could or should potentially participate. Experience generally shows that when changes are proposed to existing situations, the effects of those changes will be different for different people – they will have different stakes in the potential outcomes. Thus a fundamental part of most participatory methodologies is some form of stakeholder analysis. Technology innovation is littered with examples of projects which failed to appreciate meaningful differences between different stakeholders in particular contexts: projects which helped men but not women, or which added to the workload of some groups without acknowledging or compensating for this change, or which marginalised other groups from the decision-making process.

When dealing with different stakeholders, one of the things that distinguishes between them is that they have different viewpoints – they see the world differently. One method to try to climb into the role and view the interests of stakeholders is participant observation.

4.1 Participant observation

Participant observation usually involves spending time at the research site, building up relationships with informants. It can take different forms, for example it could be covert, where the participant does not disclose that he/she is a researcher, but it is usually overt with the participant either having a role working with informants or just observing. There are typically a number of problems a participant observer has to overcome: gaining entry and establishing relationships with informants, finding reliable and well-informed informants, and maintaining the relationships. The demands of doing the daily job or role play can leave little time for research. But the advantages of participant observation are that the researcher begins to see the world from the viewpoint of the group members because day to day the researcher experiences what the group experience.

4.2 Participatory action research

Participatory action research (PAR) means involving stakeholders in the management and/or implementation of a research project which applies theories or methods to an actual problem. Action research is an approach to research that can be used where results are seen as an important outcome of research, as well as knowledge. Put simply, action researchers are interested in testing their theories and methods by applying them to solving actual problems. Action research is therefore particularly attractive to practitioners, for whom it can be a way of refining their knowledge as a result of their engagement with their practice. Participatory action research is action research where the researcher(s) involve other stakeholders in managing and/or implementing the researching process. An example of this might include occasions where agricultural researchers involve farmers in planning field trials of new crops or techniques, or where the researchers begin by finding out what it is that farmers are interested in researching and then help them to do it.

This section has looked at participatory ways of doing research, based on essentially qualitative approaches. Participation is important in technology innovation because of a shift in ethos from doing research on technology to one of doing research with people using technology. It should be clear that there are many considerations when adopting a participatory approach. Perhaps the essential lesson is the importance of maintaining a critical awareness that other participants in the processes being researched may have a very different idea of what is happening, why it is happening and whether it is in their interest or not. Everyone has their own agenda and the agendas of research subjects are as important to them as the objectives of researchers are to themselves. The great strength of participatory methods is that they open this up for scrutiny, and allow researchers and others to find areas and activities where their agendas match and to negotiate in areas where they do not.

5 Policy research including narrative policy analysis

Policy research can concentrate on the policy arena and feed into policymaking, or it can describe policy itself. However, these are ideal types. Policy can be thought of as a form of intention, from a particular point of view. That is, polices are a way of talking about the collective intentions of institutions such as governments. Indicators of a particular policy might include various legislative, fiscal or organisational measures, but these are evidence for policy rather than policy itself.

A simplistic way of looking at policy making is the rational, linear process depicted in Figure 4. Here a problem is identified, options are investigated, a set of alternatives is drawn up and a choice made between them under a set of declared criteria. This is then implemented and hopefully the impact is assessed some time later. Research might play a role in this process during the investigation of policy options and selection between them, as well as any impact assessment.

This model is subject to challenge because, even though research processes and policy processes have influence on one another, it is not true to say that policy gives rise to research or vice-versa. Instead they can be seen as coupled – the influence works in both directions. The rest of this section will develop this more sophisticated understanding of policy and its relationship with research.

Figure 4 A linear model of policymaking
Figure 4 A linear model of policymaking

Public policy is a term used to describe a number of different things. These include:

  • Public Policy as a product: Public Policy is sometimes described in terms of the products which arise from it or which embody it, such as laws, position papers, policy briefings and so on.
  • Public Policy as an activity or process: public Policy has also been characterised as ‘what organisations do’ (Mayers & Bass, 1999, pg. ii).
  • Public Policy as an Intention: A broader way of thinking about policy is to treat it as a form of collective intention, representing the views of public or private sector institutions such as governments, corporations and NGOs. (Increasingly non-state organisations are putting resources into formulating internal and external ‘policies’)

Taking these views of policy allows one to see that a statement that a policy exists is not straightforward. Policy and policymaking processes are complex rather than linear, and there is no direct causal link with policy-oriented research. Research and policy do not generally give rise to one another in a straightforward way. What one can say is that these two complex areas of activity do often have an effect on one another. As researchers, this has implications for how we go about doing research.

The validity of research findings for a particular audience can depend utterly on the methods employed; and policy audiences are no different. The contribution which research might make to a particular policy debate may not be a researcher’s first priority. But, if they are sensitive to how their research is used, the nature of a policy audience is a consideration in research design. A method that can be used to illuminate policy debates and understand the needs of policymakers in complex policy arenas is narrative policy analysis.

5.1 Narrative policy analysis

Narrative policy analysis is a methodology employed to understand complex policy debates where there is uncertainty, complexity and confusion surrounding an issue. The methodology uses tools from literary criticism in order to understand the relationship between different narratives surrounding the issues, and to seek a meta-narrative which suggests a reframing of the situation from which progress can be made by more conventional analysis. As a method for analysing policy it is suitable in situations where conventional policy analysis fails.

As we saw in the Brown and Duguid extract in unit 3, ‘stories’ provide an important basis for research, or narrative analysis. There are many different forms of narrative analysis, and it is becoming an increasingly popular research approach. This is based on the view (Bruner, 1991 quoted in Brown, 1998) that people, faced with the necessity of “constructing and representing the rich and messy domain of human interaction,” tend to “organize [their] experience and [their] memory of human happenings mainly in the form of narrative – stories, excuses, myths, reasons for doing and not doing, and so on”. This may be to make sense of events, as a guide to action, social/cultural, knowledge producing, or political. Roe noted that ‘stories’ may be: scenarios (stories with a beginning, middle and end) or arguments (with premises and conclusions). Narratives may be analysed through discourse analysis of ‘texts’ or through ethnographic study of interview material.

An interesting study using an ethnographic approach to narrative was carried out by Brown (1998). He did a longitudinal study of the implementation of information technology in part of a large acute hospital with the support of an external software company called Delta. This involved a two phase data collection strategy with 11 formal semi-structured interviews being conducted in phase one, and 15 in phase two. Interviews lasted 60-90 minutes and 24 of them were tape recorded. These interviews were complemented by dozens of informal conversations, and a variety of supplementary documentary materials “reducing the possibility of systematic error and incompleteness”.

It became apparent that the different groups involved (the ward, the laboratory, and the implementation team) were telling very different stories (see Table 4). While all groups espoused a common goal of improving patient care in relation to the implementation project, the narratives surfaced latent motivations which were much more concerned with the groups preserving their autonomy and discretion over tasks. Narratives were used not only to make sense of what had happened, but also to legitimise the interests of the different groups. This perspective has much wider applicability and relevance than the context studied.

Table 3 A comparison of the groups’ narratives

The wardThe laboratoryThe project team
Espoused motivationsImprove quality of service to patientsImprove quality of service to ward staff; Exert control over system developmentGain clinical and financial benefits
Latent motivationsSave doctor and nursing staff time while making minimal adjustments to work practicesRetain existing IT systems, retain existing work practicesAdvance careers, and increase the dependency of the hospital on IT under our control
Development processNot understood by usNot conducted in partnership with usDevolved responsibility to Delta and the users
Going liveTraumatic and problematic experienceDifficulties emerged which exposed the limitations of the systemAs expected, the pilot disclosed problems which could then be rectified
System usageElectronic ordering is disruptive to work routines, time-consuming, and unreliable, Electronic results reporting is unworkableProcessing electronic orders is disruptive to normal work patterns, There are additional problems of speed and reliabilityOur deteriorating relationship with Delta and the lab combined with other development problems meant that improvements to the system were not made
DiagnosisFailure of project team to co-ordinate or meet deadlinesFailure of project team to listen, communicate, or meet deadlinesFailure of the ward staff to communicate their needs. Failure of the lab to co-operate. Failure of Delta to write adequate software
Perceived outcomesTime and energy have been wasted on a so-far pointless systemOur perfectly adequate systems have not been improved uponA technically competent system has been produced, but the project may have been misconceived

In this section the complexity of policy and policy processes have been explored. The aim has been to challenge and widen narrow ideas of what policy is, and to think about the relationship of policy with research. An image of policy and research as two interdependent systems has been presented. The section concluded with the presentation of narrative policy analysis, a method that has been applied to issues where the policy arena has been identified as complex and the policy debate as strongly contested and stuck.

6 Quantitative modelling-cost benefit analysis

When deciding whether (and where) to build a road, there are always winners and losers. For example, new roads take up land, may act as barriers between areas, might inflict noise on some people and have a host of other environmental impacts. On the positive side, the road will save time and money for those who use it, and may aid development (i.e. contribute positively to the social and economic milieu). As decisions made about such matters affect a great number of people in many different ways, it seems reasonable to investigate the economic costs and benefits of different options and see where they fall. Cost Benefit Analysis (CBA) developed as a decision-making tool to address these sorts of issues, and allows a researcher to evaluate a selection of possible choices or scenarios.

In CBA, a monetary valuation is assigned to each item or parameter to be studied over the period of time in question. This value is based on expert information – a judgement on the best available information on the cost or benefit of the item in question. Using a relatively straightforward rationale from economics, future costs or benefits are depreciated; that is, future costs are given less weight than present ones. Because all the parameters of each option are given monetary values, everything has to be evaluated on the same basis. It can then be shown whether overall benefits exceed costs, and which option under consideration represents the best value to society (i.e. which has the highest ratio of benefits to costs).

When using CBA, a crucial consideration is that both internal and external costs are usually taken into account. Internal costs are those which are paid by some party (like a consumer buying a carton of milk, or a road-user paying for petrol). External costs, or externalities, are costs that can be assigned, but are not paid for. In the Reading I will shortly ask you to read, car pollution is identified as an external cost. Other costs like this might be acid rain caused by air pollution, the cost of scrapping vehicles, and societal costs, such as the breakdown of communities. As there is no market in such externalities (by definition no-one actually pays for them up front), they are much harder to evaluate than internal costs, where the going rate can be based on what people actually spend.

An example of a problem which can be tackled with CBA is “Which is better overall for society and the environment – using milk bottles made out of glass, or milk cartons made of foil-lined cardboard?” Because of greater cost of manufacturing, glass bottles cost more to begin with. However, a CBA should address each issue discreetly and over its lifetime. In the case of a returnable glass bottle the overall lifetime is longer, so over that period the overall cost is lower than the equivalent number of cartons, which can only be used once each. Conversely, there are some negative costs which need to be taken into account – e.g. bottles need to be collected, washed and re-processed. In the case of the carton, a cost would have to be assigned to disposal – not just the fee for dumping or burning it, but the consequential land or air pollution effects. Depending on which values are attached to which component, either glass or cardboard might appear more beneficial in terms of cost. Thus, in CBA one must track costs and benefits very precisely and ensure that as many as possible relevant factors that could influence the final outcome are included. Failure to do so can dramatically change the results.

CBA has gained popularity as a research method because of its increased use by policy makers, including the various UK governmental departments. In theory, CBA allows a researcher to pose various solutions to a problem and apply established principles to see which solution provides the highest net benefit (or lowest overall cost if the scenarios are all pessimistic). This can give the policy-shaping team a sense of having found a definitive solution – in spite of the method’s sensitivity to the inclusion or exclusion of different factors and to the selection of expert information. CBA is a method which compares different scenarios or options based on an economic valuation of their internal and external costs and benefits. It is popular with policymakers because of its perceived numerical rigour and transparency, but has been criticised on account of the need to monetarise all decision variables and the narrow range of scenarios which are sometimes used.

An important criticism is that CBA methodology requires all factors to be expressed in terms of money. That has led to arguments about how external costs are valued – particularly valuing things like environmental impact, species loss, amenity loss, emissions etc. With the development of CBA, the values placed on external costs have tended to increase as information on environmental costs has improved, and as public awareness of the environment has increased. So for example, the extinction of a species of moth may now be given a much higher value than the time savings of car commuters, even though this may not have been the case in the past. The trend with CBA and derivative methodologies is towards placing more rigorously derived values on such external costs, but there is still much debate about how best to incorporate them, or whether they can be faithfully valued in money terms at all.

A second key criticism is that the range of options considered in a CBA is often narrow. Frequently it was used only to decide between different road building options, and not whether some other approach would maximise societal benefits over societal costs. This is addressed in the Reading by Fellows & Pitfield (2000), where the authors show that managing traffic demand can produce a better CBA result than accepting demand as given and building a new road. So while this is a criticism of the way that CBA is sometimes used, it does not apply to all uses of CBAIn their paper, Fellows & Pitfield consider easing road congestion by examining the effect of increasing car-sharing, using CBA. The geographic setting is the area surrounding Birmingham, the second most populous city in the UK, where congestion causes loss of business income through lost time, and to some extent, wasted fuel. The building of roads (e.g. a major bypass) would alleviate congestion, and the authors compare car-sharing scenarios with the building of an orbital road around Birmingham. Note that the authors are not keen to entertain the idea of an orbital road. They argue against it on the grounds that new roads are too expensive and that increased road growth only leads to more congestion.

In the Reading, Fellows and Pitfield refer to a specific CBA application called COBA. This is the CBA package used by the UK Transport Department to assess road-building schemes. COBA established standard values and procedures to ensure that all road building schemes were evaluated on a common basis, but for some people the way it is designed and has been used is controversial. By using COBA to make their analysis, they have demonstrated their point in terms of the relevant policymaker’s analytical tool.

Activity 30

Now read Fellows & Pitfield (2000), An economic and operational evaluation of urban car-sharing, making notes about the methodology. Pay attention to the following:

  • the use of expert information.
  • which costs have been included and which have been left out (e.g. the cost of traffic noise in the urban environment).
  • the assumptions behind the different scenarios that are evaluated.

Comment

A question raised by the paper, is this: Assuming the authors want to seriously consider car-sharing as an option for reducing urban traffic congestion, what other research methods might they use to triangulate on whether it is desirable? The concept of triangulation suggests that greater confidence can be placed in research findings based on the points of agreement between the results of diverse methods. Triangulation therefore relies on an understanding of the sources of bias embodied in the different methods – their strengths and weaknesses as vehicles for engaging with the research context. We will consider triangulation in unit 5. One possibility would be to do some kind of preferred status study, where people’s preferences are explored through survey and interview. Another option might be to take a case study approach: study existing car-sharing schemes and try and understand how these could be scaled up (if already in existence in Birmingham) or the lessons about implementation transferred (if the case study comes from somewhere else).

CBA is a popular method because it simplifies decisions – it reduces complexity by converting all known factors into economic values and comparing options on a single scale. Because of this, and because decisions are based on figures, it is transparent, at least as far as declaring the basis of the calculations. However, it also has its flaws. Relying on CBA alone can be problematic, because of its sensitivity to the selection of relevant factors, and the assignments of values based on expert information. Furthermore, not all relevant factors are easy to value, and some would argue that there are things which simply cannot be valued in money terms.

7 Scenario methodology

The paper you will shortly read by Robin Roy and Steve Potter, on the use of scenarios to identify innovation priorities in the UK railway industry, is largely self-explanatory, as it was specifically written to explore the method of scenario building. Although this is based upon a specific engineering sector and the specific circumstances of the (then) approaching rail privatisation, the lessons contained are entirely generic and could be applied to any area of technology foresight. Indeed it can be applied to any exploration of ‘futures’ quite aside from their technological or innovation aspects.

Activity 31

Please read Potter, S and Roy, R (2000) ‘Using scenarios to identify innovation priorities in the UK railway industry’, International Journal of Innovation Management, 4, pp 229-252.

As is noted in the paper, scenarios are particularly useful as a way of exploring a range of possible futures, and to focus attention on what could cause (or is needed for) such a future to come about. Scenario methodology has another major advantage over some methods - it can be participatory. It is flexible enough to be used both by small groups of experts and planners, but can also be used to engage rail commuters, for example, and those who might rail commute if changes were made to attract them.

Scenarios are good at identifying the answers to the following sorts of questions:

  • Are there key ‘stepping stones’ towards that future? (i.e. stages on the way);
  • Are there key barriers (e.g. might there need to be legislative change for one of those ‘stepping stones’ to be achieved);
  • Are there key thresholds? – (e.g. the cost of alternative clean fuels would have to drop below that of existing ‘dirty’ fuels);
  • Are there key ‘actors’ – people or organisations who need to back the path towards a particular scenario?
  • Are there key stimuli that might make this scenario more likely to come about?

The paper provides details of the four scenarios that were developed. These scenarios were developed by first undertaking a literature search and taking the results to a workshop of experts covering both rail marketing and technical expertise. The final scenarios emerged from this process. The paper finishes with a consideration of how rail markets actually developed in the first five years of privatisation, and reflects on the usefulness of a scenario methodology in this context.

Activity 32 Scenario Exercise

The following is an exercise that is best done as a group of 4 – 6 people, although it is perfectly possible to do it on your own. It is designed to be completed in just over an hour and allows for a final presentation to tutors and/or to other groups of students doing the same exercise.

A very useful function of this exercise is to help you quickly explore the usefulness of a scenario approach for your own research topic or one on which you may be interested in conducting research later. This is a ‘quick and dirty’ scenario scoping exercise.

Stage 1 (5 minutes)

Choose a topic area for which you want to explore possible design, technology, development or environment futures (or some other factor as discussed above). It needs to be one about which you or the group is reasonably familiar. Possible topics might be cars, shopping, housing, health and education.

The viewpoint should be from that of a company or organisation providing a product or service and seeking to explore what may affect its position in these markets. For example, a supermarket chain as retail stores; Toyota or Ford as a car manufacturer, Save the Children or a national health service for children’s health, a waste recycling non-governmental organization for local environmental waste systems.

Timescale: 15 years

Stage 2 (10 minutes)

Think about what have been the major recent factors and trends in determining the design of the product, service or system) in the last 15 years.

“Starter” ideas are:
CARSHEALTHSHOPPING
AdaptabilityLifestyle changes Smaller households/changes in family structures and demographicsConcentration on larger shops
In car equipmentChildcare systemsPrice competition
New meanings of ‘performance’Distances from formal health clinicsSpecialist retailers
Leasing or other new ways To get the use of a carAvailable financial resourcesService quality
Environmental concernsShifting from a chore to a ‘leisure experience’

Discuss what have been the really important factors and trends that have affected the way the product/service is designed. Have any technologies or factors been crucial to the way in which it has developed? Have any of these trends required a change in managing the design and delivery of the product/service?

Stage 3 (30 minutes)

Having looked at past trends, start thinking about the future. Think of different ways that the demand/market for these product and service areas could possibly develop in the next 15 years. Some questions to ask would be:

  • Are there ways the product’s function could be fulfilled in a different way?
  • Are there any important trends emerging that could force these products and services to develop in a different way to the past?
  • Are there any key technologies likely to produce a change in the way the end-service is delivered?
  • Are there any ‘invading’ technologies that might suit the future better than adapting existing technologies? If so, how might they be introduced? What would be the pioneering applications?

For example: environmental performance of cars will have to be vastly improved; environmental performance of transport as a whole will have to be vastly improved; might shared car systems be developed for cities; will out of town shopping face big planning restrictions; home deliveries for basics/concentration on leisure shopping; teleshopping; would vaccination campaigns improve child health more than provision of more local health clinics; would resources for child health better be concentrated on women’s literacy, etc.

Brainstorm all sorts of possibilities (on Post Its) and group into 3 or 4 scenarios.

Stage 4 (10 minutes)

Sensitivity Analysis (to what particular factors will scenarios be sensitive)

  • What sort of technologies, systems and designs etc. would be crucial for each scenario to come about? (Exact technologies need not be specified – but an idea of what they should do – e.g. nature of IT development – IT widening choice/price competition for shopping, providing in-car guidance systems), or overall infrastructure for child health in the poorest countries;
  • Are any of these common to more than one scenario?
  • Are there any factors that are crucial to a scenario coming about? (e.g. government policies to give alternative fuels a price advantage)
  • Will the scenario depend upon the action of any particular organisation?
  • Do you feel that any of the scenarios have an equal chance of coming about, or some are more likely than others?
  • If you were in charge of a company operating in this sector, what would be your development priorities or strategy?

Stage 5 (10 minute preparation)

Report back on your findings (10 minute presentation).

Describe your scenario via telling your audience of a journey, or a ‘shopping experience’ in 15 years time under this scenario (e.g. as a telephone conversation to a friend or your mother), or an agencies description of a child health care system in 2020. Adapt this if you have chosen a different product/service area.

You could split up and get a couple of people each in your group to prepare each of these.

Comment on any insights you gained from this exercise, both in terms of the process and your chosen topic.

Comment

One unusual aspect of this paper is that the assessment of the scenario building is undertaken by the builders themselves. It is unusual and positive to find researchers looking critically at their own work. Obviously, this is a different type of evaluation to one undertaken by external assessors – ideally both are important but often neither get done.

This method can be relatively more participative than others, in that a broad range of social actors can be actively involved if they can be mobilised and encouraged – this mobilisation is not always done well and those less expert groups get missed out or alienated. In theory, however, it is, in principle, a relatively more participative method than some other forecasting and foresight methods. But perhaps you should assess that statement after the next section!

8 Technology foresight

Many governments have introduced long range ‘technology foresight’ capacity, looking ahead perhaps 20 years. In the past there had been some enthusiasm for ‘technological forecasting’, but this technique, which often relied on extrapolating trends into the future, was increasingly seen as too mechanistic and unreliable, given the ever more complex pattern of technological, economic and political developments. The technology foresight approach, by contrast, attempts to gather views from a wide range of experts and practitioners on likely patterns of technical and market developments and refine them, interactively, to produce a consensus on the prospects for specific new technologies or lines of scientific development. It has become more process oriented and less prescriptive. It is not just technical possibilities that are considered: patterns of social and economic change may also be considered, e.g. via long range scenarios, so that potential future market demands can be identified.

But how can investors ‘pick winners’ from the wide range of new ideas that continually emerge? And how can governments identify and plan for appropriate patterns of technological development? One approach that has become popular in recent years is Technology Foresight. To explore the potential significance of emerging technologies foresight studies make use of specific techniques such as the iterative Delphi opinion refining technique. For this technique, a wide range of experts are asked their opinion about when some particular technological development may mature and are then asked to revise their views in response to the views of the others. This can lead to a more robust consensus view. In addition, use is often made of scenarios of plausible patterns of socio-economic development to provide a context for an exploration of possible technological lines of development. These are sometimes quantitative, and sometimes based on simple extrapolation of existing trends. They can also be ‘normative’ – attempting to capture social and cultural values in qualitative narratives of hopefully internally consistent patterns of development.

Technology foresight can bring together the technology policy and technology shaping approaches to the study of science, technology and innovation.

Conclusions

This unit looked at research methods pertinent to technology policy and technology innovation. It considered a literature study, case study, structured surveys, interviewing, focus groups, narrative analysis, cost-benefit analysis, scenario methodology and technology foresight. While not exhaustive of research methods, these tools were found to have greatest utility for technology research. The unit has considered both qualitative and quantitative methods. Although quantitative studies are often treated as more objective, even in strongly positivist terms, their accuracy depends on the assumptions on which they are founded. Qualitative methods were found to have particular utility when engaged in research that may have a policy impact. In unit 5 we reflect on the particular skills necessary to deploy these research methods to investigate technology policy and innovation.

References

Brown, AD (1998) Narrative, Politics and Legitimacy in an IT Implementation, Journal of Management Studies, Vol 35 Issue 1 pp35-60.
Fellows, N.T. & Pitfield, D.E. (2000) ‘An economic evaluation of urban car-sharing’ in Transportation research Part D no 5: pp. 1-10.
Gibson, T. (1996) The Power in our Hands, Charlbury, Jon Carpenter Publishing
Mayers, J & Bass, S (1999) ‘Policies that Work for Forests and people’, IIED, London.
Potter, S and Roy, R (2000) ‘Using scenarios to identify innovation priorities in the UK railway industry’, International Journal of Innovation Management, 4, pp 229-252.
Thomas, A, Chataway, J. And Wuyts, M. (1998) ‘Finding out Fast. Investigative skills for policy and development.Open University Press, Milton Keynes.