w360_2 Carrying out research for policy and advocacy work About this free course This free course is an adapted extract from the Open University course W360 Justice in action - https://www.open.ac.uk/courses/qualifications/details/w360. This version of the content may include video, images and interactive content that may not be optimised for your device. You can experience this free course as it was originally designed on OpenLearn, the home of free learning from The Open University – There you’ll also be able to track your progress via your activity record, which you can use to demonstrate your learning.

First published 2024. Unless otherwise stated, copyright © 2024 The Open University, all rights reserved. Intellectual property Unless otherwise stated, this resource is released under the terms of the Creative Commons Licence v4.0 http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en. Within that The Open University interprets this licence in the following way: www.open.edu/openlearn/about-openlearn/frequently-asked-questions-on-openlearn. Copyright and rights falling outside the terms of the Creative Commons Licence are retained or controlled by The Open University. Please read the full text before using any of the content. We believe the primary barrier to accessing high-quality educational experiences is cost, which is why we aim to publish as much free content as possible under an open licence. If it proves difficult to release content under our preferred Creative Commons licence (e.g. because we can’t afford or gain the clearances or find suitable alternatives), we will still release the materials for free under a personal end-user licence. This is because the learning experience will always be the same high quality offering and that should always be seen as positive – even if at times the licensing is different to Creative Commons. When using the content you must attribute us (The Open University) (the OU) and any identified author in accordance with the terms of the Creative Commons Licence. The Acknowledgements section is used to list, amongst other things, third party (Proprietary), licensed content which is not subject to Creative Commons licensing. Proprietary content must be used (retained) intact and in context to the content at all times. The Acknowledgements section is also used to bring to your attention any other Special Restrictions which may apply to the content. For example there may be times when the Creative Commons Non-Commercial Sharealike licence does not apply to any of the content even if owned by us (The Open University). In these instances, unless stated otherwise, the content may be used for personal and non-commercial use. We have also identified as Proprietary other material included in the content which is not subject to Creative Commons Licence. These are OU logos, trading names and may extend to certain photographic and video images and sound recordings and any other material as may be brought to your attention. Unauthorised use of any of the content may constitute a breach of the terms and conditions and/or intellectual property laws. We reserve the right to alter, amend or bring to an end any terms and conditions provided here without notice. All rights falling outside the terms of the Creative Commons licence are retained or controlled by The Open University. Head of Intellectual Property, The Open University Policy research involves investigating the current operation of an area of the law in order to inform, improve or update the law itself or how it is used in practice. This is important so that individuals can easily understand their legal rights and how to enforce them.

Figure 1 Carrying out applied policy research Figure 1 shows a photograph of a woman sitting at a desk in front of a laptop, writing in a notebook.

Joanna Otterburn, Chief Executive of the Law Commission of England and Wales highlighted this when she said: I think there are other aspects to access to justice, beyond the importance of advice and representation, that are additionally important. One of those is the work of reforming the law so that when that individual comes to navigate the justice system, they find that they can understand the basic rules in play, that their rights under it are meaningful, and that the system makes sense. (Otterburn, 2023) Law reform can be prompted by many different motives: to improve a law or regulation which does not work effectively, provide rules for situations not currently covered by the law, remove out-of-date law, simplify the law or make it easier for the public to understand and use. Whatever the reason for the change in the law, policy research provides the information or evidence to support the proposal, which makes it more persuasive and authoritative. This course will explain how to carry out effective policy research, including designing your methodology, researching ethically, collating your data, analysing it and presenting it in a written report format. This OpenLearn course is an adapted extract from the Open University course W360 Justice in action. After studying this course, you should be able to: explain the role of policy research design research and recognise the importance of research ethics carry out a range of different types of empirical research analyse data collected present evidence effectively in a written report. Policy research involves researching an area of law and analysing the data in order to provide law or policymakers with evidenced-based recommendations for law reform or changes to regulations.

Figure 2 Many organisations carry out applied policy research Figure 2 shows three square wooden blocks standing in a line on grass. The block on the left has a picture of the world being held by two hands, with two leaves on it, and the word ‘environmental’. The middle block has a picture of the world surrounded by, and connected by lines to, three people with the word ‘social’. The block on the right has a picture of the world surrounded by cog wheels being held by two hands, and the word ‘governance’.

Policy research is often carried out within an academic setting, but is also an expanding and growing career sector. Policy and research officers are employed in many different organisations, including third-sector organisations such as charities and non-governmental organisations (NGOs), think tanks, consultancies and central and local government departments. Activity 1 will enable you to discover more about the types of careers involving policy research. Activity 1 A day in the life of a policy researcher You should allow yourself 25 minutes to do this activity. Watch the following video from 13.28 – 18.54 in which Anvar Sarygulov, formerly Senior Research Fellow at the think tank Bright Blue (and now Research Grants and Programmes Manager at the Nuffield Foundation) discusses the type of work he does. Which part of his role do you find the most interesting, and why? Video 1 Starting a career in policy research (Youtube link: The Open University is not responsible for external content) Anvar discussed many different tasks that he carries out as a senior researcher, including research (qualitative research), data analysis, communication for different audiences, attending events and building a media profile. Which of these you find interesting will depend on your skills and interests. For example, communicating clearly to different audiences can be interesting, as you need to think about how best to communicate the message you want to give using plain English and relatable examples. As Anvar discusses in the video, policy research can take many forms, and you will look at these later in the section. However, when starting your policy research, it is important to have a clear aim or research question which will then guide your research and analysis. You will learn more about this in the next two sections: research ethics and research methodology. ‘Research ethics are the moral principles that govern how researchers should carry out their work’ (Wellcome Trust, 2014, p. 1). Research ethics ensures that the risk posed by research to others (such as colleagues, stakeholders, research participants, society or the environment) is considered and addressed where possible or mitigation put in place. For all research carried out within a university setting, proposals have to be approved by external objective individuals or committees to ensure that appropriate consideration has been given to the ethics of your proposal.

Figure 3 Research ethics govern how policy research is carried out Figure 3 shows an illustration of a clipboard with the word ‘ethics’ on it, surrounded by a mobile phone, calculator, notes, bar chart and clasped hands.

Research ethics began after the Second World War as a reaction to the revelations of the atrocities committed by Nazi doctors experimenting on prisoners in concentration camps. In 1964, the World Medical Association adopted a declaration on research ethics called the Declaration of Helsinki. This set out ethical rules for carrying out research on humans, prioritising the wellbeing of the individuals above all other interests (European Commission, 2013, p. 3). Subsequently, there have been ethical rules published for research involving human participants in all fields of study.

Figure 4 Henrietta Lacks Henrietta Lacks

An example of unethical research is the use of HeLa cells in medical research. Henrietta Lacks suffered from cervical cancer and underwent a biopsy in 1951 (Nature, 2020). Some of the cells taken were subsequently grown successfully in culture, and this cell line became known as HeLa (taking the first letters from her name). They are still one of the most commonly used cell lines, with millions of cells being grown in laboratories around the world (Amon and Koenen, 2022, p. 3). However, Henrietta did not know about the use of her cells in research, did not consent to it, and the cells were not anonymised. When her family discovered the existence and use of these cells many years later, it led to a number of legal battles about the use of the cells in commercial and patented processes.

Ethical rules and principles are designed to ‘protect the dignity, rights and welfare of research participants’ (WHO, no date). While the details may vary slightly between different codes, the basic ethical principles remain the same. In the next activity, you will think about how these ethical principles can be applied to the research you will carry out in your project. Activity 2 Ethical principles on research projects You should allow yourself 25 minutes to do this activity. Watch the following video where Helen Kara outlines the basic principles of research ethics. Make notes on the key points she discusses in the video. Video 2 Why is research ethics so important? [MUSIC PLAYING] HELEN KARA Hello. I want to pose a question today. And the question is, why is research ethics important? We hear a lot about research ethics. People talk a lot about it, but nobody ever seems to stop to ask, why is it so important? So that’s what I want to think about and talk about with you today. Ethics can be very instrumental, very much a means to an end for researchers to get their ethics application approved. And then they’ll say ‘I’ve done ethics’, and they’ll crack on and do their research without ever thinking about ethics again. But this, to me, isn’t real research ethics. And in my view, real research ethics is essential. So let me explain what I think real research ethics is. It’s not about filling in a form. It’s not about getting a stamp of approval from a committee of other people. It’s not about ethical theories that people talk about with very long words like ‘deontology’ and ‘consequentialism.’ What research ethics is really about, in my view – and actually, this is based on worldwide research done by the Trust Project, so it’s not just my view – research ethics is about respect, honesty, fairness, and care. The Trust Project was an EU-funded project that was run from the University of Central Lancashire here in the UK, and they asked people in different parts of the world what matters to you in research ethics. And these four points came up again and again and again. People said we want respect, honesty, fairness, and care, and not just here and there, but all the way through the research project and even beyond into aftercare, because a significant amount of research around the world has been conducted unethically and is being conducted unethically today. It’s not a problem of the past. It’s often taught as though it’s a problem of the past, but actually, it’s a problem of the present. There is unethical research practice happening in many parts of the world. I wish that was not a real thing, but it is. Having said that, it’s also the case that a lot of research is conducted very ethically, and whether directly or indirectly, sets out to create benefit. Research is often done to try to improve things – to find treatments for diseases, to find ways to help people who don’t have clean water to get clean water, to find answers to problems that we have in the world like the climate crisis. And that, of course, is ethical in itself. And then if that research is also conducted with respect, honesty, fairness, and care, that is truly ethical research. So, for me, research ethics is about making sure that this happens at all levels all the way through the research process. Of course, some of the four aspects are more relevant at particular stages in research. For example, when you’re designing a research project, you need to use more care to design that as carefully as you can. And then when you’re working with other people in the course of your research – if you’re collecting primary data, you might be working with participants, gatekeepers, and so on – then respect is very important in the way that you interact with those people. But overall, if we, as much as we can, practise respect, honesty, fairness, and care throughout the process of conducting research, then our research will be ethical. [MUSIC PLAYING]

In the video, Helen Kara summarises the outcome of a worldwide research project, the Trust project. This identified that real research ethics involves research being carried out with respect, honesty, fairness and care. These four principles should be considered all the way through the project and beyond. However, different principles may be more important at some stages than at others. For example, it is important to design the research carefully. However, when working with other people, there is also a need for respect. Helen noted that today a lot of research is being carried out unethically, but prioritising the principles of respect, honesty, fairness and care will make sure your research is carried out in an ethical way. Kara identified respect, honesty, fairness and care as being the overarching ethical principles. Consider the following ‘golden rules’ for carrying out policy research. For each golden rule, which of the main ethical principles apply? There may be more than one answer for each statement, as indicated next to each question below.

When you are carrying out policy projects involving research with human participants, there are typical issues you will need to consider. These include: the impact of taking part in the research, and any risks to the participants ensuring the questions you ask are relevant to the research and are not too broad the information you will give to the participants about the research obtaining informed consent from the participants ensuring the participants’ anonymity keeping all data obtained from the participants secure and compliant with data protection laws. You will need to check with your organisation whether there is a formal ethics approval process you need to follow before starting your research.

Figure 5 The ongoing process of research Figure 5 shows a diagram with a central box headed ‘ethics’. Around this central box are four other boxes. From the top of the central box is an arrow that points at a box above, which is headed ‘methodology’. An arrow leads to a box on the right, headed ‘research’. An arrow leads to a box underneath, headed ‘analysis’. An arrow leads to a box on the left, headed ‘writing the report’.

It is also important to remember that research ethics is an ongoing process and does not end with planning of your research and any ethical approval, as shown in Figure 5. As you carry out your research, you have a responsibility to ensure you continue to abide by ethical principles. Remember the four key principles: respect, honesty, fairness and care. You will now consider how you will use these four key principles while carrying out your policy research. Activity 3 Reflecting on research ethics You should allow yourself 15 minutes to do this activity. Reflect on how you will ensure your research is carried out ethically. How will the overarching principles of respect, honesty, fairness and care influence or change the way you approach your policy project? Each person’s reflections will depend on the project they are carrying out, and so will be different. You may have considered how the need to ensure respect will influence the way you will work with your fellow researchers or any other participants in the project. In order to be honest, you will need to make sure you record any data or findings that are contrary to the recommendations you may want to make. Fairness means that you need to make sure you approach your research collection objectively, rather than focusing on collecting data that may support your conclusions. To research carefully means making sure you design the project carefully and record any data carefully to avoid mistakes.

Figure 6 What do you want to find out? Figure 6 shows an illustration of a woman holding a large question mark. She is surrounded by four other smaller people who are asking her questions. Two of them have a smaller question mark over their head.

Your research methodology starts with your research question: what is it that you want to find out? Identifying the question you want to answer and writing it down will help you to decide what information you need to answer it. For example, imagine you want to contribute to the Law Commission’s review of the law on financial settlements after divorce. Your research question will determine what methods you use, as can be seen in Table 1. Table 1 Different research methods

Research question	Research methods
Does the public understand the law on financial settlements?	Survey for the public, focus groups
Do the courts use the principle of equality when dividing assets after divorce?	Review of a sample of court decisions through court reports, survey of family law solicitors and judges
How often do divorced couples agree financial arrangements after divorce, instead of court proceedings?	Statistical information on number of divorces compared to number of court proceedings

There are a variety of different research methods you can use to obtain information, and you will need to choose the most appropriate for your project. In your final report, you will need to ‘explain why [you] are using a particular method or technique and why [you] are not using others so that research results are capable of being evaluated either by the researcher himself or by others’ (Kothari, 2004, p. 8). To do this, there are four important principles you need to consider when planning your research, and these are explored in the next activity. Activity 4 Important principles when designing a research project You should allow yourself 10 minutes to do this activity.

The following research methods are commonly used in applied policy research. Doctrinal research uses legal databases and journals to find out what the law is and the analysis of that law by academics and legal professionals. Statistical research looks at data from publicly available data sets, such as the Office for National Statistics or government websites. Freedom of information (FOI) requests are used by researchers to collate information from public bodies. The public body must respond to the request within 20 working days. Documentary research involves reading, analysing and interpreting documents (written text, images, oral recordings and virtual sources, such as information from the internet). Comparative research finds out what laws and regulations exist on an issue in other countries. A survey collects information or data from stakeholders through an online survey. Please note that permission will not be granted for contact with those with lived experience of the topic under discussion. You will normally be expected to make contact only with organisations and professionals working in the area of law. Interviews and focus groups are recorded meetings with one person (an interview) or a small group of people (a focus group) in order to obtain information and views on an issue. Note that permission will not be granted for contact with those with lived experience of the topic under discussion. You will normally be expected to make contact only with organisations and professionals working in the area of law.

Figure 7 Research can be carried out in different ways Figure 7 shows four images arranged in a grid. They depict different ways of carrying out research. The top-left image is a row of law reports in a library. The top-right image is a collection of reports containing statistics, bar charts and pie charts. A magnifying glass sits on top of the reports. The bottom-left image is a cartoon of an iPad and mobile phone displaying boxes to tick, and the caption ‘online survey’. The bottom-right image is a cartoon of four individuals sitting around a table discussing something; one person has a laptop open and is typing.

These research methods are divided into quantitative and qualitative research. Quantitative research involves the generation of data in a way that can be ‘objectively’ measured with numbers – for example, through numerical comparisons and statistical analysis. Qualitative research involves the collection of data that is subjective, such as the participants’ attitudes, opinions and behaviour, which has to be interpreted by the researcher. As a researcher, you therefore need to be aware of your underlying motivations and your implicit knowledge, beliefs and traditions in order not to unduly influence the data you are collating. You will now consider the different research methods listed above and decide whether they are quantitative or qualitative. Activity 5 Quantitative or qualitative? You should allow yourself 10 minutes to do this activity.

Before starting your research, it is helpful to write down your proposed methodology in a research plan. The research method you use will depend on your research question and the information you need to be able to answer it. You may find that as you consider the research methods to be used you realise your research question is too broad and needs to be amended or narrowed. This is a normal part of designing research. There are advantages and disadvantages to different research methods, which may also influence your choice. Your research also needs to be realistic, both in terms of its scope and timing. Can you complete the research in the timescales available and leave sufficient time to analyse your data and write your report? If your research is too broad, you can reduce its scope by limiting its timescale, by geography or by using sampling. Alternatively, you may need to refine your research question to look at only part of the original proposal.

Your project may require you to carry out empirical research in order to obtain data and information.

Figure 8 Much information can be found online for empirical research Figure 8 shows a photograph of someone typing on a laptop; only their hands and arms are visible. Overlaid on top of the photo is a graphic of charts and graphs.

This section will discuss a number of different research methods. However, all projects are likely to start by conducting a literature review to find out what has already been published about the area of law.

A literature review is a very specific type of documentary research, focusing on what is already known about the issue you are researching. It is very important that the literature is searched properly so that you know what research has already been done.

Figure 9 A literature review involves finding out what is already written about a topic Figure 9 is a photograph of a number of open books stacked on top of each other, with coloured bookmarks indicating certain pages.

There are two kinds of literature reviews: A traditional literature review is where you search for articles and use the footnotes to find others on the same topic. A systematic literature review is where you create keywords and search databases to find all of the relevant articles. It is very important that you keep notes of the databases searched, the date, the number of articles produced, and how many were relevant. Coming up with keywords is important to direct your searches for the literature. Once you have looked at the available literature, you will need to write this up as a literature review for your final report. In policy research, your literature review is likely to start with a descriptive section, which explains what the law is. This description will be based on primary sources of law (the statute or cases you are using). You will then use the literature review to critically analyse what the scholarly sources have to say about it. You can organise your literature review in different ways. A descriptive approach (Figure 10) describes the law with reference to what other scholars say about it, thereby addressing the legal framework and the critical analysis in one. It focuses on describing to the reader what is known about the topic and its major elements.

Figure 10 A descriptive literature review describes the current law Figure 10 is a note with the heading ‘1. Descriptive’. The text says: Human Rights Act 1998. s3 Interpretation of legislation. (1) So far as is possible to do so, primary legislation and subordinate legislation must be read and given effect in a way which is compatible with the Convention rights.’ A speech bubble at the top states ‘Bloggs says that … but Patel disagrees with this interpretation’. A speech bubble at the bottom says: ‘Levine compares this with … while MacIver takes the position that …’

A chronological approach (Figure 11) examines what the sources have had to say on a law over time, and this can be particularly useful when examining changes in the law.

Figure 11 A chronological literature review describes changes in the law over time Figure 11 is a note with the heading ‘2. Chronological’. The text says: ‘Before – R v Jogee, R v Ruddock (2016) – After. A speech bubble pointing to the text Before says: ‘R v Chan Wing-Siu [1985] set the rule as … The doctrine was crystallised in R v Powell; R v English [1999] as … Smith explains the rule was problematic because … Analyses of convictions showed …’ A speech bubble pointing to the case names R v Jogee, R v Ruddock says ‘These cases changes the law in two ways … Mansour argues that …’. A speech bubble pointing to the text After says ‘The following appeal cases demonstrate that  … Subsequent research by Holmes et al. demonstrates further …’

A thematic approach (Figure 12) identifies important themes and discusses what the sources have to say about each one in turn.

Figure 12 A thematic literature review describes important themes Figure 12 is a note with the heading ‘3. Thematic’. Underneath the heading is a mind map. The central box contains the words ‘What motivates youths to join gangs’. Leading from this central box are four more boxes. The first contains the words ‘Gang attributes. How do they know they are in a gang?’ The second contains the words ‘Male and female reasons’. The third contains the words ‘Socio-psychological: (Sense of belonging)’. The fourth says ‘Instrumentalist/Materialist reasons’. This fourth box has two further boxes leading from it. The first contains the words ‘Reduce risk of victimisation by other gangs’. The second contains the words ‘No other opportunities’.

You can find out more information on literature reviews in Section 3 of the OpenLearn course Introducing research in law and beyond. In the sections that follow, you will consider various research methods which may be new to you.

The Freedom of Information Act 2000 gives the public access to information held by public authorities. Public authorities must publish information about their activities, and members of the public are entitled to request information from them. This makes a freedom of information (FOI) request a valuable tool for policy researchers who want to obtain information from a public body.

Figure 13 Freedom of information requests are useful tools for policy researchers Figure 13 shows a yellow envelope with the words ‘freedom of information’ printed across it.

The following activity will introduce you to FOI requests. Activity 6 How to access information from a public authority You should allow yourself 20 minutes to do this activity. Read the Information Commissioner’s Office article ‘How to access information from a public authority’. Then answer the following multiple choice questions.

When writing surveys or questionnaires, it is important to carefully design your questions so that they obtain the information you want and are clear and unambiguous. You need to make each word count!

Figure 14 Online surveys need to be carefully designed Figure 14 is a cartoon of an iPad and mobile phone displaying boxes to tick, and the caption ‘online survey’.

There are a number of different types of questions you can ask. Closed questions (questions that require a yes or no) are more likely to be answered by participants but may not give you the information you need. Using a Likert scale (excellent, good, neither good nor bad, bad, terrible) can often provide richer data and a more nuanced understanding of an issue. When using a Likert scale, it is best practice to provide five or seven possible answers with the same number of positive and negative responses. You may also need to offer an ‘other’ response as you may not anticipate all possible answers. Open questions (where you provide a box for participants to provide their own answer) give the richest source of data but are less likely to be answered by participants because they require more work. You should only collate personal information (e.g. age, ethnicity, disability, gender) if it is needed for the research, and participants should have the option to opt out of these questions. Your survey should start with information about what you are asking, and why, and whether the survey is anonymous. You may obtain different responses to the survey if it is not anonymous. Your first question will be a closed question asking the participant to consent to take part in the research. You are more likely to get a response to a survey if it is short and takes no more than 2–5 minutes to complete. If possible, you should always test the survey to ensure it is clear and unambiguous before releasing it to the participants. In the next activity you will have the opportunity to design a survey. Activity 7 Designing survey questions You should allow yourself 30 minutes to do this activity. Read the blog post ‘How to create an effective survey in 15 simple tips’ (Fisher, 2023) and imagine you are creating a survey to find out whether people read the terms and conditions when shopping online. Design a question that could form part of the survey. When designing effective surveys, it is important to always remember the purpose of your survey, and to keep it short and simple. Questions should be asked one at a time using simple, clear language that the people completing the survey will understand. The survey should be phrased in a neutral way. Check again if your survey does this; if so, it is likely to be an effective survey.

A research interview or focus group is a good way of obtaining in-depth information, as most questions will be open questions.

Figure 15 A focus group allows the researcher to obtain in-depth information from participants Figure 15 is a cartoon of four individuals sitting around a table discussing something; one person has a laptop open and is typing.

There are three ways you can carry out research interviews or focus groups: An unstructured interview is where there is a conversation rather than the interviewer asking questions prepared in advance. A semi-structured interview is where the interviewer has a list of questions or key points to be covered and works through them in a methodical manner. This means that each interviewee is asked similar questions to ensure consistency and enable comparison. However, the interviewer can ask supplementary questions to clarify or find out more about a particular issue. A structured interview is where the interviewer asks specific questions with a fixed range of answers (similar to a survey). You should now read The Open University’s guide to conducting an interview, up to and including the section headed ‘Transcription’. Remember that if you carry out a semi-structured or structured interview, you will need to prepare an interview plan in advance with the proposed questions. All interviews should start by giving the participant information about the interview and your research, and obtaining their informed consent to taking part in the session.

Your policy project may require you to collect information from documents, usually those that have already been published, such as information from the websites of a public body or other organisations.

Figure 16 Documentary research can be carried out online, as well as from books Figure 16 is a cartoon of a laptop screen and mobile phone. The laptop is downloading a document, while the mobile phone is displaying a document.

When using documents, you need to assess their context, credibility and interpretation (Tight, 2021). You also need to decide in advance the information you are looking for and how to record it on a template or proforma. This will enable you to easily compare information from multiple sources and analyse it. Using a spreadsheet can help to sort and filter the information you collect. You can find out how to use spreadsheets in Microsoft’s guide ‘Basic tasks in Excel’.

Once you have collected your information or data, you will need to analyse it in order to come up with recommendations for future law reform. There are a number of different ways you can analyse your data, and depending on the type of information you have available, you may need to use more than one form of analysis.

Figure 17 There are different ways of analysing information Figure 17 shows a diagram. A central circle is labelled ‘analysis’. There are six circles in an arch formation surrounding the top of the central circle, each of which has one word in it. From the left, going clockwise, these read ‘statistical’, ‘comparative’, ‘thematic’, ‘process’, ‘impact’ and ‘cost–benefit’.

Statistical evaluation involves examining the statistics or numbers you have collated, to see whether there are any relevant or significant patterns. Comparative evaluation is similar to statistical evaluation and involves comparing information from different places (such as different international laws, publicly available information from websites or responses to FOI requests) to establish patterns, casual relationships, or highlight differences. Thematic analysis involves establishing whether there are any themes in the information (e.g. existing analysis of the law from the literature review or responses to surveys or interviews). Process evaluation focuses on how well the law or regulation is being applied in practice and what can be done to improve this. Impact evaluation, in contrast to process evaluation, asks whether the law has achieved the intended impact as set out originally by the government, legislators or proposers of the law. To undertake this kind of evaluation, you need to establish what the desired impact of the law was originally, what the impact has been in practice on the target population, and whether this impact was intended or unintended. Cost–benefit analysis is a comparison of the cost of implementing the law compared to the benefits received from it. In practice, you are unlikely to carry out this type of evaluation in policy research. The rest of this section will now focus on two of these types of analysis which may be new to you: statistical analysis and thematic analysis.

There are examples every day in the news or social media where statistics are used to make a claim that seems improbable. When using or interpreting statistics, it is important to take care to check that the information is reliable and presented in a clear and unbiased way. To do this, you need to understand the context of the information, how it was collected and the limits of what it shows.

Figure 18 Analysing statistics involves careful checking of the data Figure 18 is a collection of reports containing statistics, bar charts and pie charts. A magnifying glass sits on top of the reports.

A good starting point is identifying which of the two forms of statistics you are looking at (or wish to use). Descriptive statistics describes and summarises the data and what it shows – for example, a current trend. Inferential statistics uses the data to make predictions about the future (e.g. future trends) or takes the data collected and applies it to a larger population to draw conclusions. When looking at statistics, the following questions will help you to understand their context and reliability: What is the statistical sample used? Usually, larger samples are more reliable than smaller samples. Where has the information come from, and is the source reliable? Does the visual representation use clear scales and labels? For example, is the same scale used on graphs, particularly when comparing information? Have the statistics been independently reviewed? Could the author be biased, and have they mistakenly or deliberately used the statistics to support a particular viewpoint? Has a causal link been established if this is being claimed? A correlation between two sets of data does not necessarily mean there is a causal connection between them (The Open University, no date a).

Thematic analysis is a common way of analysing qualitative data, such as responses to surveys, interviews and focus groups. It was developed by Braun and Clarke (2006) and is ‘the process of identifying patterns or themes within qualitative data’ (Maguire and Delahunt, 2017, p. 3352). Within policy research, the aim is to identify themes within the data and use those themes to make recommendations for future law reform.

Figure 19 Thematic analysis involves identifying common themes among the data Figure 19 shows a close-up photograph of a computer screen visualising a complex information network. On the right is a person’s hand with a finger pointing at a particular point of data on the information network.

Braun and Clarke’s method (Braun and Clarke, 2006) has six stages: Become familiar with the data: read the transcripts, survey responses or other data collected several times. Generate codes: highlight in the text where the same issue is being raised by different respondents by using a different code (colour, letter, number) for each separate issue. Generate themes: consider the codes identified above and identify the higher-level patterns which ‘captures something significant or interesting about the data and/or research question’ (Maguire and Delahunt, 2017, p. 3356). Review the themes: gather together all of the information about each theme (e.g. by using cut and paste) and make sure the themes are coherent, relevant and distinct from each other. Define and name the themes: ensure the names of your themes accurately reflect them and think about how the themes relate to one another Locate exemplars: when writing up the analysis, you need to tell the story of the themes, with relevant examples. Please read the following worked example of thematic analysis in the following article, available through a general internet search: Maguire, M. and Delahunt, B. (2017) ‘Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars’, All Ireland Journal of Higher Education, 3, p.3351.

Whichever form of analysis you use, when deciding on your conclusions and recommendations, it is likely that the choice you make will be influenced by your underlying values. Often, law students are told that justice is merely the product of the application of neutral rules (Aiken, cited in Curran, 2007, p. 127). However, the policymaker or lawmaker will have prioritised one or more values in choosing how to develop the law; and in making recommendations for law reform, you will also be prioritising a set of values. It is, therefore, important to reflect on and articulate the values lying behind your recommendations.

Figure 20 Your core values influence many things, including the recommendations you make in policy research Figure 20 is a photograph of a series of interlocking cogs with the following words engraved on them: ‘reliability’, ‘core values’, ‘trust’, ‘innovation’, ‘teamwork’, ‘ethics’.

You will now consider a possible reform of the Employment Tribunal’s rules on fees, to identify the values that may lie behind different recommendations. Activity 8 Identifying the values behind a recommendation You should allow yourself 30 minutes to do this activity. Imagine you work as a policy researcher in a think-tank focusing on employment law. A fee payable by all applicants was introduced for employment tribunal claims in 2013. This was subsequently found to be unlawful by the Supreme Court,R (Unison) v Lord Chancellor [2017] UKSC 51. and the fee was removed. In 2020, there were reports that the government intended to re-introduce fees for employment tribunal claims (Farragher, 2020), and your manager thinks the issue is likely to be raised again in Parliament soon. In preparation for this, you have been asked to make a recommendation about whether there should be a fee for employment tribunal claims. For each of the possible recommendations (green cards), identify the value (yellow cards) influencing them by clicking on the cards to match them. This activity shows that the proposal for law reform will differ depending on which value the researcher decides to prioritise. It is therefore important to think about your own values, which may influence the recommendations you make in your policy project. You will now think about your own core values and how they might influence your policy research. What are your core values that may be relevant to your project and influence your final recommendations? Make a note of them below. The values you identify will depend on the values that are important to you. When you start to analyse your findings and make your recommendations, look again at your answer to this question and reflect on whether your stated values remain important to your final recommendations or whether you are taking into account other values. When analysing your data, you need to be aware of the values influencing your recommendations and articulate them in your report. The final section will look at how you write the final report.

Following your analysis, you will need to write up your findings and recommendations in a final report. You will need to write in plain English and avoid jargon. However, the writing style is still formal and you should avoid emotive language (unless it is a direct quote from a participant).

Figure 21 The last stage of applied policy research is writing the final report Figure 21 is a photograph of a woman sitting at a desk, going through the pages of a report. Only the arms and torso of the woman are visible. Also on the desk is a calculator and a number of sheets of paper with graphs printed on them.

Typically your report will include the following: an executive summary the research question and methodology used a summary of the literature review a summary of the research carried out, any information found and their relevance to the issue an analysis of the research and its implications suggestions for reform and any practical considerations. As part of your recommendations you can suggest further research is needed. You must continue to research ethically while writing your report and continue to be guided by the principles you learnt about in Section 2. You will now have an opportunity to reflect on this further in the next activity. Activity 9 Research ethics and writing the report You should allow yourself 15 minutes to do this activity. Your duty to research ethically continues when writing your report. How might the duty to act with respect, honesty, fairness and care affect the way you write up your final report? When writing the report, you must behave honestly, and so you cannot report something that does not have sufficient evidence to back it up. You also have to include and account for counter-evidence (that is, evidence that contradicts your primary finding or recommendations). Finally, you will need to include a statement of the limitations of the research overall.

Generally, you will not include in your report all of the data you have found during your research. You will need to select from the raw data what is worth presenting; think about how it relates to your analysis and recommendations and how you can tell the story of your research in the best way.

Figure 22 You can present your data either in words or through charts and images Figure 22 shows an illustration of a note (left) side-by-side with a pie chart (right). The note on the left is a written extract and states: ‘One landlord reported that he would not rent a property to a dog owner due to concerns about possible complaints from neighbours about noise nuisance and dog excrement’. The example on the right is a pie chart titled ‘Percentage of landlords who would rent to a pet owner’. The pie chart has four different coloured sections (blue, pink, green and yellow). There is a key on the right linking the colours to the answers. ‘No’ answers are in pink and the pink section of the pie chart has the value 45% on it. The ‘yes’ answers are in green and the green section of the pie chart has the value 10% on it. The ‘maybe’ answers are in yellow and the yellow section of the pie chart has the value 15% on it. The ‘depends on the animal’ answers are in blue and the blue section of the pie chart has the value 30% on it.

You will also need to think about what format to present your data in to convey an effective message. Usually, this will be a mixture of text and visual representations such as graphs or tables. Quantitative data (such as numbers and statistics) are often represented by charts, which can be easily designed in Word. Qualitative data is usually represented in text but can also be represented in images. In this final activity you will have the opportunity to think about the most effective way of presenting data. Activity 10 Presenting your data effectively You should allow yourself 40 minutes to do this activity. You have been researching the use of online court hearings during and after the COVID-19 pandemic lockdowns in England and Wales. You have found statistics on the number of hearings conducted online and face-to-face, and some quotes from people who have attended remote hearings. You have been asked to decide on an effective way to present the information in your final report. Table 2 Hearings conducted in 2022 and 2023 by jurisdiction and format (%)The Bar Council, (2024)

	Civil courts		Crown courts		Family courts		Magistrates courts
	2022	2023	2022	2023	2022	2023	2022	2023
In person (%)	45	79	66	80	20	65	83	84
Online (%)	55	21	34	20	80	35	17	16

Using one of the suggestions from Microsoft’s ‘Add a chart to your document in Word’ guidance, or from Auraria Library’s ‘Data Visualization: Quantitative vs. Qualitative’ page, design a visual representation which would effectively present the information in Table 2 in a report. Using one of the suggestions from Auraria Library’s ‘Data Visualization: Quantitative vs. Qualitative’ page, design something which would effectively present the quotes below in a report. These quotes are from people who attended remote hearings in England and Wales between 2020 and 2023. What if my technology didn’t work? How are we going to speak? How is it going to run? What were the rules, because you could all talk over each other easily? I tried to get some information on audio hearings but there wasn’t any. There was no real talk of ground rules because everybody else seemed to know what was going on, so they just started. I didn’t know what was going on. One guy spoke most so I guessed that was the judge, but it was hard to follow, especially at the start. I could see my barrister on the screen but there was no way to talk to him, so I just watched. I was a spectator really and that just carried on while I stood watching. (Clark, 2021) You could present this information in a number of different ways. The statistical information on the number of in-person and remote hearings could have been presented in a graph, chart or table. The quotes could have been depicted by a word cloud, image, icon, timeline or colour-coded. Was your example clear, easy to understand and visually appealing? If so, you may want to use this method in your final written report to present the data you collect. If it was not effective, think about the reasons for this: it will help you to decide on another format to present information if needed in your final report. You have now learnt about all of the different stages of carrying out policy research. We hope you enjoy putting these skills into practice in your project.

In this course you have learnt how to carry out research for policy and advocacy work. You have considered the role of policy research and the careers associated with it, and you have reflected on the importance of research ethics and how it impacts on the design of your research. You have also looked at how to carry out a literature review and different ways to carry out empirical research, as well as analysing the data collected. Finally, you have considered how to write the final report and present your research effectively. This OpenLearn course is an adapted extract from the Open University course W360 Justice in action. If you want to explore further and find out more about the issues discussed in this course, you may find the following of interest. For more information on working as a policy researcher, see Prospects’ ‘Policy officer’ information. For more information on Henrietta Lacks, see Nature’s 2020 editorial ‘Henrietta Lacks’ in the Encyclopaedia Britannica. For more information on research ethics, read Resnik, D. (2020) ‘What Is Ethics in Research & Why Is It Important?’. For more information on research methodology, you may find the free OpenLearn course Understanding different research perspectives interesting. The full course takes nine hours to complete, but you can look at the individual sections on research ethics, research questions, research design and research methodology. For more information on the advantages and disadvantages of quantitative and qualitative research, see Chukwuemeka, E.S. (2023) ‘Advantages And Disadvantages of Quantitative and Qualitative Research’. For more information on documentary research, see ‘An Introduction to Documentary Research’ on the American Educational Research Association website. For more information on using surveys, see ‘What to know about creating good survey questions’ on the SurveyMonkey website. For more information on descriptive statistics and inferential statistics, see the Statistics How To website, which discusses descriptive statistics and inferential statistics in more detail. For more information on working with statistics, see the free OpenLearn course ‘More working with charts, graphs and tables’. For more information on writing a report, see the University of Wollongong’s Policy research advice, and The Writing Centre of the University of North Carolina at Chapel Hill’s ‘Policy briefs’. For more information on writing an executive summary see Harvard Kennedy School’s guidance ‘How to Write an Executive Summary’ For more examples of images depicting qualitative data, see the Royal Geographic Society’s ‘A Guide to Presenting Qualitative Data’. Books Kothari, C.R. (2004) Research methodology: methods and techniques. New Delhi: New Age International Ltd Publishers. Available at: https://ebookcentral.proquest.com/lib/open/detail.action?docID=431524 (Accessed: 12 December 2023). Tight, M. (2019) Documentary research in the social sciences. London: SAGE Publications Ltd. Articles Braun, V. and Clarke, V. (2006) ‘Using thematic analysis in psychology’, Qualitative Research in Psychology, 3, pp. 77–101. Curran, L. (2007) ‘University law clinics and their value in undertaking client-centred law reform to provide advice for clients’ experiences’, International Journal of Clinical Legal Education, 12, pp. 105–130. Available at: https://doi.org/10.19164/ijcle.v12i0.73 (Accessed: 8 December 2023). Maguire, M. and Delahunt, B. (2017) ‘Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars’, All Ireland Journal of Higher Education, 3, p. 3351. Other sources Amon, J. and Koenen, A. (2022) The most commonly used immortal cell lines: an introduction. Available at: https://asset-downloads.zeiss.com/catalogs/download/mic/39ef6d5a-aa58-461f-9a16-f4bd4729bca1/EN_wp_Commonly-used-immortal-cell-lines.pdf (Accessed: 22 May 2024). The Bar Council (2024) A lens on justice: the move to remote justice 2020–2024. Available at: https://www.barcouncil.org.uk/resource/remote-justice-report-may-2024.html (Accessed: 22 May 2024). Clark, J. (2021) Evaluation of remote hearings during the COVID-19 pandemic. Available at: https://assets.publishing.service.gov.uk/media/61b71ebd8fa8f5037b09c7b1/Evaluation_of_remote_hearings_v23.pdf (Accessed: 22 May 2024). European Commission (2013) Ethics for researchers. Available at: https://ec.europa.eu/research/participants/data/ref/fp7/89888/ethics-for-researchers_en.pdf (Accessed: 8 December 2023). Farragher, J. (2020) Government could bring back tribunal fees, reports suggest. Available at: https://www.personneltoday.com/hr/government-could-bring-back-tribunal-fees-reports-suggest/ (Accessed: 13 December 2023). Nature (2020) Henrietta Lacks: science must right a historical wrong. Available at: https://www.nature.com/articles/d41586-020-02494-z (Accessed: 14 December 2023). Otterburn, J. (2023) ‘Keynote speech to Law Works pro bono conference, 30 June 2023’, LawWorks Pro Bono Conference 2023. Available at: https://www.lawworks.org.uk/solicitors-and-volunteers/resources/lawworks-pro-bono-conference-2023 (Accessed: 14 December 2023). The Open University (no date a) Working with statistics. Available at: https://learn2.open.ac.uk/mod/oucontent/view.php?id=1890920 (Accessed: 13 December 2023). The Open University (no date b) How do I do a literature search? Available at: https://www5.open.ac.uk/library/finding-information-on-your-research-topic/how-do-i-do-a-literature-search?nocache=657b1835ad2bf (Accessed: 14 December 2023). The Open University (no date c) Introducing research in law and beyond. Available at: https://www.open.edu/openlearn/society-politics-law/introducing-research-law-and-beyond/content-section-0?active-tab=content-tab (Accessed: 14 December 2023). The Open University (no date d) Conducting an interview. Available at: https://help.open.ac.uk/postgraduate-study-skills/conducting-an-interview (Accessed: 14 December 2023). Tight, M. (2021) Documentary research in the social sciences. Available at: https://www.methodspace.com/blog/documentary-research-in-the-social-sciences (Accessed: 19 December 2023). Wellcome Trust (2014) Ensuring your research is ethical: a guide for extended project qualification students. Available at: https://wellcome.org/sites/default/files/wtp057673_0.pdf (Accessed: 8 December 2023). World Health Organisation (WHO) (no date) Ensuring ethical standards and procedures for research with human beings. Available at: https://www.who.int/activities/ensuring-ethical-standards-and-procedures-for-research-with-human-beings#:~:text=It%20is%20important%20to%20adhere,ethical%20standards%20are%20being%20upheld (Accessed: 8 December 2023). This free course was written by Liz Hardie. It was first published in 2024. Except for third party materials and otherwise stated (see terms and conditions), this content is made available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 Licence. The material acknowledged below is Proprietary and used under licence (not subject to Creative Commons Licence). Grateful acknowledgement is made to the following sources for permission to reproduce material in this free course: Images and AV Course image: lightwise / 123rf.com Figure 1: © Fizkes | Dreamstime.com Figure 2: pcess609 / iStock / Getty Images Plus Figure 3: © Godruma | Dreamstime.com Figure 4: HappySloth / Shutterstock Figure 6: Viktoria Kurpas / Shutterstock Figure 7 (top left): wavebreakmediamicro / 123rf.com Figure 7 (top right) and Figure 18: tadamichi / Shutterstock Figure 7 (bottom left) and Figure 14: vectorpouch / Shutterstock Figure 7 (bottom right) and Figure 15: ntlstudio / 123rf.com Figure 8: VideoFlow / Shutterstock Figure 9: Oleksandr Korzh / Shutterstock Figure 13: iqoncept / 123rf.com Figure 16: alinaalina1 / 123rf.com Figure 19: © Nicoelnino | Dreamstime.com Figure 20: etiamos / 123rf.com Figure 21: CrizzyStudio / Shutterstock Video 2: Dr Helen Kara, https://www.youtube.com/watch?v=1Aqt64VN4_4 Tables Table 2: Hearings conducted in 2022 and 2023 by jurisdiction and format (%): Adapted from (2024)., A lens on justice: The move to remote justice 2020 2024., ©2024 The Bar Council Text Evaluation of remote hearings during the COVID 19 pandemic: Clark, Janet. (2021) Evaluation of remote hearings during the COVID 19 pandemic., HM Courts and Tribunal service., © Crown copyright 2021., Reproduced under the terms of the Open Government Licence v3.0., ( nationalarchives.gov.uk/doc/open-governmentlicence/version/3) Every effort has been made to contact copyright owners. If any have been inadvertently overlooked, the publishers will be pleased to make the necessary arrangements at the first opportunity. Don't miss out If reading this text has inspired you to learn more, you may be interested in joining the millions of people who discover our free learning resources and qualifications by visiting The Open University – www.open.edu/openlearn/free-courses.

Discussion 2024042601