You can find lots of excellent information via the Internet. But there are potential pitfalls. Anyone can post anything at all on the Web – there is no review by experts, no editorial control. Sites may appear today and disappear tomorrow. Information can be inaccurate, misleading and, in some cases, dangerous. At the same time, clever presentation and innovative web design can make sites extremely seductive. This is particularly important when you are looking for information about health, illnesses and treatments. There are at least 100,000 health information web sites and it has been estimated that 98 million adults worldwide have used the Internet to access health information.
How can you be sure which information to trust? There are some simple steps you can take to find the best possible information available.
Finding the best information
One approach is to use Internet gateways to search for relevant sites. Search engines like Google and Alta Vista generally work by using robots (sometimes called spiders) to trawl the Web and find sites. This means that the sites they find are of variable quality. The sites you find using an Internet Gateway have been selected by professional librarians or information specialists, using quality criteria.
In health there are a number of specialist gateways which recommend good quality sites and information sources.
The US based Healthfinder site is probably the most comprehensive gateway site available at present
NHS Direct Online is a sister service to the NHS Direct telephone service, aims to become the UK equivalent of the Healthfinder site. It is an excellent starting point for UK health information. It includes an on-line encyclopaedia of conditions and treatments, links to thousands of high quality information leaflets and helpsheets, self-help groups and specialist organizations.
Organised Medical Networked Information (OMNI) is a specialist UK based gateway to resources in medicine, health and health services. It aims to provide comprehensive coverage of the UK resources in this area and access to the best resources worldwide.
Patient UK is a directory of UK health, disease and related websites. It is edited by two GPs.
MedlinePlus is a gateway produced in the United States by the National Library of Medicine and aimed at the general public.
If you are going to search for information in more than one place you need to be organised and systematic. There are three steps you need to follow:
- Search & Record
- Review & Evaluate
Plan Be clear about what you want to know.
What is your question?
What are the keywords you are going to use to carry out the search?
Are there synonyms or equivalent words you might need to try?
Where are you going to search? (Which search engines or gateways are you going to use?)
Search & Record Carry out your search systematically.
Keep a record of each source you search, the key words you use and what you find.
If you don’t find as much information as you need, try broadening the key words you are using.
If you find too much, try narrowing the key words.
Review & Evaluate When you think you have found enough information to answer your question, stop.
Review what you have found and filter out anything which doesn’t address your question.
Then evaluate what you have left, using the PROMPT approach.
Judging the quality of information – the PROMPT approach
PROMPT is an easy way of remembering six dimensions of information quality. It offers you a simple framework for assessing all kinds of information – websites, articles, books or leaflets. We hope you will find it useful.
What does PROMPT stand for?
The way in which information is presented has a profound effect on the way we receive and perceive it. There are many aspects of poor presentation – any of which can create a barrier between the message and the audience.
Look at this website – it is Jim Jacobson’s home page. Can you identify some of the aspects of poor presentation it demonstrates? You might have thought about colours, fonts, white on black, poor use of graphics, irritating animation, annoying pop-up boxes, music – you name it!
The same principles of course apply to printed information. Just to remind you of the importance of language in communication, look at the website of the Plain English Campaign and their ‘Golden Bull award’ winners.
Does presentation matter? We think that presentation can have serious consequences. We might ignore and therefore miss out on information which has excellent content, but poor presentation. We might misinterpret information because of poor presentation. Someone might not be able to follow fire or safety instructions, for instance. If poorly presented information fails to get its message across it can waste the time of the recipients and the money spent on producing it.
Assessing presentation – what to look for:
- Writing style
This is an important aspect of information quality but it is not a property of the information you are looking at, but rather its relationship to the question you are trying to answer. There are a number of ways in which it may or may not be relevant to your needs. For example:
- level (it may be too detailed/specialised or too general/simple for your needs – e.g articles in medical books and journals tend to focus on unusual or extreme cases of an illness which won't be relevant to the majority of sufferers and could cause unnecessary alarm)
- differences between people (e.g. physiological differences between children and adults or ethnic/cultural differences in disease incidence) means that one piece of information may not ‘fit all’ and could be dangerously misleading
- emphasis (it may not contain the kind of information you are seeking – this is often a question of emphasis, which may not be identifiable from the abstract)
Consider this scenario
Your twelve year old daughter is doing a school project on ‘the Brain and how it works ‘. As usual she has left herself very little time to do the research and, as you are visiting the Library for another purpose, you offer to see if they have any books or other resources which might be useful. You find the following three sources of information:
International Review of Neurobiology: Vol 45 Brain plasticity and epilepsy / edited by Jerome Engel 2001
The brain game - from the National Endowment for Science, Technology and the Arts (NESTA)
Which of these do you think are likely to be relevant for the s
Does relevance matter? Health information which is irrelevant in some of the ways listed above could at best waste your time and at worst be misleading and cause unnecessary anxiety or confusion.
Assessing relevance – what to look for
- Be clear about what you are looking for from the start – this will help you to be ruthless in discarding things which don’t fit the bill.
- Try and avoid having to read things in full – look at the abstract or summary, if there is one, for quick indications of ‘what it is about’.
In an ideal world, ‘objective’ or ‘balanced’ information would present all the evidence, all the arguments and leave you to weigh this up and draw conclusions. In the real world, this is probably unachievable – everything has some kind of position of interest.
This puts the onus on you to be aware of these positions and take account of them when you interpret the information.
In some cases, authors may be explicitly expressing a particular viewpoint – this is perfectly valid as long as they are explicit about the perspective they represent. Hidden bias, whether or not it is deliberate, can be misleading.
Consider this scenario:
A friend of yours is a heavy smoker and you have been trying to persuade her to give up. You read in the newspaper that the NHS are going to make Nicotine Replacement Therapy (NRT) available on prescription and you suggest this to your friend. She says that she would like some evidence that it really works. You offer to collect some information for her. This is what you find:
You have probably identified some vested interests. Here is a summary of some different kinds of vested interest:
Financial vested interests – protecting or selling a product or service
Media manipulation – a ‘good story’ (selling papers or pushing up ratings)
Political propaganda – influencing public opinion
Government ‘propaganda’ – influencing public behaviour
Academic self-interest - need to publish for career or financial rewards.
Assessing objectivity – what to look for:
- Perspectives – do the authors state clearly the viewpoint they are taking?
- Opinions: academic articles will often present unsubstantiated theories for debate. Look out for opinion presented as if it were fact.
- Language: can be a useful danger sign. Look out for language which is either emotionally charged or vague.
- Sponsorship: academic research may be sponsored by industry (e.g. pharmaceutical companies). This does not necessarily make the research less objective but it may make its interpretation selective. Make sure that all potential vested interests are clearly identified.
Unless you have specialist knowledge of the subject, you are unlikely to be able to assess the reliability or accuracy of a piece of information. The methods used to produce the information are, however, one important indicator. Let’s look at scientific research reports as an example of the kinds of information where method is a key indicator of reliability.
Scientific research reports Whether you are reading an academic paper or a newspaper report, you can ask some common sense questions about the way research has been carried out.
NEW WONDER CURE FOR HANGOVERS
A scientist in Salisbury claims to have found a natural cure for hangovers using a mixture of vinegar, raw eggs, sugar and soya beans. He has carried out extensive research over a number of years with students at the local agricultural college. Four out of five students were free of symptoms within 30 minutes, he says…
Are you convinced by this? If not, what questions would you like to ask about the way the research was conducted?
You might have wanted to know…
Question Research terminology
How many students was it tested on? Sample size
How were they selected? (e.g. How drunk did they have to be?) Population definition
Did the research compare their recovery with similar students who didn't take the cure? Control group
How did he define and measure recovery? Outcome measures
So you don't have to be an expert in research methods to ask a few basic but very important questions.
Assessing method – what to look forIf some kind of research or data collection is involved:
- Is it clear how the research was carried out?
- Was the sample big enough?
- Was it the right kind of sample?
- Was there a control group?
- Were the right outcome measures used?
- For surveys - were the questions robust enough?
- Was there any potential hidden bias?
- The provenance of a piece of information (i.e. who produced it? Where did it come from?) may provide another useful clue to its reliability.
- It represents the ‘credentials’ of a piece of information that supports its status and perceived value.
- It is therefore very important to be able to identify the author, sponsoring body or source of your information.
- But there is also an argument that this is unfair and that information should be judged on its own merit, rather than on the basis of who produced it and where it was published.
Authors If you know who the author is you can:
- find out whether they are an acknowledged expert in the subject area
- find out what other papers or books they have published perhaps trace unpublished material like their PhD thesis
- find out if they are known to have a particular perspective on the topic and whether their views are controversial perhaps contact them in person.
N.B. There are dangers here – a well known ‘expert’ may be the first listed author on an article which is based on the work of one of his/her research students.
Organisations If you know who the sponsoring organisation is you can check out:
- Is it a commercial company?
- A voluntary organisation? (Note that it could still be sponsored by another organisation with its own perspective)
- A statutory body?
- A professional body or trade union?
- A research organisation?
- Is it well-established or ephemeral/short-lived?
- Can you identify the people involved in the organisation?
- Could you contact them if you wanted to verify the information?
- Do the people have ‘other interests’ or links which might have a bearing on the way you regard the information?
Publications It can be useful to know how something is published.
- Any individual can publish anything on the World Wide Web, or post to a discussion list – this has to be judged on its own merit and with reference to the author’s credentials.
- Journals and newspapers, whether in print or electronic, will in almost all cases have an editor and/or an editorial board that decides editorial policy which influences what will be published.
- Most academic print journals are peer reviewed – this means that articles submitted will be evaluated by at least two experts in the field before being recommended for publication.
- Some electronic journals do not have a peer review process.
So while provenance in itself doesn’t prove the value of a piece of information it can hold a lot of important clues.
Assessing provenance – what to look for:
- Details of authors – who they work for, what else have they published? what is their status?
- Organisations involved – their work, their interests, their connections, who funds them?
- Publications – who is the editor? who is on the editorial board? What is their editorial policy? Is it peer reviewed?
The date when information was produced or published can be an important aspect of quality. This is not quite as simple as saying that 'Good information has to be up to date’.
Think about a newspaper article, published in 1999, about the recently discovered link between BSE and CJD. Would you consider this article to be out of date?
Well – it is and it isn’t. In other words – one person’s out of date newspaper cutting is another person’s historical document. Some documents (the Domesday book, for example) are indeed ‘timeless’ in that they will always be regarded as useful and relevant. In other cases it will be very important for information to be up to date.
Finding the relevant date may not be straightforward. In some cases it is difficult to ascertain the true ‘age’ of the information:
- There may be no publication date.
- There may be no indication of when the information was last updated.
- The date of publication may not reflect the date the information was produced e.g.it can take many years for research results to be written up and published.
- Information may not be very old, but may have been superceded (e.g. statistical series or regular reports). This means you have to be familiar with these sources and how they are produced.
- Information may appear to be ‘old’ but is in fact the most recent information on the topic and, as such, of value.
Summary – what to look for
- Is it clear when the information was produced?
- Does the date of the information meet your requirements?
- Is it obsolete? (Has it been superseded?)
- Or is it the most up to date information available on the topic?
PROMPT is a simple checklist to help you to think about information quality. We are not suggesting that you carry out a detailed evaluation of every piece of information you come across. But if you familiarise yourself with the PROMPT six dimensions, you will find that you can scan things very quickly and identify their strengths and weaknesses. It is about developing a critical approach and that just takes a bit of practice.
Some useful websites for looking at information quality
The Centre for Health Information Quality (CHIQ) was set up by the Department of Health in 1997 to‘act as a source of expertise and knowledge for the NHS and patient representative groups on all aspects of patient information in the aim of improving the NHS’s capability, competence and capacity to provide good evidence-based patient information’.
DISCERN is a specialist tool that focuses on information to help support treatment choices. It is unique in that it has been developed and validated in a rigorous way. It is used to appraise information for NHS Direct and NHS Direct Online and by many other information providers.
QUICK ideally the skills of critical evaluation of information should be taught and nurtured from an early age. The Quick Guide to checking information quality has been developed specifically to help young people to critically evaluate written information.
The material in this ‘Getting the best information…’ website has been adapted from extracts from the former Open University course - Making sense of information in the connected age (U120) and the current course Knowledge, information and care (K223)
If you would like to study more about finding, evaluating and using information, you might like to consider studying one of these in your own time. Take a look at these courses on the study page.