Skip to content
Science, Maths & Technology

Confusing Terms In Statistics

Updated Monday 10th January 2005

Kevin McConway explains why, for a statistician, reliable and significant have specific meanings

Pills Copyrighted image Icon Copyright: BBC

Most professions have a tendency to use jargon that is impenetrable to outsiders. Statisticians are no exception to this general rule. If you look in a statistics textbook, you’ll probably be unlucky enough to find words like ‘kurtosis’ and ‘heteroscedasticity’ (unless it’s an American book, when they’ll spell it ‘heteroskedasticity’).

Don’t worry, I’m not going to explain what these mean; but statisticians display another kind of jargon use that can be even more confusing. We use everyday words, but give them special meanings that differ from the meaning in everyday use. Sometimes the difference in meaning is small, sometimes it is large. A full list of such words would be rather long.

It would include: bias, block, bootstrap, censored, contrast, deviance, deviation, distribution, error, expected, hazard, improper, influence, information, jack-knife, kernel, leverage, likelihood, mode, model, moment, moral, normal, pie, regression, scree, stress, tail, variance; and many others.

Actually, this is not as confusing as you might think. With some of the words, the technical statistical meaning is so close to the everyday meaning that no important confusion is likely to arise. Others are generally used in a technical context, so that it is clear that they are not being used in their everyday sense.

However, this is not always the case. I want to describe two words, each of which has a well-understood everyday meaning, and a technical statistical meaning that is rather different. Both of these words are used in contexts where it may not be clear whether they have the everyday meaning or the technical meaning. The two words are ‘significant’ and ‘reliable’.

Significant

First, ‘significant’. Its statistical meaning is a little complicated. Suppose I’ve invented a new pill that is supposed to cure headaches. I’ve tried giving it to a few people with headaches and most of them got better, but I know that headaches often go away on their own, and I know that there are already some pretty effective headache cures around.

So I decide to do an experiment. I get a group of volunteers who all have headaches. I choose half of them at random and give them my new pill. I give the other half a standard dose of aspirin. After an hour I ask all of them whether their headache has gone away, and I record the results. I find that more of the people who took my new pill got better than did the people who took aspirin.

Does that mean my pill works? Well, it might, or it might not. Perhaps, by chance, the group to whom I gave the new pill happened to include more people whose headache would have got better anyway, whatever they had taken. But I can do a calculation that will throw light on this possibility.

I can calculate what’s known as a P value, which is a kind of probability. The smaller this P value is, the less likely it is that my results are simply due to chance. (The connection between the P value and the likelihood that my results are due to chance is a little complicated, but let’s ignore that detail.)

If my P value is small enough, I conclude that my new pill probably does work better than aspirin. In the jargon, I would say that the difference between the two groups is statistically significant.

There’s already a potential cause for confusion here, in that it’s a small P value that gives a significant result (in the statistical sense), not a big P value as you might intuitively expect. Most statistics students get confused over this at some point.

But, the important thing to remember is this: if I say a difference is statistically significant, all I mean is that ‘we can pretty well rule out the possibility that the result is due to chance alone.’

In its everyday use, ‘significant’ means ‘having a meaning or importance’. But a difference that is statistically significant might actually have very little importance in a practical sense.

Suppose I’d actually done my headache pill experiment on a huge group of people. The experiment might indicate that my new pill only cures, let’s say, one more headache in a thousand than aspirin does, but even a small difference like this may be statistically significant if the number of people in the experiment is large enough.

If my new pills cost, let’s say, a hundred times as much as aspirin, this very slightly increased performance may be of no practical significance at all, even if the difference is statistically significant. It could also happen that a result is not statistically significant but still has practical significance — we can’t rule out the possibility that the result is just due to chance, but it might indicate the need for a bigger experiment.

Significant or reliable? A cat on scales by Alasam under Creative Commons Creative commons image Icon alasm via Flickr under Creative-Commons license
Reliable? Can this cat trust the bathroom scales? [Image by alasam under CC-BY-NC-ND licence]

 

Reliable

Now let’s turn to ‘reliable’. Suppose I have a way of measuring something. Many measuring techniques won’t always give you exactly the same result if you repeat the measurement again. But, in the statistical sense, a measuring method is said to be reliable if it tends to give similar numbers when you repeat the measurement.

At home I’ve got a very accurate balance for weighing things, and I also have a set of bathroom scales that is rather old. If I weigh an object on the balance, and then weigh it again, I might not get exactly the same result, but I know that the two results will vary by only a small amount..

However, if I weigh something twice on my old bathroom scales, the results will differ more, on average. So, in the statistical sense, the balance is more reliable than the bathroom scales.

However, that doesn’t necessarily mean that the balance will give more accurate results. On both weighing devices you can set the zero; in other words you can see what reading they give when nothing is on the weighing surface, and adjust this reading so it is zero.

Suppose I forgot to do this with the balance, and it actually read 200 grams with nothing on its weighing surface. Then I weigh the same thing several times. I would still get more or less the same result every time, but the recorded weights would all be about 200g too big. Now suppose I do the same thing with the bathroom scales, but adjusting the zero properly before I began.

The results of repeated weighings will still differ more on the bathroom scales than on the balance, so the balance is still more reliable in the statistical sense. But, on average, the results from the bathroom scales might be nearer to the true weight of the object.

The clear message is that reliability, in the statistical sense, is not the only thing we should be concerned about. Validity, or lack of bias, are important too. These terms refer to how close the average result of the measuring process is to the true value of what’s being measured.

With the zero set wrongly on the balance and right on the bathroom scales, measurements made on the balance are more reliable but less valid than those on the bathroom scales. These ideas of reliability and validity are used with social and psychological measurements rather more than with physical measurements, but the basic ideas are the same.

The upshot is that a measurement which is reliable in the statistical sense might not be reliable in the everyday sense — you might not want to rely on it or trust it, if its validity is not great enough.

So, if you hear someone talking about a result being ‘significant’ or ‘reliable’, you must always try to clarify the sense in which the words are being used. Maybe they aren’t saying quite what it sounds as if they are saying!

 

For further information, take a look at our frequently asked questions which may give you the support you need.

Have a question?

Other content you may like

Die Hard Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Die Hard

Robert Llewellyn and Dr Jonathan Hare take on Hollywood Science, testing the science that filmgoers take for granted. Here they look at the famous bungee jump Bruce Willis makes in the film Die Hard

Article
Diary of a data sleuth: When the data you don't collect affects the data you do Creative commons image Icon eviloars under CC-BY-NC licence under Creative-Commons license article icon

Science, Maths & Technology 

Diary of a data sleuth: When the data you don't collect affects the data you do

Is there anything sinister in the statistics which appears to show left-handed people die before their time?

Article
Statistics: What is it or what are they? Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Statistics: What is it or what are they?

How can we hope to understand statistics if we can't even be sure if they're singular or plural? Kevin McConway has an answer

Article
Modelling displacements and velocities Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Modelling displacements and velocities

In this free course, Modelling displacements and velocities, you will see first how to convert vectors from geometric form, in terms of a magnitude and direction, to component form, and then how conversion in the opposite sense is accomplished. The ability to convert between these different forms of a vector is useful in certain problems involving displacement and velocity, as shown in Section 2, in which you will also work with bearings.

Free course
3 hrs
Language, notation and formulas Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Language, notation and formulas

Communication is as vital in mathematics as in any language. This free course, Language, notation and formulas, will help you to express yourself clearly when writing and speaking about mathematics. You will also learn how to answer questions in the manner that is expected by the examiner.

Free course
5 hrs
The Code: Grandmaster of Mosaics activity icon

Science, Maths & Technology 

The Code: Grandmaster of Mosaics

Deep in the dusty pages of an ancient tome is a legend centuries old that only you can overcome. Do you have the ability to identify symmetries and unlock the mosaic in this clever game?

Activity
Does maths offer a better model for multi-candidate elections? Creative commons image Icon Peter Miller under Creative Commons BY-NC-ND 4.0 license article icon

Science, Maths & Technology 

Does maths offer a better model for multi-candidate elections?

First past the post - the tradition in UK general elections - isn't a great way to deliver the most agreeable result. Cristoph Borgers suggests we make it less of a horse race.

Article
Diagrams, charts and graphs Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Diagrams, charts and graphs

Diagrams, charts and graphs are used by all sorts of people to express information in a visual way, whether it's in a report by a colleague or a plan from your interior designer. This free course will teach you how to interpret these tools and how to use them yourself to convey information more effectively.

Free course
5 hrs

Science, Maths & Technology 

Working with diagrams

Working with diagrams is essential for students of science, technology, engineering and mathematics. This free course is packed with practical activities and tips which make learning from and with diagrams more enjoyable and rewarding. One part of this course deals with the reading of diagrams and the other part with the drawing of diagrams.

Free course
8 hrs