Science, Maths & Technology

Confusing Terms In Statistics

Updated Monday 10th January 2005

Kevin McConway explains why, for a statistician, reliable and significant have specific meanings

Pills Copyrighted image Icon Copyright: BBC

Most professions have a tendency to use jargon that is impenetrable to outsiders. Statisticians are no exception to this general rule. If you look in a statistics textbook, you’ll probably be unlucky enough to find words like ‘kurtosis’ and ‘heteroscedasticity’ (unless it’s an American book, when they’ll spell it ‘heteroskedasticity’).

Don’t worry, I’m not going to explain what these mean; but statisticians display another kind of jargon use that can be even more confusing. We use everyday words, but give them special meanings that differ from the meaning in everyday use. Sometimes the difference in meaning is small, sometimes it is large. A full list of such words would be rather long.

It would include: bias, block, bootstrap, censored, contrast, deviance, deviation, distribution, error, expected, hazard, improper, influence, information, jack-knife, kernel, leverage, likelihood, mode, model, moment, moral, normal, pie, regression, scree, stress, tail, variance; and many others.

Actually, this is not as confusing as you might think. With some of the words, the technical statistical meaning is so close to the everyday meaning that no important confusion is likely to arise. Others are generally used in a technical context, so that it is clear that they are not being used in their everyday sense.

However, this is not always the case. I want to describe two words, each of which has a well-understood everyday meaning, and a technical statistical meaning that is rather different. Both of these words are used in contexts where it may not be clear whether they have the everyday meaning or the technical meaning. The two words are ‘significant’ and ‘reliable’.

Significant

First, ‘significant’. Its statistical meaning is a little complicated. Suppose I’ve invented a new pill that is supposed to cure headaches. I’ve tried giving it to a few people with headaches and most of them got better, but I know that headaches often go away on their own, and I know that there are already some pretty effective headache cures around.

So I decide to do an experiment. I get a group of volunteers who all have headaches. I choose half of them at random and give them my new pill. I give the other half a standard dose of aspirin. After an hour I ask all of them whether their headache has gone away, and I record the results. I find that more of the people who took my new pill got better than did the people who took aspirin.

Does that mean my pill works? Well, it might, or it might not. Perhaps, by chance, the group to whom I gave the new pill happened to include more people whose headache would have got better anyway, whatever they had taken. But I can do a calculation that will throw light on this possibility.

I can calculate what’s known as a P value, which is a kind of probability. The smaller this P value is, the less likely it is that my results are simply due to chance. (The connection between the P value and the likelihood that my results are due to chance is a little complicated, but let’s ignore that detail.)

If my P value is small enough, I conclude that my new pill probably does work better than aspirin. In the jargon, I would say that the difference between the two groups is statistically significant.

There’s already a potential cause for confusion here, in that it’s a small P value that gives a significant result (in the statistical sense), not a big P value as you might intuitively expect. Most statistics students get confused over this at some point.

But, the important thing to remember is this: if I say a difference is statistically significant, all I mean is that ‘we can pretty well rule out the possibility that the result is due to chance alone.’

In its everyday use, ‘significant’ means ‘having a meaning or importance’. But a difference that is statistically significant might actually have very little importance in a practical sense.

Suppose I’d actually done my headache pill experiment on a huge group of people. The experiment might indicate that my new pill only cures, let’s say, one more headache in a thousand than aspirin does, but even a small difference like this may be statistically significant if the number of people in the experiment is large enough.

If my new pills cost, let’s say, a hundred times as much as aspirin, this very slightly increased performance may be of no practical significance at all, even if the difference is statistically significant. It could also happen that a result is not statistically significant but still has practical significance — we can’t rule out the possibility that the result is just due to chance, but it might indicate the need for a bigger experiment.

Significant or reliable? A cat on scales by Alasam under Creative Commons Creative commons image Icon alasm via Flickr under Creative-Commons license
Reliable? Can this cat trust the bathroom scales? [Image by alasam under CC-BY-NC-ND licence]

 

Reliable

Now let’s turn to ‘reliable’. Suppose I have a way of measuring something. Many measuring techniques won’t always give you exactly the same result if you repeat the measurement again. But, in the statistical sense, a measuring method is said to be reliable if it tends to give similar numbers when you repeat the measurement.

At home I’ve got a very accurate balance for weighing things, and I also have a set of bathroom scales that is rather old. If I weigh an object on the balance, and then weigh it again, I might not get exactly the same result, but I know that the two results will vary by only a small amount..

However, if I weigh something twice on my old bathroom scales, the results will differ more, on average. So, in the statistical sense, the balance is more reliable than the bathroom scales.

However, that doesn’t necessarily mean that the balance will give more accurate results. On both weighing devices you can set the zero; in other words you can see what reading they give when nothing is on the weighing surface, and adjust this reading so it is zero.

Suppose I forgot to do this with the balance, and it actually read 200 grams with nothing on its weighing surface. Then I weigh the same thing several times. I would still get more or less the same result every time, but the recorded weights would all be about 200g too big. Now suppose I do the same thing with the bathroom scales, but adjusting the zero properly before I began.

The results of repeated weighings will still differ more on the bathroom scales than on the balance, so the balance is still more reliable in the statistical sense. But, on average, the results from the bathroom scales might be nearer to the true weight of the object.

The clear message is that reliability, in the statistical sense, is not the only thing we should be concerned about. Validity, or lack of bias, are important too. These terms refer to how close the average result of the measuring process is to the true value of what’s being measured.

With the zero set wrongly on the balance and right on the bathroom scales, measurements made on the balance are more reliable but less valid than those on the bathroom scales. These ideas of reliability and validity are used with social and psychological measurements rather more than with physical measurements, but the basic ideas are the same.

The upshot is that a measurement which is reliable in the statistical sense might not be reliable in the everyday sense — you might not want to rely on it or trust it, if its validity is not great enough.

So, if you hear someone talking about a result being ‘significant’ or ‘reliable’, you must always try to clarify the sense in which the words are being used. Maybe they aren’t saying quite what it sounds as if they are saying!

 

For further information, take a look at our frequently asked questions which may give you the support you need.

Have a question?

Other content you may like

John Napier Copyrighted image Icon Copyright: Used with permission free course icon Level 2 icon

Science, Maths & Technology 

John Napier

This free course looks at Scotsman John Napier, best known to for his treatise on Protestant religion. However, it was his interest in a completely different subject that radically altered the course of mathematics. After 40 years of dabbling in maths, he revealed his table of logarithms in the early 17th century.

Free course
3 hrs
Beating the odds: The maths of a World Cup 2014 win Copyrighted image Icon Copyright: Michael Flippo | Dreamstime.com article icon

Science, Maths & Technology 

Beating the odds: The maths of a World Cup 2014 win

Dr Katie Chicot provides us with a statistical insight into this year’s World Cup event.

Article
Using a scientific calculator Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Using a scientific calculator

Do you have a Casio fx-83 ES scientific calculator (or a compatible model) and want to learn how to use it? This free course, Using a scientific calculator, will help you to understand how to use the different facilities and functions and discover what a powerful tool this calculator can be!

Free course
10 hrs
Geometry Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Geometry

Geometry is concerned with the various aspects of size, shape and space. In this free course you will explore the concepts of angles, shapes, symmetry, area and volume through interactive activities.

Free course
5 hrs
Emmy Noether: Bucking the historical trends Creative commons image Icon Mathematical Association of America under CC BY-ND 2.0 licence under Creative-Commons license article icon

Science, Maths & Technology 

Emmy Noether: Bucking the historical trends

Mathematics isn't solely the pursuit of men. We should celebrate great thinkers like Emmy Noether and the superb work of women in mathematics today

Article
Number systems Copyrighted image Icon Copyright: Used with permission free course icon Level 2 icon

Science, Maths & Technology 

Number systems

Number systems and the rules for combining numbers can be daunting. This free course, Number systems, will help you to understand the detail of rational and real numbers, complex numbers and integers. You will also be introduced to modular arithmetic and the concept of a relation between elements of a set.

Free course
20 hrs
How can statistics bring dead languages back to life? Copyright free image Icon Copyright free: Gellinger audio icon

Science, Maths & Technology 

How can statistics bring dead languages back to life?

Languages long unspoken are being brought back to life through statistical models.

Audio
5 mins
Numbers: An introduction to subtraction Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Numbers: An introduction to subtraction

Do you want to improve your ability to subtract one number from another, especially if decimals are involved, without having to rely on a calculator? Numbers: An introduction to subtraction, is a free course that will help you get to grips with subtraction and give you some practice in doing it.

Free course
1 hr
Take a trip to infinity Creative commons image Icon Guiseppe Savo under CC-BY-NC-ND licence under Creative-Commons license article icon

Science, Maths & Technology 

Take a trip to infinity

The concept of ‘infinity’ has enchanted and perplexed the world’s leading mathematicians for centuries. Through her outreach work in schools, Dr Katie Chicot, Staff Tutor in the OU’s Department of Mathematics and Statistics, has been introducing thousands of students to the wonders and mysteries of the infinite. Let her be your guide for a brief excursion through eternity.

Article