Science, Maths & Technology

Confusing Terms In Statistics

Updated Monday 10th January 2005

Kevin McConway explains why, for a statistician, reliable and significant have specific meanings

Pills Copyrighted image Icon Copyright: BBC

Most professions have a tendency to use jargon that is impenetrable to outsiders. Statisticians are no exception to this general rule. If you look in a statistics textbook, you’ll probably be unlucky enough to find words like ‘kurtosis’ and ‘heteroscedasticity’ (unless it’s an American book, when they’ll spell it ‘heteroskedasticity’).

Don’t worry, I’m not going to explain what these mean; but statisticians display another kind of jargon use that can be even more confusing. We use everyday words, but give them special meanings that differ from the meaning in everyday use. Sometimes the difference in meaning is small, sometimes it is large. A full list of such words would be rather long.

It would include: bias, block, bootstrap, censored, contrast, deviance, deviation, distribution, error, expected, hazard, improper, influence, information, jack-knife, kernel, leverage, likelihood, mode, model, moment, moral, normal, pie, regression, scree, stress, tail, variance; and many others.

Actually, this is not as confusing as you might think. With some of the words, the technical statistical meaning is so close to the everyday meaning that no important confusion is likely to arise. Others are generally used in a technical context, so that it is clear that they are not being used in their everyday sense.

However, this is not always the case. I want to describe two words, each of which has a well-understood everyday meaning, and a technical statistical meaning that is rather different. Both of these words are used in contexts where it may not be clear whether they have the everyday meaning or the technical meaning. The two words are ‘significant’ and ‘reliable’.

Significant

First, ‘significant’. Its statistical meaning is a little complicated. Suppose I’ve invented a new pill that is supposed to cure headaches. I’ve tried giving it to a few people with headaches and most of them got better, but I know that headaches often go away on their own, and I know that there are already some pretty effective headache cures around.

So I decide to do an experiment. I get a group of volunteers who all have headaches. I choose half of them at random and give them my new pill. I give the other half a standard dose of aspirin. After an hour I ask all of them whether their headache has gone away, and I record the results. I find that more of the people who took my new pill got better than did the people who took aspirin.

Does that mean my pill works? Well, it might, or it might not. Perhaps, by chance, the group to whom I gave the new pill happened to include more people whose headache would have got better anyway, whatever they had taken. But I can do a calculation that will throw light on this possibility.

I can calculate what’s known as a P value, which is a kind of probability. The smaller this P value is, the less likely it is that my results are simply due to chance. (The connection between the P value and the likelihood that my results are due to chance is a little complicated, but let’s ignore that detail.)

If my P value is small enough, I conclude that my new pill probably does work better than aspirin. In the jargon, I would say that the difference between the two groups is statistically significant.

There’s already a potential cause for confusion here, in that it’s a small P value that gives a significant result (in the statistical sense), not a big P value as you might intuitively expect. Most statistics students get confused over this at some point.

But, the important thing to remember is this: if I say a difference is statistically significant, all I mean is that ‘we can pretty well rule out the possibility that the result is due to chance alone.’

In its everyday use, ‘significant’ means ‘having a meaning or importance’. But a difference that is statistically significant might actually have very little importance in a practical sense.

Suppose I’d actually done my headache pill experiment on a huge group of people. The experiment might indicate that my new pill only cures, let’s say, one more headache in a thousand than aspirin does, but even a small difference like this may be statistically significant if the number of people in the experiment is large enough.

If my new pills cost, let’s say, a hundred times as much as aspirin, this very slightly increased performance may be of no practical significance at all, even if the difference is statistically significant. It could also happen that a result is not statistically significant but still has practical significance — we can’t rule out the possibility that the result is just due to chance, but it might indicate the need for a bigger experiment.

Significant or reliable? A cat on scales by Alasam under Creative Commons Creative commons image Icon alasm via Flickr under Creative-Commons license
Reliable? Can this cat trust the bathroom scales? [Image by alasam under CC-BY-NC-ND licence]

 

Reliable

Now let’s turn to ‘reliable’. Suppose I have a way of measuring something. Many measuring techniques won’t always give you exactly the same result if you repeat the measurement again. But, in the statistical sense, a measuring method is said to be reliable if it tends to give similar numbers when you repeat the measurement.

At home I’ve got a very accurate balance for weighing things, and I also have a set of bathroom scales that is rather old. If I weigh an object on the balance, and then weigh it again, I might not get exactly the same result, but I know that the two results will vary by only a small amount..

However, if I weigh something twice on my old bathroom scales, the results will differ more, on average. So, in the statistical sense, the balance is more reliable than the bathroom scales.

However, that doesn’t necessarily mean that the balance will give more accurate results. On both weighing devices you can set the zero; in other words you can see what reading they give when nothing is on the weighing surface, and adjust this reading so it is zero.

Suppose I forgot to do this with the balance, and it actually read 200 grams with nothing on its weighing surface. Then I weigh the same thing several times. I would still get more or less the same result every time, but the recorded weights would all be about 200g too big. Now suppose I do the same thing with the bathroom scales, but adjusting the zero properly before I began.

The results of repeated weighings will still differ more on the bathroom scales than on the balance, so the balance is still more reliable in the statistical sense. But, on average, the results from the bathroom scales might be nearer to the true weight of the object.

The clear message is that reliability, in the statistical sense, is not the only thing we should be concerned about. Validity, or lack of bias, are important too. These terms refer to how close the average result of the measuring process is to the true value of what’s being measured.

With the zero set wrongly on the balance and right on the bathroom scales, measurements made on the balance are more reliable but less valid than those on the bathroom scales. These ideas of reliability and validity are used with social and psychological measurements rather more than with physical measurements, but the basic ideas are the same.

The upshot is that a measurement which is reliable in the statistical sense might not be reliable in the everyday sense — you might not want to rely on it or trust it, if its validity is not great enough.

So, if you hear someone talking about a result being ‘significant’ or ‘reliable’, you must always try to clarify the sense in which the words are being used. Maybe they aren’t saying quite what it sounds as if they are saying!

 

For further information, take a look at our frequently asked questions which may give you the support you need.

Have a question?

Other content you may like

Take a trip to infinity Creative commons image Icon Guiseppe Savo under CC-BY-NC-ND licence under Creative-Commons license article icon

Science, Maths & Technology 

Take a trip to infinity

The concept of ‘infinity’ has enchanted and perplexed the world’s leading mathematicians for centuries. Through her outreach work in schools, Dr Katie Chicot, Staff Tutor in the OU’s Department of Mathematics and Statistics, has been introducing thousands of students to the wonders and mysteries of the infinite. Let her be your guide for a brief excursion through eternity.

Article
Unscramble the clue to The Code Copyrighted image Icon Copyright: Production team article icon

Science, Maths & Technology 

Unscramble the clue to The Code

Got your free mathematical puzzle? One of the clues to The Code Treasure Hunt can be found by unscrambling the puzzle on this page.

Article
Surveys: The Art Of The Possible Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Surveys: The Art Of The Possible

There are many factors that make up a 'good survey'. Kevin McConway looks at the background behind the workings of surveys

Article
Using vectors to model Copyrighted image Icon Copyright: Used with permission free course icon Level 2 icon

Science, Maths & Technology 

Using vectors to model

This free course, Using vectors to model, introduces the topic of vectors. The subject is developed without assuming you have come across it before, but the course assumes that you have previously had a basic grounding in algebra and trigonometry, and how to use Cartesian coordinates for specifying a point in a plane.

Free course
16 hrs
Diary of a data sleuth: Football injury time Copyrighted image Icon Copyright: Michael Flippo | Dreamstime.com article icon

Science, Maths & Technology 

Diary of a data sleuth: Football injury time

Does injury time lead to last-minute goals? This is what happened when we asked Lecturer in Telematics Tony Hirst to investigate. 

Article
Coincidence, correlation and chance Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Coincidence, correlation and chance

Helen Joyce and Andrew Stickland look at the probability of coincidence, correlation and chance in our everyday lives

Article
Surfaces Copyrighted image Icon Copyright: Used with permission free course icon Level 3 icon

Science, Maths & Technology 

Surfaces

Surfaces are a special class of topological spaces that crop up in many places in the world of mathematics. In this free course, you will learn to classify surfaces and will be introduced to such concepts as homeomorphism, orientability, the Euler characteristic and the classification theorem of compact surfaces.

Free course
20 hrs
Symmetry Copyrighted image Icon Copyright: Used with permission free course icon Level 2 icon

Science, Maths & Technology 

Symmetry

We all encounter symmetry in our everyday lives, in both natural and man-made structures. The mathematical concepts surrounding symmetry can be a bit more difficult to grasp. This free course, Symmetry, explains such concepts as direct and indirect symmetries, Cayley tables and groups through exercises, audio and video.

Free course
20 hrs
Modelling with first order differential equations Copyrighted image Icon Copyright: Used with permission free course icon Level 2 icon

Science, Maths & Technology 

Modelling with first order differential equations

This free course, Modelling with first order differential equations, lays the foundation of Newtonian mechanics and in particular the procedure for solving dynamics problems. The prerequisite skills needed for this free course are the ability to solve first and second-order differential equations, a knowledge of vectors, and an understanding of the concept of a force

Free course
16 hrs