Science, Maths & Technology

Statistically significant: The US Supreme Court takes a view

Updated Tuesday 12th April 2011

A lawsuit over a cold remedy has forced the US Supreme Court into deciding what might be statistically significant.

The rulings of the US Supreme Court aren't the place most people would start looking for a discussion of what the phrase "statistically significant" might mean. But in March 2011, America's highest court of justice made a decision that involved exactly that.

Contemplation of Justice: A figure outside the US Supreme Court Creative commons image Icon Clearly Ambiguous under CC-BY licence under Creative-Commons license
Contemplation of Justice: A figure outside the US Supreme Court building

The case involved a case originally brought by investors in a company called Matrixx Initiatives Inc., who manufacture a range of over-the-counter medicaments including one called Zicam Cold Remedy.

The investors alleged that Matrixx had failed to disclose material information to them, when it had received reports that people who had taken Zicam Cold Remedy had suffered a loss of sense of smell. When these reports did become public, the share price fell, thus (allegedly) losing the investors money.

People do lose their sense of smell sometimes, for reasons that have nothing to do with a drug. Matrixx argued that the number of cases of loss of sense of smell in people who had taken Zicam was not significantly higher, in the statistical sense, than the number one would expect to see if the drug had no effect on sense of smell.

For that and other reasons, Matrixx argued that they had no obligation to disclose the reports to their investors. The case depended to a major extent, therefore, on the idea of statistical significance testing.

What's all that about? I've given a brief explanation in another OpenLearn article, Confusing terms in statistics, so I won't repeat all that, but briefly the point of doing a significance test is to see whether it is plausible that some feature in data can be explained by chance alone.

In the Matrixx case, perhaps they did get more cases of loss of sense of smell than one might have expected if the remedy had no effect on sense of smell, but maybe this could have been due entirely to chance – people who were going to lose their sense of smell anyway might just have happened to take the cold remedy.

If, after performing a statistical significance test, the conclusion was that the data are consistent with this "it's all just chance" explanation, one would report that the results were "not statistically significant".

Matrixx did claim this, but went further and argued that, because they were not statistically significant, they did not have to be disclosed.

An interesting feature of the Matrixx case is that arguments to the court were not submitted only by lawyers, investment experts, and so on, but also by Deirdre McCloskey and Steve Ziliak. They are economists and statisticians who are well known for publishing a book called The Cult of Statistical Significance, which, as you might guess from the title, is not complimentary about the way that significance testing has been used.

McCloskey and Ziliak submitted an amicus brief, or more precisely an amicus curiae brief (from Latin words meaning "friend of the court"). This is a submission to the court from a person or organisation that is not one of the parties involved in the case, but which is intended in some way to throw light on the case.

They did not argue wholesale against the use of the ideas of statistical significance. Instead, they argued as follows:

If a feature in data is not statistically significant, it's reasonable to take that to mean that the feature might plausibly be due solely to the workings of chance.

But just because it might be due to chance, that doesn't mean it is due to chance. That's only one possible explanation.

There will be others, and thus a decision on whether some piece of information is relevant can't be based on statistical significance alone. The court agreed with this line in their judgment. They said quite clearly that "Matrixx's premise that statistical significance is the only reliable indication of causation is flawed."

Most, if not all, statisticians would agree with them. Statistics students often get annoyed with what seems like excessive pedantry from their teachers. The statisticians will insist that one does not accept the hypothesis that some data feature is due to chance alone.

One merely fails to reject such a hypothesis, if the results are not statistically significant.

The idea here is that "accept" sounds too much as if one is saying, "Yes, we know it's just chance". "Fail to reject" allows the possibility that it might just be chance, but allows other possibilities too.

There's reason in our pedantry. To quote another cliché, absence of proof isn't the same as proof of absence, and a result that's not statistically significant is absence of proof that something is happening that goes beyond mere chance.

A couple of aspects of this discussion that might interest you, depending on your background:

First, what is really going on with statistical experts like Ziliak and McCloskey rubbishing significance tests? Surely they are a key and fundamental aspect of the way statistics is done?

Well, not everyone would be quite as critical as Ziliak and McCloskey, but actually, many statisticians would be very critical of the unthinking and routine use of significance testing that goes on in many areas of enquiry. (If you understand a little of the technical ideas behind significance testing, you might enjoy reading another criticism from a slightly different angle, Mindless Statistics, by Gerd Gigerenzer, the well-known author on risk. [The article opens as a pdf.])

Second, some people find it surprising at first sight that the law should be concerned with ideas and methods of statistics. But if you think about it, it's not really surprising at all.

Both statistics and the law involve dealing with uncertainty – admittedly often with different goals in sight, but uncertainty is there as an inescapable aspect in both. Gary Slapper, Professor of Law at the Open University, has written about statistics and the law

Statisticians and lawyers would not always agree on what should be done about the uncertainty, but perhaps that's inevitable – the law has to come to a clear decision on every case, however much uncertainty there is, whereas we statisticians can sometimes have the luxury of saying that there's simply not enough evidence to decide.

Further reading

Discover how to study statistics or law with The Open University.

If you're interested in the legal details, the briefs submitted to the Supreme Court in the Matrixx case (including the McCloskey and Ziliak brief) are online, as are a pdf transcript of the oral argument, and the  "opinion" (i.e. judgment), again as a pdf.


For further information, take a look at our frequently asked questions which may give you the support you need.

Have a question?

Other content you may like

Sacred and Secular: Section 2 - Finding legal judgments and legal commentary Creative commons image Icon Gordon Giles under CC-BY-ND-2.5 licence under Creative-Commons license article icon

Society, Politics & Law 

Sacred and Secular: Section 2 - Finding legal judgments and legal commentary

Judgments are given by judges sitting in different types of courts and these courts exists in different types of legal system. It is useful to understand this context in order to find and analyse judgments. This section explores those differences before moving on to identify where the judgments can be found.  

Naked before the court?: The reality of human rights Creative commons image Icon mrrobertwade under CC-BY-NC-SA licence under Creative-Commons license audio icon

Society, Politics & Law 

Naked before the court?: The reality of human rights

Appearing naked before a judge, and voting while in prison: How far should respect for human rights go?

15 mins
Ancient Mathematics Copyrighted image Icon Copyright: Production team article icon

Science, Maths & Technology 

Ancient Mathematics

When did mathematics begin? A natural question to ask, but unfortunately a very difficult one to answer, explains June Barrow-Green

Take a trip to infinity Creative commons image Icon Guiseppe Savo under CC-BY-NC-ND licence under Creative-Commons license article icon

Science, Maths & Technology 

Take a trip to infinity

The concept of ‘infinity’ has enchanted and perplexed the world’s leading mathematicians for centuries. Through her outreach work in schools, Dr Katie Chicot, Staff Tutor in the OU’s Department of Mathematics and Statistics, has been introducing thousands of students to the wonders and mysteries of the infinite. Let her be your guide for a brief excursion through eternity.

Interpreting data: Boxplots and tables Copyrighted image Icon Copyright: Used with permission free course icon Level 2 icon

Science, Maths & Technology 

Interpreting data: Boxplots and tables

This free course, Interpreting data: Boxplots and tables, is concerned with two main topics. In Section 1, you will learn about another kind of graphical display, the boxplot. A boxplot is a fairly simple graphic, which displays certain summary statistics of a set of data. Boxplots are particularly useful for assessing quickly the location, dispersion, and symmetry or skewness of a set of data, and for making comparisons of these features in two or more data sets. Boxplots can also be useful for drawing attention to possible outliers in a data set. The other topic, which is covered in Sections 2 and 3, is that of dealing with data presented in tabular form. You are, no doubt, familiar with such tables: they are common in the media and in reports and other documents. Yet it is not always straightforward to see at first glance just what information a table of data is providing, and it often helps to carry out certain calculations and/or to draw appropriate graphs to make this clearer. In this free course, some other kinds of data tables and some different approaches are covered.

Free course
16 hrs
Maths for Science Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Maths for Science

Observation, measurement and the recording of data are central activities in science. Speculation and the development of new theories are crucial as well, but ultimately the predictions resulting from those theories have to be tested against what actually happens and this can only be done by making further measurements. Whether measurements are made using simple instruments such as rulers and thermometers, or involve sophisticated devices such as electron microscopes or lasers, there are decisions to be made about how the results are to be represented, what courses of measurements will be used and the precision to which the measurements will be made. In this free course, Maths for Science, we will consider these points in turn.

Free course
14 hrs
Who says? The origins of data that gets used to make decisions Copyrighted image Icon Copyright: Scott Adams/Universal Uclick, via educational usage article icon

Science, Maths & Technology 

Who says? The origins of data that gets used to make decisions

Numbers add authority to news stories and policy decisions. Where do the numbers come from? Should we trust them?

Patterns and Landscapes Copyrighted image Icon Copyright: Production team article icon

Science, Maths & Technology 

Patterns and Landscapes

Marcus du Sautoy delves into the work of Gauss and Riemann, the two mathematicians who started to discover the order behind prime numbers

Working with charts, graphs and tables Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Working with charts, graphs and tables

When you come across information represented in charts, graphs and tables youll need to know how to interpret this information. This free course, Working with charts, graphs and tables, will help you to develop the skills you need to do this. This OpenLearn course can be used in conjunction with, More working with charts, graphs and tables, which looks into more ways to present statistical information and shows you how to use charts, graphs and tables to present your own information.

Free course
15 hrs