Statisticians must be crystal clear about their objectives before they apply analytical tools to their data. How can they ensure their analysis actually answers the question they want to ask?
In 1994, U.S. Major League baseball players went on strike in response to the teams’ owners hoping to control costs by capping salaries. The two sides disagreed about the average salaries the players were receiving. A simple enough question to settle, one might think: just look at the data and calculate the average. But not so simple a question to answer if players used the median while club owners used the mean, because they were really trying to answer different questions.
Elementary statistics textbooks define the arithmetic mean and the median, typically drawing attention to the sensitivity of the mean to outlying values in skewed distributions. Because of this sensitivity, they often give a rule of thumb that the median is to be preferred for such distributions. For example, the fact that a company’s median salary is $20,000 pa might give a more accurate impression of general employee remuneration than the fact that its mean salary is $100,000 (the CEO is paid a vast amount). The median certainly tells us that the majority of employees earn substantially less than $100,000. On the other hand, the median salary gives a misleading impression of the total wage bill, which is the product of the mean and the number of employees. The point is that which average is appropriate depends on what you want to know – it depends on the question you want to answer.
That contrast between those two averages may be familiar, but it illustrates a bigger issue: the critical importance of deciding what question you want to answer before plunging into an analysis. Moreover, the relative statistical properties of the median and mean – their variance, robustness, statistical tests of their value, etc – are irrelevant if one answers your question and the other does not.
The statistician may have (or at least should have) learnt about the limitations and requirements of their tools. But the formulation of the question to be answered is an extra-statistical matter
Statistics and data science teaching naturally focusses on how to use techniques. It assumes that the question to be formulated – call it the “substantive” question – has been formulated, and the analyst’s role is to use data to find a reliable and accurate answer to that question. That is, to apply the tools the analyst has been taught to extract meaning from the data, develop sound operational systems based on the data, and so on. The statistician may have (or at least should have) learnt about the limitations and requirements of their tools: distributional assumptions necessary for valid application, what to do in the face of incomplete data, how to handle discrete vs continuous variables, and so on. But the formulation of the question to be answered is an extra-statistical matter.
The statistician begins with that substantive question and maps it to a “statistical” question. This is a formal translation, into statistical and data terms, which is a representation of the substantive question. Once the statistical question has been stated, then statistical concepts and methods can be brought to bear. In this vein, you may be familiar with the comment attributed to George Box: “You have a big approximation and a small approximation. The big approximation is your approximation to the problem you want to solve. The small approximation is involved in getting the solution to the approximate problem”1. His “big approximation” is the mapping from the substantive question to the statistical question. And his “small approximation” is any approximation necessary to solve that statistical formulation.
The importance of the big approximation was emphasised by John Tukey, when he wrote “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise”2 (his italics). It is a universal issue, affecting all application domains, but here is pharmacist Lewis Sheiner’s take on the matter: “The intellectual illness of drug evaluation that I have discussed here can be cured, and it will be cured when we restore intellectual primacy to the questions we ask, not the methods by which we answer them”.3
But before we can go further in designing our trial, we need to know whether we are studying the “explanatory” question or the “pragmatic” question
As many have said, finding the right answer begins with the right question. Ask the wrong question and you are likely to get the wrong answer. So, if your map from the substantive question to the statistical question is poor, your answer may be poor, misleading, wrong, irrelevant, or inadequate in some other way.
By now you may well be thinking that the point is an obvious truism, a platitude. It took me a long time, and years of working in a range of different application domains, from medicine, to finance, to government, to realise just how pervasive were mistaken maps from the substantive to the statistical question; to realise how often statisticians were answering the wrong question. I opened, above, with a simple example in which different averages were appropriate for answering different questions. But here are some deeper illustrations.
Medical statisticians design and analyse clinical trials, the bedrock of modern evidence-based medicine. The core concept is the random assignment of patients to different treatment groups so that any differences in outcome can be attributed to the different treatments, at least in expectation. But before we can go further in designing our trial, we need to know whether we are studying the “explanatory” question or the “pragmatic” question.
But comparing arithmetic means of log-transformed data is answering a question very different from comparing arithmetic means on the raw data
The explanatory question tells us what biological or physiological effect the treatment actually leads to: its “real” effect, if you like. In contrast, the pragmatic question tells us what effect the treatment leads to when the treatment is used in practice, taking into account the population using it, the way patients adhere to or fail to adhere to treatment regimens, and other aspects of real-life use. Pragmatic effects overlay explanatory effects with complex social and psychological matters. The two questions are different and will require different designs and different analyses. (I can still recall my excitement when I learnt this from the book Clinical Trials by Daniel Schwartz, Robert Flamant, and Joseph Lellouch4. Amongst the various differences are the wish to avoid identifying the poorer treatment as better (pragmatic) versus the wish to avoid concluding that A is better than B (or B is better than A) when there is no difference (explanatory); whether withdrawals due to side-effects are regarded as treatment failures (pragmatic) or irrelevant since the impact on the illness of those who took the treatment is what matters (explanatory); the population to be drawn upon for the study (stringent exclusion criteria to yield a relatively homogeneous population in an explanatory study versus broadly inclusive criteria in a pragmatic study which hopes to generalise to a broad population); and other aspects of the difference.
Clinical trial researchers may object that my description is hopelessly simplified, and I agree, but that merely serves to emphasise even more the care that needs to be taken in matching the method of analysis to the question you want to answer.
Data is often transformed prior to analysis. In particular, statisticians do this so that the data conforms more closely to distributional assumptions made by the tools they are using (these matters having been taught when they learnt how to use the tools). For example, a very familiar transformation is a log-transformation, often used to make positively-skewed data follow a more symmetric distribution so that tests such as a t-test may be applied. But comparing arithmetic means of log-transformed data is answering a question very different from comparing arithmetic means on the raw data. It is, in fact, comparing the geometric means of the raw data. So, for example, if our substantive question was really concerned with a total (as with my opening baseball example) we could be misled if we compared means of log-transformed data. The statistical transformation, motivated by the requirements of the statistical tool, has led to us answering the wrong question.
It is possible that, if they had different substantive questions, both groups were right. Or perhaps neither
Interaction can be measured in various ways. In a classic case5, two groups of researchers, analysing the same data, arrived at diametrically opposite conclusions. The aim of the study was to explore whether having an intimate relationship was protective against depression as a reaction to adverse life events. That is, whether relationship and adverse event interacted.
The reason they drew different conclusions was straightforward from a mathematical perspective: they were using different definitions of interaction. One used a multiplicative definition, and the other an additive definition. But that mathematical explanation does not tell us which, if either, group was right. That depends on the precise substantive question they wanted to answer, and on which of the two statistical questions – multiplicative or additive interaction – properly represented that question. It is possible that, if they had different substantive questions, both groups were right. Or perhaps neither.
Statistical, machine learning, and AI classification tools are often evaluated on the basis of their misclassification rate – the proportion of objects they assign to the wrong class. But misclassification rate, by definition, assumes that misclassifications from class i to class j carry the same penalty for all different i and j. This is certainly sometimes reasonable (e.g. perhaps when classifying the two words “yes” and “no” in a simple two-word speech recognition system) but in many, conceivably most, situations it is not appropriate (e.g. classifying patients to disease classes, or classifying transactions to fraud/non-fraud). To minimise the loss a classification method will lead to requires penalties of the different kinds of misclassification to be taken into account. Failure to take account of the penalties in such cases is a failure of the mapping from the substantive question to its statistical formulation.
I invite you to cast your mind back and consider: in each case was it the right thing to do?
In clinical measurement, one sometimes needs to compare two methods, perhaps to see if a new (e.g. cheaper or less invasive) method can be used in place of the standard method. In such situations, the (Pearson) correlation coefficient has often been used as an indication that the two methods agree sufficiently closely. However, as one of statistics’ most highly cited papers pointed out, the correlation coefficient does not answer the substantive question: it is invariant to linear distortions of one scale relative to the other, including changing the units of measurement and it is influenced by the range of variation of the underlying true value. The authors, Bland and Altman, ask (and answer): “Why has a totally inappropriate method, the correlation coefficient, become almost universally used for this purpose?”. Or, as I might ask, “Why has a statistical question which fails to represent the substantive question properly become so widely used?”
The examples above may be caricatures of the complexity of real analyses, and I know there are many subtleties that I have glossed over, but they do contain the truth that statistical tools are sometimes applied without thinking carefully about whether they answer the question which is intended. This may sometimes be a consequence of the “hammer fallacy” (the use of a tool with which one is familiar even if it is not the ideal tool for the job), but it may represent deeper challenges of formulating the mapping from the substantive question to the statistical question.
My new book, “What’s the Question? Deciding What You Really Want to Know“7, looks in depth at the issue. It moves from familiar basic statistical tools, such as averages and simple comparisons, to machine learning, cluster analysis, correlation, regression, interaction, aggregation, discrimination, and more, giving examples of where inadequate care in question-formulation has led to the wrong answer. All of the examples above, and many more, are explored in depth. The preface of the book concludes by saying “The reader will probably have used at least some of the methods described in this book. But I invite you to cast your mind back and consider: in each case was it the right thing to do? And beyond that, are the publications which form the bedrock of your discipline providing answers to the questions you mean to ask?”
The bottom line is that it would be naïve to expect that, if you ask the wrong question, you will get the right answer.
References
- Fieberg, J.R., Shertzer, K.W., Conn, P.B., Noyce, K.V., and Garshelis, D.L. (2010) Integrated population modeling of black bears in Minnesota: implications for monitoring and management. PLoS ONE, 5(8): e12114.
- Tukey, J.W. (1962) The future of data analysis. The Annals of Mathematical Statistics, 33, 1-67.
- Sheiner, L.B. (1991) The intellectual health of clinical drug evaluation. Clinical Pharmacology and Therapeutics, 50, 1-49.
- Schwartz, D., Flamant, R., and Lellouch, J. (1980) Clinical Trials. Academic Press, London.
- Everitt, B.S. and Smith, A.M.R. (1979) Interactions in contingency tables: a brief discussion of alternative definitions. Psychological Medicine, 9, 581-583.
- Bland, J.M. and Altman, D.G. (1986) Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, February 8, 307-310.
- Hand, D.J. (2026) What’s the Question? Deciding What You Really Want to Know. Taylor and Francis, London.
David J. Hand is emeritus professor of mathematics and senior research investigator at Imperial College, London, where he formerly held the chair in statistics. He is the author of books including ‘The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day’ (2014; Bantam Press) and the recent ‘What’s the Question? Deciding What You Really Want to Know’ (2026; Taylor and Francis).
You might also like: The joy of sampling
Hospital league tables: a statistician’s view
