A key concept in drawing an accurate conclusion from statistical data is considering the possibility of sampling errors. These errors can be explained by an analogy which compares the processes involved in legal trials, and the processes in clinical trials.
Clinical trials of a new drug may involve four phases. The aim of Phase I is to assess safety issues associated with the drug. Typically, this phase is conducted on a relatively small number of healthy participants. Phase II investigations are intended to verify the findings from Phase I on participants who are patients, and to make a preliminary assessment of the efficacy of the drug. Phase III clinical trials are devoted to comparing the new drug with the current standard treatment, or perhaps no treatment using a placebo. Often the term ‘clinical trial’ is used loosely to describe a Phase III clinical trial. Phase IV trials monitor the new drug after it has been released on the market.
To explain the concept of sampling errors, let’s compare legal trials with clinical trials. In the table below, one column contains features of a legal trial. The other column presents the analogous features of a Phase III clinical trial to compare a new drug with the standard treatment.
| Legal trial | Clinical trial |
| Initially, we assume that the defendant is innocent. | Initially, we assume that the new drug is no better than the standard treatment. (This is the “null hypothesis”.) |
| The alternative possibility is that the defendant is guilty. | The alternative possibility is that the new drug is superior to the standard treatment. (This is the “alternative hypothesis”.) |
| Evidence is gathered and presented to the court. | A Phase III clinical trial is conducted; data are collected and analysed. |
| Decision of the court can be (i) guilty or (ii) not guilty. Note that the court does not decide that the person is innocent. |
The conclusion of the analysis can be either (i) the data indicate that new drug is superior, or (ii) the data does not indicate that new drug is superior. |
|
There are two types of potential error in the decision. (1) An innocent person is found to be guilty. (2) A guilty person is found to be not guilty. |
There are two types of potential error in the conclusions. (Type 1) The data indicate that the new drug is superior whereas it is not. (Type 2) The data does not indicate that the new drug is superior even though it is. |
In clinical trials, patients may be allocated at random to two treatment groups. We may regard our two samples of patients as being chosen at random from two populations of patients. There is chance involved in this process. This randomness can lead to these two types of sampling errors, called Type 1 and Type 2 respectively.
A Type 1 error occurs when one concludes from the results of the experiment that the new drug is better than the standard drug, even though this is not the case. A Type 2 error when one derives that the new drug is not better than the standard one, even though it is better. We can see similar potential errors in the legal process.
Not only do statisticians admit that these errors can occur, they control the probability that they will occur, by judiciously choosing the sample size. One endeavours to choose a sample size so that the probability of Type 1 error and the probability of Type 2 error are small, typically 5%, or less, depending on the context of the problem. The researcher can set these probabilities in advance, and then calculate the required sample size.
The key point is that often, the sample size can be calculated and there are mathematical formulae for calculating sample sizes2. Deciding on sample sizes is not a matter of guess work.
The importance of calculating sample sizes scientifically cannot be overstated. Any attempt to test some hypothesis without it comes down to an ethical issue of wasting the time of participants, researchers and resources3. Although the above analogy is set in the medical context of a clinical trial, the arguments may apply to any research project that involves statistical analysis4.
References
- 1. Pocock, S.J. (1983) Clinical Trials: A Practical Approach. Chichester, John Wiley and Sons.
- 2. Machin, D.; Campbell M.J.; Tan S.B. et al. (2009) Sample Size Tables for Clinical Studies. Third ed., Chichester: John Wiley and Sons.
- 3. Altman, D. (1980) Statistics and ethics in medical research III: How large a sample? British Medical Journal 281: 1336-1338.
- 4. Billingham SAM, Whithead AL, and Julious SA (2013) An audit of sample sizes for pilot and feasibility trials being undertaken in the United Kingdom registered in the United Kingdom Clinical Network database. BMC Research Methodology, 13:104.
