Let's ignore the issue of whether the respondents are representative of the population as a whole. Certainly we would expect the type of people who respond to a BBC Radio 5 survey to not adequately represent the whole UK population both in age and gender, as well as the range of political and religious beliefs. I also want to ignore the form of the questions, how they were asked, and how many questions were on the complete survey.
These are important issues for deciding whether the survey results have any national importance, and a strong argument can be made that one thousand people, even when re-weighted, do not speak for the whole UK population of 63 million.
What is most interesting is trying to understand what type of people disagree with or would not attend a same-sex marriage. Within the article a range of statistics are reported, but as you will see they might not give you all the information you want – not enough detail is reported.
The key concept is something called cross-tabulation, arranging the responses (the yes or no to attending) by multiple features. So you do not report percentages just by age or sex, but by all combinations of age and sex together.
The survey of 1,007 people reports that '22% said they would spurn an invitation to attend a same-sex wedding ceremony'. This gives us one summary number (aka a statistic) called a marginal percentage, let's see why when we reconstruct the following table.
We know the total number of people surveyed and the overall percentage that would not attend. This is in the margin of the table on the left, hence a marginal percentage:
| Sex: | Female | Male | Total | |
| Attend: | Yes | |||
| No | ||||
| Total | 1,007 | |||
| Sex: | Female | Male | Total | |
| Attend: | Yes | 78% | ||
| No | 22% | |||
| Total | 1,007 | |||
| Sex: | Female | Male | Total | |
| Attend: | Yes | 785 | ||
| No | 222 | |||
| Total | 1,007 |
We are not told what percentage of responders were men directly, so we could guess it was 50% – if the survey was representing the whole population.
In fact we do not need to guess at the percentage of responders since we're also given the percentages of people who would not attend by sex, '29% of men asked said they would not attend, compared with 16% of women'. Using this we can work out what the table might look like (this might not exactly match the real data, since we've had to round to whole numbers):
| Sex: | Female | Male | Total | |
| Attend: | Yes | 455 | 330 | 785 |
| No | 87 | 135 | 222 | |
| Total | 542 | 465 | 1,007 |
Hence 46% (465/1007) of the responders were men. Although it has taken some work, I have been able to reconstruct the table so far and see the actual numbers.
It was also reported that 'younger people were more likely to support same-sex marriage, with 80% of 18 to 34-year-olds backing it, compared with 44% of over-65s'. Importantly these percentages are for a different question, younger people supporting same-sex marriage does not automatically lead to younger people attending a ceremony, nor do we know the relationship between supporting and attending. Why now switch to reporting a different question? And what about us 35 to 64-year-olds?
At this point I have no hope of reconstructing the table of who would not attend by sex and age, there is not enough information about the 1,007 surveyed. So how many older men or women would not attend is left a mystery. To better understand this survey we need the actual numbers that are missing from the table below.
| Sex: | Female | Male | Total | |||||
| Age: | 18-34 | 35-64 | 65+ | 18-34 | 35-64 | 65+ | ||
| Attend: | Yes | 785 | ||||||
| No | 222 | |||||||
| Total | 1,007 | |||||||
We have an unclear picture of who would not attend, and an even vaguer idea of who supports same-sex marriage. By who, we mean people counted by age and by sex. This is important for anyone who wants to understand the changing attitudes of the population over time, and campaigners and policy makers trying to influence public opinion. As is commonly noted, older people are more likely to vote than younger people.
I will highlight a single comment by Robert Pigott in the article, 'Most religious groups remain opposed – and Radio 5 Live's poll shows they are not alone in deeply held objections to gay marriage'. About 75% of the UK population reported belonging to a religion, so is that different to belonging to a religious group? Can we further divide people by being in a religious group, having a religion but not being in a religious group, and not having a religion – how does that change with age and sex? If the fifth of people who would not attend are also members of a religious group, then they are alone in their objections to same-sex marriage – we cannot say from the survey as reported.
It is clear that one thousand people are really not enough to reflect the views of the nation, but really we want to know how different groups of people feel. In our last table we have divided everyone into twelve groups: by male/female, by 18-34/35-64/65+, and by would they attend a same-sex wedding or not.
Perhaps as few as 100 people are deciding how certain groups of the population – and by that we mean you, the reader – feel about same-sex marriage, and we're no closer to finding out how many older men and women who are religious (but not part of a religious group) would not attend!
|
To complete the table we could try guessing the proportion of male responders until the percentages matched those reported. For example, assuming 70% of responders were women and using that 16% of them would not attend, we have the following steps: |
||||
| Sex: | Female | Male | Total | |
| Attend: | Yes | 785 | ||
| No | 222 | |||
| Total | 70% | 30% | 1,007 | |
| Sex: | Female | Male | Total | |
| Attend: | Yes | 785 | ||
| No | 222 | |||
| Total | 705 | 302 | 1,007 | |
| Sex: | Female | Male | Total | |
| Attend: | Yes | 84% | 785 | |
| No | 16% | 222 | ||
| Total | 705 | 302 | 1,007 | |
| Sex: | Female | Male | Total | |
| Attend: | Yes | 592 | 785 | |
| No | 113 | 222 | ||
| Total | 705 | 302 | 1,007 | |
| Sex: | Female | Male | Total | |
| Attend: | Yes | 592 | 193 | 785 |
| No | 113 | 109 | 222 | |
| Total | 705 | 302 | 1,007 | |
|
However, in this table 36% (109/302) of men would not attend, which is very different from the reported 29%. Our assumption of 70% female responders is wrong. We could keep changing the percentage of female responders until we get 29%. Alternatively, instead of trial and error, we can work the table out directly using some mathematics/algebra (a set of four simultaneous equations). |
||||
Footnotes
- Thankfully the article reports the number of people who took part in the survey, but it then proceeds to report only percentages – these sometimes hide the fact that there are very few people answering. Writers should always state the percentage and actual counts of people. For example, 22% (222/1007) of people surveyed would not attend.
-
More detailed results from the survey are available on the ComRes website revealing that three questions were asked in a telephone survey, presumably as well as questions such as the responders age and sex. ComRes detail their methodology, in particular that the survey is corrected to re-weight the sample for 'sex, age, social class, household tenure, work status, number of cars in the household and whether or not respondent has taken a foreign holiday in the last 3 years' against a reference sample. Although more detail is given in the available PDF download, it is still presented as marginal totals and marginal percentages – the number of men who support, the number of 18-24-year-olds who support, but not the number of men aged 18-24 who support same-sex marriage. We need cross-tabulation to see how support changes with age and sex!
