Previously published in the Daily Maverick
Wanting to believe something to be true has no effect on whether it is, or becomes true. We all know this, at least in the abstract (except for Rhonda Byrne and her readers). Nevertheless, in the race to be the first to compress some insight or factoid into a 140 character tweet or a provocative headline, confirmation bias can take over. Instead of suspending judgement until we know all the facts, we sometimes ignore our doubts and regard incomplete, misleading or even false information as persuasive.
Scientific literacy is the biggest loser here, because in the struggle to make sense of things, we forget that our beliefs become better justified through surviving our attempts at falsifying them – not when we confirm them, or think that we’ve done so thanks to unreliable information. A trivial example of this can be found in many of the 330 (at the time of writing) comments to the News24 article describing the results of a recent survey on religious belief.
According to the article, the Win-Gallup International Religiosity and Atheism Index found that the percentage of South African people who consider themselves religious has dropped from 83% in 2005 to 64% in 2012. Now, all the local media houses that covered this simply reproduced the South African Press Association (Sapa) newsfeed, so there isn’t a science journalist or editor that I can call to ask why nobody did any rudimentary fact-checking before reproducing this and other claims.
Because spending merely a couple of minutes on the survey data (pdf) reveals that – by contrast to the claim offered on page six that “in each country a national probability sample of around 1000 men and women” were surveyed, the country-by-country breakdown on page 15 tells us that only 200 South Africans were surveyed.
Assuming that this sample was a representative one, the margin of error now becomes something closer to 7% rather than the 3-5% claimed by Win-Gallup. And if a similarly low number of South Africans were surveyed in 2005 (that data is not publicly available), a pessimistic reading of the data results in a shift from 76% to 73% in the number of people who consider themselves religious.
Alternatively, if the sample was drawn entirely from a certain Province or biased in some other fashion, the results from 200 interviews become virtually meaningless. Unfortunately, although the MD of the Gauteng-based Topline Research Solutions (who are listed as having conducted the survey work in South Africa) responded to my email enquiry, he referred questions regarding the sampling methodology to Gallup’s “Group Head for Opinion Research” – based in Pakistan.
Emails to the other two Topline staffers listed on the Gallup poll bounced with a “user unknown” error – even though one of the two is still listed as being the Sales and Marketing Manager on their website. Make of this what you will, but I can’t say that I’m left feeling confident that the South African sample exists at all, never mind being representative.
There are other reasons to immediately be suspicious of this data. The 2001 census counted 79.77% of South Africans as being Christian, never mind the more general “religious”. The 2011 census data won’t provide any update on this percentage, seeing as the question on religion was dropped, but a drop from 84% (all except the non-religious and “undetermined” in the 2001 census) to 64% seems highly implausible, judging by the frequency with which religious sentiments are uttered and endorsed in popular discussion.
Implausible doesn’t mean untrue, of course – it might well be that the numbers have shifted as described in the survey. But if they have, South Africa would be less religious than the United Kingdom is, at least according to their 2011 census which had the non-religious accounting for 33% of their population. And if you believe that we’re more godless than the UK, your name is probably Errol Naidoo.
Then, the language of the (single) question in the Win-Gallup poll doesn’t allow for quality data. Respondents were asked the following question: “Irrespective of whether you attend a place of worship or not, would you say you are a religious person, not a religious person or a convinced atheist?”
“Religious person” is somewhat ambiguous, as it could imply something formal, whether or not the responded attends a place of worship. If I were one of those New Ager-types who thought I had a personal relationship with something ineffable, whether god or angel, I might say that I’m not religious, while any sane onlooker would assert that I most certainly am. Being “religious” is a label of identity, and is chosen or rejected for a range of idiosyncratic reasons, making this question very difficult to answer or interpret.
Likewise, I’m not even sure that I’d call myself a “convinced atheist”, because while the existence of god(s) might well be the thing that I doubt most, if being “convinced” requires being certain, I’d have to tick the “not a religious person” box. And again, all who know me would most likely have predicted a different selection.
A finer-grained account of what being “religious” means to those who describe themselves as such is a different matter, because it can inform strategy whether you’re on the religious or the non-religious side of the debate. The data from the Win-Gallup poll, on the other hand, simply feeds into our confirmation bias. It allows for the religious to lament and the irreligious to gloat, neither for any good or principled reason.
And speaking of principle and good reason, a concluding note on those non-existent science journalists: we might never again see such a job description, except for the few who still survive at a handful of newspapers. But this isn’t an excuse to simply recycle wire copy, whether you’re a newspaper or a member of the public. The fact that experts are no longer doing the filtering for us means we need to pay more attention – not simply become more gullible.