The BBC mislead, for a laugh *update*

[BPSDB]

It’s hard being the kind of person that takes things seriously sometimes.  Especially when you didn’t realise it was all an entertaining joke.  You may remember this BBC story about the geographical distribution of happiness in the grey and rain soaked United Kingdom.  Those unfortunate to live in the Athens of the North are arch-miserablists relying on the annual Comedy Festival to bring a sprinkling of japes and joshes to their pathetic lives while the denizens of Powys cavort and gambol and go their merry way.  Researchers

[u]sing data from the British Household Panel Survey, where people were asked about their sense of well-being, the researchers were able to draw up a map of happiness down to district level across England, Scotland and Wales.

After adding in factors such as employment, health and educational qualifications, the team found that the area of Brecknock, Montgomery and Radnor in Powys was the happiest place.
[...]
Edinburgh, despite its cosmopolitan reputation and internationally acclaimed fringe festival, was bottom of the list.

This article had the headline “Britains[sic] Happiest Places Mapped“, was published on the BBC website with the author making sure to acknowledge that this research was carried out by academics using proper data, from the “British Household Panel Survey” no less, so I took it seriously. Unfortunately half way down the article was this statement:

However, the researchers stress that the variations between different places in Britain are not statistically significant.

Oh. If a result is statistically significant it is unlikely to have occurred by chance, conversely if a result is not statistically significant then it likely occurred by chance. This means that levels of happiness only display chance variation when correlated with geographical locations. Thus there is no link between geography and happiness. Therefore it is false, wrong, untrue and deceitful to state that some geographical locations are happier than others. The data does not show this.
Inspired by the badscience forum thread on this article I decided to point this out to the BBC .

Your story on Britain’s happiest places contains the following quote “the researchers stress that the variations between different places in Britain are not statistically significant.”
If a difference is not statistically significant then it is inseparable from chance variation and no statement about the differences in happiness between different towns or cities can be made other than to state there aren’t any measurable differences.

Your whole story is misleading as the title and several paragraphs at the beginning all tell of a difference before you reach the quote on significance.

Considering that your Magazine section has been running a series of very interesting articles on the misuse of statistics it is very embarrassing that you could write such a story as the above.

I apologise for the slightly indignant and pompous tone but when corresponding with an establishment icon such as the BBC I seem to channel the demeanour and pedantry of a portly middle aged bank manager. Despite this the BBC were kind and courteous enough to send me the following reply:

I take your point. We felt that as the story was of a light-hearted nature, and that as the conclusions were not of great importance, or significance, it made for a light and entertaining read.

The researcher did uncover geographical differences, albeit small.

We did stress in the copy that the differences were not statistically significant, and that factors other than geography were more important.

Being a serious minded person I am taking this reply at face value.  They are quite right in that the conclusions were not of great importance, because of that lack of statistical significance. However, I am not sure I follow the logic that reporting a non-finding as a finding is light and entertaining.  To me it seems slightly unethical. I am quite capable of amusing an audience by telling them a tall tale but I suspect they would react badly if I pretended that the story was true despite it clearly not being so.  My audience might even start to heckle and throw things if I continued to state that there was evidence to back me up, albeit no evidence really.  No statistical significance means chance variation not small differences.

If this was all there was to the story I would have let it lie and you would not be reading these words.  However Allo V Psycho over on the badscience forums has been kind enough to provide more information.

The BBC managed to send a camera crew and exuberantly breathy reporter Colette Hume to happy Haye-on-Wye where, to an inevitable REM soundtrack (a song supposedly about the Tianeman Square massacre btw), the merry middle classes prosper in their good fortune and share their sunbeams of delight.

“This is Hay-on-Wye, it’s full of very happy people. Researchers mapped happiness and well being in 273 parts of the country. They say environment, air quality and crime rates all have an impact on how happy we are”.

However, no mention was made of the lack of statistical significance underpinning the article and the viewer is led to believe that Haye-on-Wye is the happiest place in Britain.  Frankly this rather makes a mockery of the response I got from the BBC. It is hard to conclude that the broadcast piece was anything other than misleading, erroneous and pig ignorant.

It is really quite distressing that television reporters feel that facts, evidence and honesty are not important in presenting a story.  There is absolutely no justification for not understanding basic concepts such as statistical significance. This casual disregard for scientific accuracy is sadly all too typical of today’s media. I assume that a certain level of intellect is required to become a journalist, editor or producer and no doubt a university education helps as well. This would mean it is unlikely that the individuals involved in this story were too thick to know about statistics, I can only assume that they decided to deliberately gloss over the lack of statistical significance in the written piece and deliberately ignore it in the broadcast piece. One wonders what the BBC really think of the intellectual capacity of their audience. jdc325 recently claimed that ‘The Media Think You Are Morons‘ because of their refusal to reference scientific papers whose findings they are reporting. It is hard to disagree with his opinion.

Thanks to Ithika, soveda, Ambrielle, science_fox, Ephiny, Chris Preston, James MacKay, Dr* T and Allo V Psycho

Update

This has been covered by Dougal Stanton, Apgaylard, Ben Goldacre and by Mark Pack over at LibDemVoice.  Apgaylard has published his correspondence with the BBC on this matter and has even managed to get a grudging apology while Ben Goldacre has this revealing quote from the academic behind the research, Dr Dimitris Ballas,

“I tried to explain issues of significance to the journalists who interviewed me. Most did not want to know.”

Also, here is my latest correspondence with the BBC on this issue,

Thank you for your response. However, I feel it is inadequate given
that a broadcast piece on the same story failed to mention the lack of
statistical significance while asserting the same conclusions.
http://news.bbc.co.uk/1/hi/uk/7585449.stm
“The most sparsely populated county in Wales has been revealed as the happiest place in the UK.

Meanwhile, according to the study from the Universities of Sheffield and Manchester, Edinburgh is the most miserable.

Colette Hume found out why people in Hay-on-Wye are so cheerful. “

Quite frankly it beggars belief that the BBC were willing to send a reporter and crew to Hay-on-Wye to report made up facts without mentioning the lack of statistical significance meaning that these facts were not true.

I can only conclude that the public were deceived for the purposes of light entertainment.

And the reply,

Many thanks for your message. My responsibility is for the text on the online site, and I was careful to include the fact that the statistics were not significant. I agree that the television package should have made reference to that fact, and while I cannot speak for my television colleagues I assume that the intention of their piece was to provide a “well I never” moment, rather than hard science.