The perils of privately commisioned polls – Auschwitz and Beer
Posted by gimpy on March 9, 2009
Both today’s Telegraph and Yahoo News carry a press association story condemning children for not understanding the holocaust and confusing Auschwitz with a beer. According to the Telegraph:
a survey of more than 1,000 secondary school pupils aged 11-16 revealed that a quarter still did not know its purpose.
Of those, about 10 per cent were not sure what it was, 8 per cent thought it was a country bordering Germany, 2 per cent thought it was a beer, the same proportion said it was a religious festival and a further 1 per cent said it was a type of bread.
This survey was carried out by the firm Dubit Limited who specialise in market research involving teenagers, and commissioned by Miramax and the London Jewish Cultural Centre to promote the release of the film The Boy in the Striped Pyjamas on DVD. The data from the survey appears worrying and the reportage is quick to condemn the perceived ignorance of the young about the greatest atrocity to have taken place on European soil.
However there are a number of problems with this survey, as a private company whose polling method is clearly of commercial value, Dubit do not describe in any detail their methodologies on their site. This means that the validity of the poll in terms of the questions asked, statistical weightings used and data collection method are impossible to analyse and such research would not pass muster in the halls of academia. It is not possible to know how credible this poll can be with respect to its methodology and data set.
But this is not the greatest concern I have with this story as it is reported. The curriculum for history specificially states that the Holocaust is to be covered in some detail, but not until Year 9, when the pupils are 13-14 years old. This means that a large part of the dataset used by Dubit, covering ages 11-16, would not have studied the Holocaust, thus skewing the findings of the poll. Given this it is a fairly positive finding that three quarters of the polled students know the purpose of Auschwitz. This was not the line taken by the press, Dubit, Miramax and the London Jewish Cultural Centre. Were none of these organisations or individual journalists capable of checking what proportion of 11-16 years olds would be expected to understand the Holocaust based on the curriculum, or did they just not care? Miramax have publicised their film, the London Jewish Cultural Centre have increased their media profile and Dubit have taken money for a useless survey. The only loser here is the public who have been misled by all of the above.
Frank the SciencePunk said
I guess “87% of children correctly identify Auschwitz” isn’t such a great headline.
softestpawn said
Dozy Meeja.
Your badscience title was better though; “children too young to have been taught about the Holocaust, don’t know about the Holocaust”
jdc325 said
“Dozy Meeja”
Heh. Fact-checking doesn’t sell papers – but inaccurate information and/or headlines do. Apparently.
loopedstrangeness said
So the Telegraph gets some nice, sciencey sounding “data” to back up the usual lament about the state of the youth today and everyone else gets the publicity they were after.
Interesting post, and has made me think about how much stuff out there that’s not obviously “bad science” starts to fall apart when you start digging.
Neuroskeptic said
“and has made me think about how much stuff out there that’s not obviously “bad science” starts to fall apart when you start digging.”
Right. I remember a radio show I heard once – I forget what it was – and they were talking about how there had been a natural disaster and various relief agencies turned up to help. And they all started competing to make it sound as bad as possible. One would give a press conference saying 1000 people had died, then another would say it was 2000, and so forth.
That really surprised me because I thought charities would be above that kind of thing. But if you think about it, it’s not all that surprising. They stood to gain something by making the situation sound worse than it was (donations), and there was no-one around to call bullshit on them, so from one perspective they would have been mad not to do it. And it was all in a good cause – people had died and more were in danger of dying. Just not as many as they ended up saying.
I guess the lesson is that you’ve got to be skeptical even of people with good intentions.
Ian said
Hmm – when you see that it’s only 2% that relate Auschwitz to beer, you have to conclude that it’s the media who draw the conclusion / create the soundbite, not the research company! But at least the media did give you the facts, rather than hide the stats.
Plus – did you know – that “in the halls of academia” researchers do not need to be governed by the rules of the Market Research Society? So research from a reputable commercial agency is more likely to have the checks and balances in place to ensure fair reporting. Maybe academics are the ones that should be analysed more heavily when it comes to soundbites and data. I have seen so many academic opinions backed up by tiny polls, and qual research being treated as quan.
Cards on the table – I’m from the research company, Dubit. Gimpy says that we should have some detailed methodologies on our site: not sure what you mean by that, but for the avoidance of doubt, we have a representative panel of 37,000 young people across the UK, and abide by all best practice. We have a full team of qualified data analysts and they are proud of what they do, so we don’t cut corners. We don’t tell clients what to research however, so it is up to them to decide age groupings, and we always ensure we have at least 600 kids in the sample size. In this case it was 1200, so pretty thorough, and was weighted to bring in line with the UK youth population.
Hope that clears up the factual parts – I’ll let you guys debate what the media did with it!
gimpy said
Thank you for your response Ian. But I’m not sure I understand it, you state that it is the media that draw the conclusion/create the soundbite in your first sentence, but then in your second you claim you have sufficient checks and balances to ensure fair reporting. So are the media responsible for not taking into account the age at which the history of the Holocaust is taught or are you?
If the media are misrepresenting your poll then what steps will you to take to correct this?
The advantage of academia is that the raw data is made available on the publication of research. Would you be willing to release the raw data, personal details redacted, so we can judge for our selves whether your team of analysts have done a good job?
Ian said
Ah, sorry for the confusion. By fair reporting I mean that the checks and balances ensure that the data is statistically accurate and representative of the audience sampled in the survey. Not that the “journalistic” reporting of the data is fair – we can’t be accountable with how people interpret data.
Now, they do state that the number equating Auschwitz with Beer is only 2%, which is correct. They have actually given you the data. Anyone reading the article would see that, I think, and understand that the headline is quoting only a part of the survey. I am not sure why you feel the data is suspect simply because a client has chosen to pick up on one of the statistics?
Remember, we do the survey, we report the data, and we are not in a position to question how it is used or interpreted by a client. On your point about age group, in many cases, the client is in a better position to determine the audience they require to sample, as they are closer to the issue or market. Researchers are not always asked to review the audience or subject matter in any depth before conducting a survey.
gimpy said
Once again thank you Ian, I really appreciate your taking the time to comment here.
Perhaps I have been a bit harsh in apportioning blame on you for the actions of the media but I still have concerns over the presentation of your data. Did your report mention that the age gap covered would include children who had not covered the Holocaust at school so any discrepancy could be explained by that? Also, would you consider releasing your data set? I am making no allegation of improprietary, I would just like to see whether there is a correlation between ignorance about the Holocaust and age/educational attainment.
Daily Express, Better You, The Leadership Factor and Laura Clout: the Yes Minister approach to market research « Holford Watch: Patrick Holford, nutritionism and bad science said
[...] point to note is an interesting comment from a market research professional on Gimpy’s blog – claiming that “in the halls of academia” researchers do not need to be governed by the rules of the [...]
Bob said
Is 1000 a large enough sample for any of this to be relevant? I mean 1000 could be considered a large number but compared to the size of a school I’m not sure it’s significant.
Mr. Barnett said
I believe that MORI use a sample size of 1058 for their polls, perhaps it’s only the political ones, though.
And yes, assuming that the holocaust, which, if I have read correctly elsewhere had more Christian and Gypsy victims than Jews, is taught in the middle of Y9, anything more than 50% is a bonus.
I don’t think you can call the primary school contact with the Anne Frank diary as teaching, since the underlying theme is of the Franks’ fear of Nazi opression and the specific purpose of Auschwitz and similar camps is not emphasised. Indeed, the full horror was not known to the pulic anywhere during the war.
Then again, I teach chemistry, what do I know?
gimpy said
Mr. Barnett, I’m not sure what your argument is. I’m pretty sure it’s taught that Jews weren’t the only victims of the Holocaust but they did form the majority of its victims and were the largest ethnic or other group singled out by the Nazis for extermination.
Steve Jones said
The Telegraph article (at least) did not have anything “condemning children for not understanding the holocaust”. Even if the level of understanding is unacceptable (open to debate and analysis of course), then it would be far more logical to put the blame on society, educational policy and so on. But I don’t see anything in the article condemning the children, and that appears to be your own take on the aim of the article.
Now I have every reason to doubt the neutrality and disinterest of the parties that sponsored this survey, and I’d certainly like to see journalists take a rather more sceptical line on such items. However, I for one don’t read this as a “condemn the children” article any more than anybody is blaming them for the appalling standard of many GCSE scientific papers.
gimpy said
Thanks for the comment Steve. Sorry my link to the Yahoo article has gone and I’ve noticed that the Telegraph article was updated after I wrote this post (I really should make a habit of archiving more). As regards your comment on my take, I interpreted the media’s consistent focus on confusing Auschwitz with a type of beer as being part of a regular series bemoaning the state of modern education and the ill informed youths that come out of the system. This is of course my personal opinion and I don’t think I’ve pretended otherwise.
Steve Jones said
Well there is a general point to be made that any shortcomings of understanding or behaviour of new generations are, to a large extent, due to the policies of the preceding ones. So to condemn younger generations as such (and I’ve no doubt that there are some that do so) is really a condemnation of the policies and society that got them there (or maybe just errors in expectation).
As an example, I can certainly see big mistakes being made by some of the no-doubt well-intentioned policies of politicians to make science “more accessible” or “relevant”. In practice, what they appear to have done in many cases is replace the consistent practice and understanding of the foundations of science and scientific method with what are often just superficial points about some manifestation of technology and related social policy. Talking about science rather than doing it. To raise such a point is not to condemn those children affected, but to challenge some of the basis of educational policy. Of course, defensive politicians often react to this by saying that the critics of perceived loss of standards are criticising the children – this is a regular game that is played around the time of the reporting of examination results. Whatever the truth or otherwise of changes in standards, it is this type of deflecting criticism from the actual target that annoys me.
Of course I haven’t seen the original version of the Telegraph’s article, but I’d be rather surprised if it was blaming the children as such rather than the way they had been educated. Now it could well be that much of the press does have a consistent theme bemoaning the state of modern education and the results of it, but that’s a very different thing to blaming the children. I get the impression you think there is little basis in fact of that, but in the area I know a little of (in that I am a physics graduate), it is certainly the case that the numbers taking that subject at A level have fallen, and I believe something similar has happened with mathematics at A level as reported in various publications like the BBC, Independent and Guardian. So even if there hasn’t been an issue of declining standards (and Ofqual have recently condemned physics GCSE standards) then any policies there have been to make science more attractive would appear to have failed woefully.
So – in conclusion, I think move the apparent target of the criticism from educational policy to children is a tactic beloved of politicians, and to be brutally honest, in the area of at least some part of the educational policy (physics and mathematics), then I think there is a case to answer and that this isn’t just a media witch-hunt.
DignGuittee said
Best Legal Steroid. Pump n Pose is a pioneer in the Bodybuilding Supplement Industry. Our field is in the advancement, study, and production of muscular site enhancements, anabolic steroid supplements, and body building supplements. We have a variety of products to enhance your physique. It is usually recommended to order stacks, a combination of supplements. One stack might consist of anadroll, deccabolan, and dianabol.
http://www.musclelabsusa.com/images/muscular-development-ad2.jpg