Review: Skeptics in the Pub – The Canalhouse – 14 April

Lies, damn lies & statistics

Jenny Freeman, associate professor of medical statistics at the University of Leeds, came to The Canalhouse to talk about statistical literacy. The talk was looking at critical numbers and what questions we need to ask about them.

Jenny starts by looking at the difference between data, information and knowledge. Two numbers such as 2.9 and 3.0 are data. A bridge height of 2.9m and a lorry height of 3.0m would be information. However, the lorry driver still needs the knowledge to actually avoid the bridge. Especially as lorry drivers hitting rail bridges are charged by the hour for delays they cause (this can be up to £3,000 per hour for some of the routes into London)

The first question that we need to ask about a number is, “is it a big number?” For example if six people get six after taking a medicine is it a problem? If it’s six out of six then it is but if it’s six out of a million then it isn’t.

In 2013, the government announced £150M for school kitchens. Is that a lot of money? How many schools are there? How many pupils? There are around 17,000 schools in the country with 4.2M pupils. That works out around £9,000 per school. However, even the very cheapest commercial oven costs £1,000.

Recently the Conservative Party announced a policy to increase the inheritance threshold to £1M, up from the current limit of £325,000. Currently only 5% of people in this country pay anything at all.

When Facebook claim that 1.4M pictures are uploaded every second, is that realistic? It works out at around 240 pictures per user per day.

Next Jenny looked at framing. Saying that 1% of people had an adverse reaction to something sounds much worse than saying that 99% of people felt better.

Percentages smaller than 1% are difficult to interpret. Hence it’s much easier to look at 3 out of 10,000 rather than 0.03% Percentages in general can cause some confusion though. For example when VAT went from 17.5% to 20% it wasn’t a 2.5% increase it was a 14.3% increase (it was a 2.5 percentage point increase)

We also need to think about where numbers come from. When we read that it costs on average £20,000 to get married, who has provided those numbers? It turns out that they come from a wedding magazine who might just have some advertising interests in mind.

Next in the spotlight are polls and surveys. Who is asking the questions? Who is being asked; is there any selection bias happening? What is being asked; could there be any response bias?

A lot of newspapers ran an article following the Scottish independence referendum that said from 1,000 people surveyed, 71% of 16-17 year olds voted for independence. That sounds like quite an impressive number until you drill into the data and see that there were only 14 16-17 year olds actually surveyed! 10 out of 14 doesn’t make for such an impressive headline though.

Then we should be looking at what’s actually being measured. Looking at the example above of investing a lot of money into nutrition in schools; what will be the impact? How will they measure success? Do you look at improved attendance, concentration and behaviour? Do you look at reduced truancy or obesity?

Jenny then moves onto average. A lot of times this is used interchangeably with typical when it isn’t and a lot of times, the type of average isn’t mentioned. For example in the UK, the modal average (most common) for wages is £275 whereas the median (the middle one) is £ 377 and the mean (the one you remember from school where you add everything up and divide by the number of things) is £463. So depending on how you were trying to spin things, you could say that any of them are the average.

Next it’s a particular favourite of mine the difference between correlation and causation. A study in America showed that there was correlation between the per capita cheese consumption and the no of people who die from becoming entangled in their bed sheets. However, there is clearly no causation there (unless people are writhing about more due to cheese dreams and then dying)

We also need to remember that what is true at population level may not be true at an individual level an vice versa. For example a recent article claimed correlation between high BMI and longer life expectancy. So, does that mean that we should all head out to McDonalds? Well, obviously not.

People with high BMI are more likely to live in richer countries that have better healthcare. People who are in ill health tend to lose weight. Also, when you’re clearly overweight, there is more likely to be some kind of intervention on your behalf.

Then we hear about probability. This can be one of those things that it’s quite hard to grasp as it’s often counter-intuitive. For example, despite the odds, most weeks somebody wins the lottery.

The media are especially bad at reporting on probability. At least a couple of times per year, we’ll see a story about a family who have had three children all on the same day. The odds of this are then presented as being 48,000,000-1 but are in fact 137,000-1.

They’re also poor at signalling the difference between absolute risk and relative risk. How many time do we hear that doing something will “double your risk of cancer”? A recent story claimed that eating bacon increases your chances of getting colorectal cancer by 20%. How bad is that?

5 in every 100 people get colorectal cancer. If they all eat three rashers of bacon everyday then it goes up to 6 out of 100. So, the relative risk is 20% as reported but the absolute risk increase is actually only 1%. Plus you need to consider that this is based on 100 people eating 50g of bacon every day.

Jenny finishes her talk with a list of what to think about when looking at numbers in the media:

  • What is being counted?
  • How?
  • How big?
  • Have the numbers gone up/down?
  • How certain?
  • How valuable?
  • Average?
  • Are there any targets?
  • Think about correlation/causation
  • Risk

During the following Q and A session, we learn that former doctor Andrew Wakefield based his fraudulent claims that there is a link between vaccination and autism on a sample size of just 11 children. That was a story that is still putting lives at risk today. On a lighter note, we also learnt that Florence Nightingale was the first female fellow of the Royal Statistic Society.

It was an interesting talk but I’m not sure that it was pitched at the right level for a room of sceptics. Personally, I’d read about a lot of the above in blogs by people such as Ben Goldacre. Speaking with some of the audience afterwards, they were also familiar with a lot of the concepts. So, while this would have been an ideal talk for more of a “lay” audience, I would have liked a bit less breadth and a bit more depth on some of the subjects.

Skeptics in the Pub returns to The Canalhouse on 12th May at 7:30pm where Iszi Lawrence presents The Z-List Dead List, for more information visit the official website

Review by Gav Squires

 

Join our mailing list

Sign up to receive our regular newsletter

Leave a Reply

Your email address will not be published.