Wednesday, 29 September 2010

Misusing numbers to sell project results

The more I know about statistics, the less confident I am about my own use of numbers and the more suspicious I am of any publicised data. The oracle on this topic is Dr Ben Goldacre, author of Bad Science, one of my favourite reads.  A new book out by Charles Seife called Proofiness; the dark art of mathematical deception is along similar lines.

When it comes to scaling up the results of improvement work across a wider population, the danger of inflating the opportunity is significant. These are some of the numerical problems that have come my way in the past few week:

  1. Hospital a got a 30% improvement, therefore if we multiply this across the whole county, this 30% improvement will give us a £1.3 billion saving.  Humpf? So I need to know how representative of the national picture was Hospital A. I need to know the numbers and not the percentages behind this statement. I need to know how generalisable the improvement process is. And that is before we get to understanding how the savings are calculated.
  2. Pathway b is delivering 212.3 more patients in a shorter time of 8 days.  Humpf? More than what and shorter than what? The 212.3 gives me the wobbles because I've never seen 0.3 of a patient. Seife discusses the art and deceptiveness of the specific number in his book - see a relevant excerpt.
  3. 87% of hospitals have adopted the use product y. 87% of what population - national, region, the city? What counts as use - have they used it once or are they using it all the time. Did they try it out or have they made it part of their regular routine? Did just one ward test it once or have more (all?) wards tested it and are using it all the time?  These questions are important in understanding the true nature of what has, or has not, been spread throughout a system.
Are there any headlines of numbers that make you wonder?

No comments: