Focusing on High Impact changes in the NHS may be damaging. Try MIME instead.

Identifying the "top 10" or the "top 5" high impact changes that NHS organisations can implement - then pushing them to do so, has consequences.

1. Unless the identified high impact change is shown to be generalisable (that is it has been tested in different contexts and a similar result has been achieved), then there is a significant probability that the change may not be high impact at all.  When we take one result from one place, then roll up the possible benefits across all organisations we are making a fundamental mathematical, and change process, error.   If we do want to do large scale mathematics then we need to know the baseline at each potential organisation, the match in context to the originating result, and then do a weighted calculation across the system.

2. Not all high impact changes are equal - in the amount of effort and resource (read ££££) they take to implement. A great result may sound good, but if it takes so much resource to implement that the payback time is 10, 15 or even 20 years, then yes it is high impact - but not in the way intended.  The challenge is to find a way to get the impact - but with less cost associated in the process of doing so.


MIME = Maximum Impact, Minimum Effort.


(and of course, LIME, low impact, maximum effort, should always be avoided)

Do your "high impact" exhortations meet the MIME challenge?

How safe are clinical systems - report on the evidence by The Health Foundation

This report for The Health Foundation is excellent. I like the way it is based on research and whilst it doesn't cover any specifically new ground, it does provide insight for anyone wishing to improve the safety of patients in hospital. 

"While the knowledge that poor systems can cause harm is not new, this report provides groundbreaking evidence of the extent to which important clinical systems and processes fail, and the potential these failings have to harm patients.

The results of this study, covering seven NHS organisations, identify the variation in the reliability of five key healthcare systems and processes:

• availability of information when making clinical decisions
• prescribing
• handover
• availability of equipment in operating theatres
• availability of equipment for inserting intravenous lines.

The research, led by Professor Bryony Dean-Franklin, was conducted by The Centre for Patient Safety and Service Quality (CPSSQ) at Imperial College, and Warwick Clinical Systems Improvement (CSI), University of Warwick."

Helping leaders understand variation

What does your leader do when you put a control chart or run chart in front of them? There is an excellent paper available from the Institute for Healthcare Improvement on the topic of helping leaders understand variation.

Lloyd R. Helping leaders blink correctly: Split-second decisions have patient safety implications (Part 1). Healthcare Executive. 2010 May/June;25(3):88-91.

This article describes two of four necessary skills health care leaders need to develop in order to "blink" appropriately (i.e., make decisions based on robust analysis and interpretation of data): understanding the messiness of improving health care, and determining why you are measuring.

Download part 1 here

Download part 2 here

Productivity 7: How replicable is productivity?

This is the seventh in a series of productivity notes by Sarah Fraser.  If one organisation or region is classified as productive, can we make generalised statements that the rest of the organisation or nation can implement the same and be as productive?

A new report by York University is making big claims.

"The NHS could cut expenditure by £3.2billion without reducing the number of patients treated if all parts of the country were as productive as the South West, according to a report published today by the Centre for Health Economics at the University of York."

The report has many maybe's and possibly's as to whether the rest of the NHS could see the same cost savings if they performed like the South West Region.  I have no doubt the SW is producing excellent care. My concern is headlines like this paper produces sets unrealistic expectations on others. The delivery of healthcare is significantly contextual in its nature. Services all over the country reflect the complex make up of the areas they serve. The report summary on their website states:

"South West may also benefit from a more stable workforce, vacancy rates for non-medical staff being well below the national average. Lower productivity in the hospital and community sectors may be because more work is undertaken in primary care."

I believe this is enough uncertainty to warrant being very cautious about ratcheting up national numbers. Additionally, there is significant use of the "average" in this report. I am not convinced that averaging data and using the average as a measure is a good one for healthcare. As I pointed out in my earlier Productivity note, there is a big difference between accuracy and precision; basically, it is possible for there to be little variation across the regions (precision) but they are all delivering the wrong solution (accuracy).

Like all theories, this research is helpful to a degree (mostly in applying judgement) and like all theories, needs to be treated with a pinch of salt. If you are going to quote the headline £32billion on the stage then make sure you've read and understand the limitations of the report.

(To those who read the previous Productivity note about definitions - productivity is defined in this research report as output / input - how much output you get for the inputs...)

Productivity 2: Reduced variation is not enough

This is the second in a series of productivity notes. Reducing variation is only part of the productivity process.

Reorganising processes so they are precise and prediction is helpful but not enough. In the diagram, the red dots are hitting the target in a predictable way. However, while they may be precise, they are not accurate. The green dots, with less precision, are more accurately placed around the bulls eye.

What I learn from this is the need to

• know the definition and position of the bullseye (what is the purpose of the process being improved)
• measure for accuracy as well as measuring variation
• fix accuracy first, then go for reduced variation

Improvement Projects: Do no harm

A common issue raised by project managers who are trying to implement existing good practice with individuals and teams is one of resistance to change. I am constantly seeking ways to reframe the term "resistance" as a means of moving away from a potentially obstructive and destructive frame of reference.

I've been wondering whether one of the reasons people appear to "resist" adopting even what is well evidenced as good practice is because of a natural and at times perfectly reasonable conservative attitude towards risk. The medical profession has the theme of "do no harm". My feeling is often we are asking professionals to take on the solutions designed by others and in different contexts without providing the potential adopters with the evidence that the results are both relaible and generalisable. Reliable in the sense they can be repeated int he same context with the same results. Generalisability is what is proved when the intervention (improvement process) can be done in a different context and obtain similar results.

Without this evidence of generalisability in our improvement work I feel professionals will continue to be suspicious of changes.

In additon, do we ever publish the knock on consequences and the adverse effects of improvement work? A quick trawl of improvement projects published in high impact journals in the last 2 months demonstrates the attitude that improvement work is all good. None fo 12 papers that I looked at provided (or even hinted) at any negative consequences. Without honesty abotu improevemnt work and results I suspect we will continue to encounter "resistance" to change - and I will consider this an appropriate response to any solution being touted for implementation where there is no demonstartion of generalisability and no discussion about identified adverse consequences.

Validity, reliability & generalisability of project results; the science of improvement?

I am aware that the use (and sometimes invention) of management gobbledygook words to describe actions and intentions regarding quality improvement may sometimes be more of a hindrance than a help. After an inspirational week spent with the Veterans Affairs Quality Scholars in Vermont I got thinking more about this.

"Spread" and "Sustainability" are my two pet problems as words. Spread is difficult to describe and many people use it in different ways and for different dynamics, thus creating more confusion. I have worried for a long time that there is no such issue as sustainability in QI if we are doing proper continuous improvement.

So I wonder what reframing and mindset shifts we get when we use words from the science discipline - after all, many call what we do the "science of improvement".

Validity

Generally in science validity measure the extent to which the test. experiment or method actually does what it has been designed to do.

I wonder how many improvement projects get "good results" yet the overall aim is not identifiably or actually reached. For example, a project designed to length of stay in hospital may use average LOS to measure. Over 6 months this may show an increase, despite much work. As the average includes the denominator of beddays, what may have happened is other work on prevention and lowering readmissions rates has decreased the total number of beddays. Apart from average not being a good improvement measure, if is possible the actions taken for LOS were not focused on the overall aim - which could have been to reduce cost, change the experience for the patient etc.

In statistical terms this questions whether the sample used exhibits the characteristics of the population.

This may be one of the reasons why spread does not happen. We choose populations outside the norm (people willing to change, where the context is prepared, give them help etc) and then when they get good results we require the "norm" to copy them. In many cases the results, the change process, the toolkit produced is designed for a very small population and has poor validity across the wider intended adopting group.

Reliability

Statistically this is the amount of credence placed in a result; the precision of the measurement as repeated over a specific period of time.

For improvement projects that use control charts reliability will show as the extent to which the process is controlled. On a more macro level, reliability of a measure is weakened when the measurement method changes over time or when the measurement is open to "gaming".

The most popular measures for gaming (and I think lacking in reliability) are ones like 95% of patients to wait no longer than 4 hours.

Reliability can also mean the probability that a measurement (or intervention) will perform its intended function over time and within a given set of conditions

This definition reminds me of the talk about "sustainability". If results drop off or the way of measuring becomes "unsustainable" - usually due to other changes in the system, then the problem may be more one of design than a loss of momentum (or whatever way you conceive of "sustainability").

Generalisability

To draw specific inferences; to make generally or universally acceptable

This is about demonstrating that the improvement work carried out in Ward 10 is applicable throughout the hospital. To what extent can the other 15 wards copy what has been done and get the same result? In my experience we often end up with say half the other wards adopting something and of those half not all of them get the same results as the originating ward (they may in fact do better). This is about spread, and to effect generalisability the originating project needs to be able to describe their contextual factors and anything that may be contributing to their results. Without this, adopting teams, and management who would like the work adopted, would have little knowledge as to the generalisability of the work.

I also see a lot of "the results from hospital A was a £10,000 saving. This means all ten of our hospitals can achieve a total of £100,000 if they did the same." I suggest calculation is meaningless without a demonstration of the probability of generalisability.

The SQUIRE Guidelines have been developed to help overcome some of the lack of rigour in publishing improvement work. In particular they address the contextual / generalisability issue.