Paper: Surgical Safety Checklist - more than checking a box

Fabulous - the authors have published a study where the results show there's a problem. I've been harping on about the lack of negative studies for years. So well done to the authors.

Basically, they've done some follow up work to see how well used the surgical safety checklist is. No surprises to discover that is most cases only 4 of the 13 items were checked. They suggest there is a problem with the fidelity of implementation, and a poor dissemination strategy and implementation.

Fidelity really is key. There's no point in shouting from the rooftops that 100 hospitals are using a checklist, for example, if it is not being used as intended. The results will not be as good as the pilot study. The return on investment for the project will not be as good. And, who knows, in some programs, the results may actually be worse than if the program hadn't been disseminated.

Fidelity. Important.

Surgery. 2012 Jul 6. [Epub ahead of print]
Implementing a surgical checklist: More than checking a box.
Levy SM, Senter CE, Hawkins RB, Zhao JY, Doody K, Kao LS, Lally KP, Tsao K.

Book Review: Sustaining lean healthcare programme; a practical survival guide - Eaton & Phillip

Different book styles appeal differently, to different people. This one appealed to me because it is relatively short (94 pages plus appendices), well organised, uses bullet points and lists, checklists, diagrams and has some short examples to illustrate points.

Contents include:

1. Where are you on your journey
2. Why do only 24% succeed?
3. Going Lean
4. The top ten signs of a failing programme
5. Creating a lean healthcare organisation
6. Four key checklists
7. The next eight things to do...

So do only 25% succeed? The authors suggest there are 8 critical success categories Communications. resources, involvement, training, implementation, compass, achievement and leadership (yes, these do spell "critical"). If you like wordplay then read the book and discover PRISM, CAD, VSE, RIE, RPE, FIT, FMEA etc. Not as bad as it looks in a list here. I found them a good description, and sometimes reminder, of basic principles.

The chapter on the top 10 signs of a failing program is easily read. For each reason there is an explanation of what you may experience, why it happens and then what you can do about it.

The four checklists you need? One each on people, success, tools and culture. Nothing really new to me, just nicely organised.

I particularly liked the Appendices which included key lean phrases and concepts (worth the price of the book for its organisation and simplicity), audit form, guide to common saying (fabulous!), and then a number of "how to" guides.

I know Mark Eaton has actually implemented lean in healthcare. This book is a demonstration that when written by someone who has got their hands dirty it turns into valuable desktop reference.

Paper: Strategies for sustaining a quality improvement collaborative and its patient safety gains

Int J Qual Health Care. 2012 Jun 4. [Epub ahead of print]
Strategies for sustaining a quality improvement collaborative and its patient safety gains.Parand A, Benn J, Burnett S, Pinto A, Vincent C.

In their words, the conclusions are

"This study has presented what principle programme coordinators across 20 NHS organizations considered to be the key strategies to sustain their own improvement programme and its successes, during the supported phase of the programme and 1 year on. Recommendations are to consider these practical strategies in order to improve chances of maintaining changes and continuing a quality improvement programme beyond the formal cessation of the intervention."

I'm not sure what this means other than 'watch and wait' and 'learn' - which is fine.

Sustainabilty of QI results; we're not sure how to make it happen

I love systematic reviews. Someone else does the hard work of synthesising the literature and pointing out the strengths and weaknesses.

If you come across anyone purporting to tell you how you can sustain the results of your improvement work, then send them this paper.  Basically, we're not sure how sustainability happens other than it is a complex matter, and not one for a checklist or one day seminar. What I like about this systematic review is it points out the complexity, the system dynamics issues and the need for published work to identify the contextual factors in the sustainability (or not) or results.

Excellent work here by the authors.
Thank you.


Implement Sci. 2012 Mar 14;7(1):17. [Epub ahead of print]
The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research.
Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M.




Partial Abstract

"RESULTS:
Although "sustainability" was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves.
CONCLUSIONS:
Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment."

Sustainability - how about 52 years?

After 52 years, the Marshall and Fraser Racing MAY be letting down the tyres and packing away their tools. I say MAY, because, if I know them, they might just, possibly, pop out for a few more races next year...

Many of you will have met this team virtually as I have used them as an example of sustainability. I'll leave you with the question I always leave an audience - how will you sustain the results of your work for 50+ years?  I mean results of course. The car looks very different than it did 50 years ago, but their focus is on the outcome.

Congratulations to the team, who, 52 years later, are still on top.

Class Winners of the Championship in 2011.

New Paper: Short and long term effects of a QI collaborative on diabetes mgt

I'm always on the look out for evidence of the sustainability of QI projects. A new paper in Implementation Science has some thoughts on this though I am not convinced 1 year classes as "long term".

Implement Sci. 2010 Nov 28;5(1):94. [Epub ahead of print]

Short- and long-term effects of a quality improvement collaborative on diabetes management.

Schouten LM, Hulscher ME, Van Everdingen JJ, Huijsman R, Niessen LW, Grol RP.

ABSTRACT:

INTRODUCTION: This study examined the short- and long-term effects of a quality improvement collaborative on patient outcomes, professional performance, and structural aspects of chronic care management of type 2 diabetes in an integrated care setting.

CONCLUSIONS: At a time of heightened national attention toward diabetes care, our results demonstrate a modest benefit of participation in a multi-institutional quality improvement collaborative focusing on integrated, patient-centered care. The effects persisted for at least 12 months after the intervention was completed. 

Improvement Projects: Do no harm

A common issue raised by project managers who are trying to implement existing good practice with individuals and teams is one of resistance to change. I am constantly seeking ways to reframe the term "resistance" as a means of moving away from a potentially obstructive and destructive frame of reference.

I've been wondering whether one of the reasons people appear to "resist" adopting even what is well evidenced as good practice is because of a natural and at times perfectly reasonable conservative attitude towards risk. The medical profession has the theme of "do no harm". My feeling is often we are asking professionals to take on the solutions designed by others and in different contexts without providing the potential adopters with the evidence that the results are both relaible and generalisable. Reliable in the sense they can be repeated int he same context with the same results. Generalisability is what is proved when the intervention (improvement process) can be done in a different context and obtain similar results.

Without this evidence of generalisability in our improvement work I feel professionals will continue to be suspicious of changes.

In additon, do we ever publish the knock on consequences and the adverse effects of improvement work? A quick trawl of improvement projects published in high impact journals in the last 2 months demonstrates the attitude that improvement work is all good. None fo 12 papers that I looked at provided (or even hinted) at any negative consequences. Without honesty abotu improevemnt work and results I suspect we will continue to encounter "resistance" to change - and I will consider this an appropriate response to any solution being touted for implementation where there is no demonstartion of generalisability and no discussion about identified adverse consequences.

Validity, reliability & generalisability of project results; the science of improvement?

I am aware that the use (and sometimes invention) of management gobbledygook words to describe actions and intentions regarding quality improvement may sometimes be more of a hindrance than a help. After an inspirational week spent with the Veterans Affairs Quality Scholars in Vermont I got thinking more about this.

"Spread" and "Sustainability" are my two pet problems as words. Spread is difficult to describe and many people use it in different ways and for different dynamics, thus creating more confusion. I have worried for a long time that there is no such issue as sustainability in QI if we are doing proper continuous improvement.

So I wonder what reframing and mindset shifts we get when we use words from the science discipline - after all, many call what we do the "science of improvement".

Validity

Generally in science validity measure the extent to which the test. experiment or method actually does what it has been designed to do.

I wonder how many improvement projects get "good results" yet the overall aim is not identifiably or actually reached. For example, a project designed to length of stay in hospital may use average LOS to measure. Over 6 months this may show an increase, despite much work. As the average includes the denominator of beddays, what may have happened is other work on prevention and lowering readmissions rates has decreased the total number of beddays. Apart from average not being a good improvement measure, if is possible the actions taken for LOS were not focused on the overall aim - which could have been to reduce cost, change the experience for the patient etc.

In statistical terms this questions whether the sample used exhibits the characteristics of the population.

This may be one of the reasons why spread does not happen. We choose populations outside the norm (people willing to change, where the context is prepared, give them help etc) and then when they get good results we require the "norm" to copy them. In many cases the results, the change process, the toolkit produced is designed for a very small population and has poor validity across the wider intended adopting group.

Reliability

Statistically this is the amount of credence placed in a result; the precision of the measurement as repeated over a specific period of time.

For improvement projects that use control charts reliability will show as the extent to which the process is controlled. On a more macro level, reliability of a measure is weakened when the measurement method changes over time or when the measurement is open to "gaming".

The most popular measures for gaming (and I think lacking in reliability) are ones like 95% of patients to wait no longer than 4 hours.

Reliability can also mean the probability that a measurement (or intervention) will perform its intended function over time and within a given set of conditions

This definition reminds me of the talk about "sustainability". If results drop off or the way of measuring becomes "unsustainable" - usually due to other changes in the system, then the problem may be more one of design than a loss of momentum (or whatever way you conceive of "sustainability").

Generalisability

To draw specific inferences; to make generally or universally acceptable

This is about demonstrating that the improvement work carried out in Ward 10 is applicable throughout the hospital. To what extent can the other 15 wards copy what has been done and get the same result? In my experience we often end up with say half the other wards adopting something and of those half not all of them get the same results as the originating ward (they may in fact do better). This is about spread, and to effect generalisability the originating project needs to be able to describe their contextual factors and anything that may be contributing to their results. Without this, adopting teams, and management who would like the work adopted, would have little knowledge as to the generalisability of the work.

I also see a lot of "the results from hospital A was a £10,000 saving. This means all ten of our hospitals can achieve a total of £100,000 if they did the same." I suggest calculation is meaningless without a demonstration of the probability of generalisability.

The SQUIRE Guidelines have been developed to help overcome some of the lack of rigour in publishing improvement work. In particular they address the contextual / generalisability issue.

Large scale change or spreading good practice?

As a Rotarian I have recently been involved in a project to help the Masai Mara community in Tanzania get clean water near their homes. We're working with an inspirational fledgling charity called Weston Turville Wells for Tanzania. They have already enabled the installation of wells, are looking for more as well as more ambitious projects.

My link to this work has helped me understand more about some of the differences between "spreading good practice" and "large scale transformation". Well, it has sort of helped - it also raises more questions for me...

Think of community of 4,000 people which is made up of smaller communities (think small town and suburbs). It costs around £3,000 to install a well that will support a small sub-set of a suburb. One way to help the whole 4,000 community is to take the "good practice/good idea" of the single well and replicate this across the community, say building 15 or 20 wells. Twenty wells will cost around £60,000. There is no doubt that this will transform the lives of the whole community.

Another option, is to spend £60,000 and build a pipeline from a source of water in a mountainous area not far away. The impact of doing this is also a large scale transformation for the community. It is ambitious in scale, impact and ingenuity is required to implement a reliable solution.

So which would be your choice? Option 2 sounds great and at first glance looks sustainable. However, it will take many months, possibly a year or so, to raise the funds, get the permissions and then build. During this time the community is without a viable water supply within any close distance of their homes. Option 1 feels suspiciously like a quick fix with the added difficult dynamic of the spread of the wells not feeling (my feeling) democratically spread. However, money can be found "per well" and work can start in a very short time. The individual well is a solution is tried and tested. In addition the charity has ensured there is local support from within the community to help maintain them.

I can see that both options are transformative for the community. The outcome will make a difference to daily life and health. The process may also be transformative and everyone gets involved with the well / pipeline building process. Option 1 is more emergent, flexible and bottom up type approach. Option 2 is more planned, organised and top down approach. The both/and would be to have a pipeline and some wells - but then I'm torn between the extra cost of doing this where more wells can be provided for another community...

For the record, we've gone for supporting the provision of a well - right now. The essence being that in some cases a quick fix is a good one and when it comes to water, every day without it is a trial.


What would you do?

Organisation-wide spread plans need vision

With most groups I work spread seems, roughly, to fall into two categories: (a) spreading good practice across large geographic communities, involving multiple organisations, and (b) spread within a single organisation, managed by a single top team.

In the case of the single organisation spread effort, the aim is twofold (i) breath - getting the good practice to the further reaches and corners of the whole organisation, and (ii) depth - helping all the relevant individuals in the organisation through the change process of implementing the new practices.

Most organisations I've been working with recently have been struggling with developing the plans to achieve the breadth and depth they desire. In most cases they are working through what appears to me to be a technical, step by step planning project type mechanism. I feel this is helpful though for me there is a very important missing ingredient which might be best handled right at the start and before detailed plans are written up. Perhaps this is best explained through an analogy.

Imagine weaving a large tapestry; one of those you might have seen hanging on the wall in a museum. Large creations like these are usually participative activities. Now think what it would be like to be stitching part of the tapestry without knowing what the whole picture looked like. Yes, it would be helpful to know your own piece, but far more satisfying to know where you fit into the whole. Imagine now the activity you could undertake as a team in thinking through what your organisation will look and feel like when the whole organisation has adopted the new practices. And then how you might share this picture. This visioning and sharing process is important and there is an entire discipline dedicated to what is called "goal orientated behaviour" - in case you are interested in following this up.

Keeping with the tapestry image, we can extend it a bit further by thinking about the structures we need to put in place for a large and complex activity. How will we watch for the gaps? By answering this sort of question we might develop a different type of spread measurement system for the organisation; a measurement-by-exception attitude rather than a wearing down by a comprehensive performance monitoring system.

And the threads? How do we feel about them? Some may need to be left hanging as they will link to other projects and issues? How will we identify them? Can we "colour-code" them? How can we make sure that they are not the type of threads that when you pull on them the whole tapestry won't just unravel and we're left with nothing to show for our efforts? For me, this is about being aware as a project team. About seeing the spread of the good practice as part of a larger system.

I know project plans are important, however, I am perturbed by the predominance of the step-by-step guides that disconnect individuals and teams from the heart fo their work. The quickie fixit mentality may be leading organisations away from the breadth and depth they so desire.

It takes time to weave a tapestry.

2008 Fraser Creative Commons Attribution-Non-Commercial-No derivative Works

Warning: "holding the gains" means missing the point

For those well versed in the art of Cockney rhyming slang, then you'll know that the "dog and bone" is how a Cockney Londoner refers to the phone and "apples and pairs" the stairs. They might now say "Would you "Adamn & Eve" (believe) it, but we seem to have a new term."
I keep coming across "spread and sustainability" as though these two remarkably different concepts and dynamics were intertwined. They appear in requests for short presentations, in documents, booklets, advice to project leaders and exhortations for improving results in healthcare.

The quickie defintion usually given after "spread and sustainability" or "sustainability and spread" is "sustainability" means we need to hold our gains and then we need to "spread" the gains / results to others in the organisation.

This looks fine on paper. But let's think about this. What exactly is meant by "holding the gain"? In the majority of improvement projects the gain is valued and assessed against a measurement. For example, the reduction in rate of infections on the ward would be measuring an outcome which is helpful and can keep a team focused on the end result. Some measures focus on a process, such as the number of staff who have adopted the use of a checklist for a certain procedure. Here is my concern.

When it comes to working on what it is that needs to be "held" many teams will focus on the the "what's" they have created. This will be the checklists, the new procedures etc. While helpful, it takes a humble team to be able to keep focused on the outcome and continue to adapt their processes and behaviours to continue to maintain and perhaps improve on their results. In many cases this may mean they have to alter their original soultions (thus, of course, impacting what others may want to spread - but that is another issue for another day...).

How many leaders and improvement teams take the time to work out the qualitative aspects of their results? What really enabled them to deliver the improvement? While the attention is so often on the output, the "thing" that was created, by ignoring or avoiding reviewing the human, social and emotional behavioural processes involved in the development of it, they are at risk of losing what they have gained. I believe that it is in this tacit knowledge that the true gain lies. It is here that the learning resides. The checklist is merely the tangible, and temporary, manifestation of their experience.

The discipline of learning organisations and learning teams is well known, though sadly often disconnected from the conscious experience of many improvement project teams. When the concept of learning gains a hold, then we may hold some gains.

(c) 2008, Sarah Fraser

Charismatic leaders may hinder sustainability of improvement

Of course we know that the skill of the truly great leader is to leave behind a team and a succession plan that not only maintains the knowledge and practices that have generated their success so far, but also supports their ongoing learning and development.

Fine words. How often do you see that in practice where the person leading the team can be described as "charismatic"? I could blog for weeks on the definition of charisma at work, so have settled for now on it being a combination of personal traits that enable an individual to reach out to other's emotions in a way that taps their motivation to do things. Usually charming and persuasive these individuals get their own way amongst their systems, for better or worse. As far as public figures go, I'm thinking of people like Mother Teresa, Adolf Hitler, Winston Churchill, Martin Luther King etc.

So how does charisma affect the sustainability of results in improvement projects?

Here are some questions for you to think about when you next choose the person to lead a key project or programme in your organisation where the sustainability of results will be important.

Are you choosing the person because of their charisma? Maybe you feel the need to have someone with the energy and passion to start the project and to motivate others.

To what extent could the project become so identified with the individual that there is a risk that

• team members eventually become demotivated because they get less recognition / air time
• when (and it is most likely a when rather than an if) the leader moves on, the majority of the knowledge and PR moves on with them
• no-one is prepared to "tell the emperor he has no clothes", to provide feedback, thus diminishing the learning opportunities
• succession planning becomes impossible and the person is deemed irreplacable

So you're thinking, none of our project leaders are ever that charistmatic! Well, according to the theory, we are most likely to find many of these traits amongst the more innovative and early adopters of new practices (though not exclusively as some laggards can be highly and effectively persuasive...). It is likely that many pilot projects and early innovations are led by leaders who demonstrate the ability to capture the imaginations and hearts of their followers. These same individuals will later leave to follow other interests, for the similar reasons they got involved with the first pilot.

Take a close look around you. What patterns do you see linking the leaders of your improvement projects and the sustainability of the reults of those projects?

Now that's a research project worth doing. Anyone want to collaborate?

(c) 2008, Sarah Fraser