Paper: Realist randomised controlled trials: A new approach to evaluating complex public health interventions

Realist reviews of complex healthcare interventions are all the rage - and justifiably so. The realist process is far better as discovering and questioning the complex interactions within health and social care than more linear methods. Specifically, it investigates the how and why of interventions, rather than a singular focus on the what.  It's the way forward.

Realist randomised controlled trials: A new approach to evaluating complex public health interventions.
Bonnel, Fletcher, MOrton, Lorenc, Moore.

Soc Sci Med. 2012 Dec;75(12):2299-306. 

Book Review: The ABCs of Evaluation - Boulmetis & Dutwin

I've rediscovered another older book (2005) on the bookshelf. I was browsing round the home library searching for inspiration in developing an evaluation for a leadership development program. I remembered what I liked about this book the first time round.


The authors have produced a book which is set out logically. It covers the choices you have and steps to take to carry out an evaluation. I found the section describing different evaluation models and which one to choose was just what I was looking for:
1. Discrepancy Model: use when the program is so interwoven within the organisational context it is difficult to discover freestanding changes
2. The Goal Free model focuses on the participants needs and is independent of the organisation's needs.
3. The transactional model is similar to the goal free model with the exception that the evaluator plays a more active role in the evaluation process.
4. The decision makingg model focuses less on the success of the program and more on the longer term sustainability
5. The most popular model is the goal-based where the outcomes are assessed against the organisation's objectives (or program objectives)
The authors do also list other evaluation models and I would be interested in using systems analysis with my healthcare programs. The version using art criticism where you bring in an expert in the field to review progress is an interesting one I had never thought of. The adversary model where participants present their views and a "jury" hears their evidence might be useful for the rapid assessments we sometimes need to make.

The evaluator's role is covered as a continuum from the start of the program through to the end as well as the level of active participation in the evaluation process.

I was interested in what an evaluation is supposed to measure. Having got used to most healthcare evaluations answering "has the program delivered its objectives" it was good to see that "was the program efficient in terms of cost and staff time - was the expense worthwhile?". What we often mean when we say sustainability, the authors use impact. I like this as it covers not just the sustainability of the results but also the magnitude of the results as contributing to the organisation's long term objectives.

The book also includes the types of data, how to present it and the process of writing up your report.

I'd recommend this book if you're starting out in evaluation or need a refresher. If you're a seasoned program evaluator I suspect you will find other books go to more depth.

5 Program Evaluation Resources

Here is a list of resources you may find useful if you're evaluating a large program.

1. A basic guide to program evaluation has lots of text but it organised into useful headings. Not too long and is good if you're looking for the rationale for carrying out an evaluation.
2. The Evaluation Cookbook is a splendid resource - apart from having a great title. Comprehensive yet not overwhelming. My favourite resource for evaluation.
3. The Evaluation Handbook from the Kellogg Foundation is a meaty text aimed at program managers on their funded programs - though its useful for similar, large scale programs.
4. A User Friendly Handbook for Evaluations is good if you have a scientific bent to your evaluation process.
5. Taking Stock: A Practical Guide to Evaluating your Own Programs is useful if your focus is on carrying out internal evaluations to assess your project work.