"How beneficiaries see complex health interventions: A practice review of the Most Significant Change in ten countries."

14 views
Skip to first unread message

rick davies

unread,
Feb 13, 2021, 6:07:17 AM2/13/21
to MostSignificantChange (MSC) email list
Tonkin, K., Silver, H., Pimentel, J., Chomat, A. M., Sarmiento, I., Belaid, L., Cockcroft, A., & Andersson, N. (2021). How beneficiaries see complex health interventions: A practice review of the Most Significant Change in ten countries. Archives of Public Health, 79. https://doi.org/10.1186/s13690-021-00536-0

Hi all

You can download a copy of this new article via the link above. I would be very interested in your reactions, both appreciative and constructively critical 

I have invited the corresponding author to join this email list

best wishes, rick davies

rick davies

unread,
Feb 15, 2021, 8:31:08 AM2/15/21
to Stephen Perry, mostsignificantchang...@googlegroups.com
Hi Stephen

Thanks for your email below, and especially your comment about the problem of picking up non-existent changes

To some extent, I think this problem arises through a lack of understanding of some of the foundational assumptions underlying design of MSC as a method.

The first assumption is that change is ever present, in all people's lives and at all periods of time. This is a philosophical position also espoused both by Greek and Indian philosophers many centuries ago (Heraclitus and Siddhattha Gotama - the Buddha),. 
So the first challenge in asking people about most significant changes is to get people to identify the changes that have taken place in a reference period of interest, compared to before then. It is categorically not possible that nothing has changed. However it is possible that people may not have noticed changes, at least on first glance. My experience suggests that with sufficient interview skills people can subsequently identify changes that have happened. The challenge is to enable people to see changes that have happened.

The second assumption is that significant is not an absolute and categorical judgement, based on some universal or mandated criteria.  Significance is always a relative judgement, made by comparing different things i.e. different changes, and a judgement that is very much in the eyes of the particular person making that judgement. So, after we have enabled/encouraged a person to identify various changes that have taken place in a reference period we need to ask them which of those changes is the most significant, when compared to all the others they have noticed during this time.  

So, for example, what was the most significant change that I observed in my life yesterday, while living at home here in Cambridge? And first glance, it was like many other days during the Covid lockdown here in the UK. I sat at my desk for long periods of time, reading and writing, I had breakfast and lunch and dinner with my partner, we watched television, and my partner and I went for a walk. On reflection, it was the walk that was the most significant change. I had been on walks in previous days, but yesterday we took a different route alongside the Cam river, where the floodwaters had recently receded and unlike in previous days the flood meadows were now iced over, and almost skate-able on.  Bringing back memories of my own love of ice-skating decades ago, and thoughts of using the new ice-skating rink in Cambridge sometime in the coming year, touch wood, if Covid and my ageing bones permit!

Now, bringing the discussion back to the possibility of someone reporting that there had been no significant change arising from their encounter with a development project intervention. My first inclination would be not to ask specifically about changes arising from a development intervention, but more broadly about changes in their life in a period of time corresponding with when the development intervention took place. I would be very interested to see the extent to which the changes were reported related to the development intervention versus other unrelated events in their lives. Whatever is happening with the development intervention, its significance will always be relative to events happening in the wider context of people's lives.

If you do want to narrow the focus down to events relating to the development intervention alone then I think the task of seeing any changes  could be more difficult where there has been a complete cessation of all activities over two or more periods of time.  How common would that be I wonder? Even where this did happen I would be very surprised if people stopped thinking and talking about the development intervention during these periods, so those thoughts and reactions could then be the focus of the interviews attention.

To pursue this issue of non-existent changes further, it might be useful if anyone on this email list could bring up a concrete example of where people have responded by saying 'but there hasn't been any change '.

Regards, Rick



or


On Mon, Feb 15, 2021 at 9:05 AM Stephen Perry <mars...@marsteve.net> wrote:

Hi Rick,

 

Thanks for sharing.

 

We spoke some time back about my OECD-linked development statistics data project thinking about the use of MSC in their outcome monitoring framework. They were quiet for a few months, but have come back to me asking about how it would practically look. This is a good sign.

 

On the paper, what a great initiative! My particular take-away, given the project I’m working on, is that MSC does not capture when change does not happen. This is probably the vast majority of cases.

 

Keep well,

Steve

--
If you have any concerns about any of the postings on this email list please email me directly at rick....@gmail.com
---
You received this message because you are subscribed to the Google Groups "MostSignificantChange (MSC) email list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mostsignificantchange-msc-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mostsignificantchange-msc-2020-email-list/36a05b02-f223-4c22-a361-bc6ecd455c67n%40googlegroups.com.

Andrews N.D.

unread,
Feb 15, 2021, 11:24:07 AM2/15/21
to rick davies, Stephen Perry, mostsignificantchang...@googlegroups.com
I hope it is OK to share a short snippet from MSC work in Wales, which may relate to Stephen's comment that 'MSC does not capture when change does not happen' 

Some time ago, I was involved in an event that was focused on an evaluation of co-production in Wales. One of the participants was an older guy, who when asked what good or bad changes had come about as a result of his involvement in co-production said: 'Nothing has change... absolutely nothing'... something he repeated vehemently when asked. However, it was apparent that something had changed, and that was his attitude towards co-production. Once he saw this, he was able to share a very powerful and useful MSC story. I have found that people sometimes need help thinking about what we mean by change even before they consider which is most significant. 

Regards, Nick


From: mostsignificantchang...@googlegroups.com <mostsignificantchang...@googlegroups.com> on behalf of rick davies <rick....@gmail.com>
Sent: 15 February 2021 13:21
To: Stephen Perry <mars...@marsteve.net>; mostsignificantchang...@googlegroups.com <mostsignificantchang...@googlegroups.com>
Subject: Re: [MostSignificantChange] "How beneficiaries see complex health interventions: A practice review of the Most Significant Change in ten countries."
 

Fiona Kotvojs

unread,
Feb 16, 2021, 3:57:50 AM2/16/21
to Andrews N.D., rick davies, Stephen Perry, mostsignificantchang...@googlegroups.com

Hi Stephen,

 

My experience is that MSC is able to identify real change well – both expected and unexpected. This is one of the reasons I use it so much, particularly where I believe the unexpected (and unplanned) changes will be important.

 

There are times when development interventions don’t achieve change for specific stakeholders. I believe that is OK and personally, I don’t want an evaluation tool to indicate changes that did not occur. So when MSC does not identify a change, I don’t have a problem with that provided:

  • I have asked what changes occurred and really pushed to identify changes (I.e always get a list of changes and then have the person select the MSC)
  • I am not getting the same answer from everyone (which would probably indicate a problem with my questions, interview technique (including cross-cultural issues) or sample)
  • There is nothing from other evaluation methods (including document review) to suggest otherwise.

 

Because, when done properly, MSC is harder for respondents to ‘fake’ answers (i.e give you the answer they think you want to hear), I also believe it is a good tool when you want to ensure this doesn’t happen. Another reason I use it a lot.

 

In my case, where an application of MSC has shown little change or only for a few, this has been important for the management of the intervention. It indicated that something wasn’t working as planned and needed to be changed. In particular, it often showed that specific stakeholder groups were not being well targeted. Change in delivery, change in MSC findings. That is a good outcome of an evaluation.

 

The other thing to remember is that MSC is a form of extreme case analysis – unless you analyse all the changes identified (which I do as part of a secondary analysis), MSC doesn’t tell you about averages. It tells you about extremes – hence the name – “MOST”. So if you only want to know about averages, other methods may be better. Almost every evaluation I undertake includes a component for management to learn what works and what doesn’t. This is usually done through the extremes, another reason I use MSC a lot.

 

Hope this helps.

 

Fiona

 

 

Dr Fiona Kotvojs GAICD

Design, Monitoring and Evaluation Specialist

Kurrajong Hill Pty Ltd

 

fi...@kurrajonghill.com.au

Phone: 0448 453 422

Reply all
Reply to author
Forward
0 new messages