Episode #94: Measuring What Matters and Operationalizing Outcome: A Conversation with Sarah J. Wallace

Aphasia Access Conversations

Welcome to this Aphasia Access Aphasia Conversations Podcast. My name is Janet Patterson. I am a Research Speech-Language Pathologist at the VA Northern California Healthcare System in Martinez, California, and a member of the Aphasia Access Podcast Working Group. Aphasia Access strives to provide members with information, inspiration, and ideas that support their efforts in engaging with persons with aphasia and their families through a variety of educational materials and resources. I am the host for today’s episode that will feature Dr. Sarah J. Wallace from Queensland, Australia. These Show Notes accompany the conversation with Dr. Wallace but are not a verbatim transcript.

In today’s episode you will hear about:

• clinical meaningfulness and research wastage: defining and addressing,
• minimal important change: defining and measuring,
• four “Monday Morning Practices” to create clinically meaningful outcomes.

Dr. Janet Patterson: Welcome to our listeners. Today I am delighted to be speaking with Dr. Sarah J. Wallace from the University of Queensland. In this episode we will be discussing the topic of operationalizing treatment success: what it means, the research efforts supporting this idea, why it is important to think about as we plan and deliver aphasia treatment, and suggestions for implementation in daily clinical practice.

Dr. Wallace is an NHMRC Emerging Leadership Fellow, NHMRC Senior Research Fellow in the School of Health and Rehabilitation Sciences at the University of Queensland in Australia. She is also a Certified and practicing Speech Pathologist. Her research interests include communication disability in ageing and enabling and measuring meaningful change in language and communication impairment in individuals with post-stroke aphasia. She uses qualitative and mixed methods to explore the lived experience of communication disability and works in partnership with consumers and clinicians to co-produce clinical interventions and methodological approaches that support the production of meaningful outcomes.

Among her interests in aphasia assessment and rehabilitation is a focus on measurement of aphasia and rehabilitation outcomes, in particular, outcomes that are real and are meaningful to persons with aphasia. Sarah led the ROMA group, Research Outcome Measurement in Aphasia, a group that has published three papers reporting efforts to identify standard outcome measures used in aphasia research. In addition, with colleagues across the world, she published a paper examining methods of operationalizing success in aphasia treatment in research and daily clinical practice. Foremost in this body of work is what I perceive to be Sarah’s desire to bring together ideas from persons with aphasia and their family members, assist clinicians and researchers to identify effective and efficient rehabilitation techniques, and to measure treatment outcome in a relevant and scholarly rigorous manner.

Welcome to Aphasia Access Conversations, Sarah, and thank you for joining me today.

Dr. Sarah Wallace: Thanks, Janet, for this invitation.

I would like to start today by acknowledging the traditional owners of the lands from which I'm joining today, the Turrbal and Yuggera people, and pay my respects to their ancestors and their descendants who continue cultural and spiritual connections to country.

Janet: Thank you very much. I appreciate that acknowledgement.

Sarah, throughout your career, you have published papers focusing on aspects of aphasia rehabilitation, many of which explore the topic of measuring and standardizing outcomes in aphasia rehabilitation. How did you become interested in exploring this aspect of aphasia?

Sarah: Before I completed my PhD, I worked first clinically, as a speech pathologist, and later in a government policy role in the area of aged care quality and safety. I really loved both of these roles for different reasons. As a clinician, I could make a difference at an individual level. But with the government role, I realized the huge impact you can have when you're influencing practice from a systems level. So, when I went on to complete my Ph.D., I really knew that I wanted to do something big picture. At the time, there had been a few big studies coming out with no results. There was a lot of talk about how important it is to get research design right. Then as part of my work at the time, I was reading the World Health Organization, World Report on Disability, and that's where I really started learning about this concept of research wastage and the importance of having a really considered approach to the way we measure outcomes when you want to use data efficiently beyond an individual study. That really appealed to me, particularly given that, within aphasia, we tend to have small sample sizes and really need to make the most of the data that we collect.

Janet: Sarah, we often hear the term clinically meaningful in relation to aphasia outcomes. How would you define that term from the perspective of a person with aphasia? And also, from the perspective of aphasia clinicians and researchers?

Sarah: This is an excellent question. This is something that I was really interested in during my Ph.D. It's this idea of what is a meaningful outcome. And who actually gets to decide that? And are we measuring what matters to the people who live with aphasia, and the clinicians who work with them? I remember reading at the time, and one of my favorite quotes is from a paper by a researcher called Andrew Long. He says, in practice what actually gets measured depends on who wants the data, and for what purpose. I really think that the idea of clinically meaningful depends on who you're asking, and why you're asking. As an example, in the studies that we conducted with people with aphasia and their family members, they thought improved communication was really important. But they also identified a range of outcomes that related to participation, to attitudes, to psychosocial well-being. But then things change when you look at a different stakeholder group. We also spoke to clinicians and managers around the world, and they identified a range of outcomes. But the really interesting part was that improved language itself wasn't actually considered essential. The top outcome that they came up with actually related to family members, that they understand how to communicate with the person with aphasia. I think what it comes down to is the message that I've really tried to share from my research is that different outcomes matter to different people. And we can measure them in so many different ways. And that this is something that we really have to think carefully about.

Janet: Listening to your responses to these first two questions, I can feel the energy! I can feel this passion looking at aphasia rehabilitation from a larger perspective, outside the actual treatment that gets delivered, and thinking about how we make sure that our treatment is the right thing, and is measuring the right thing, whatever, as you say, the right thing is. It depends on who's looking for the data. You've maintained that focus of how can we become a better entity, better clinicians, if you will, at the broader scope? Does that make sense to you?

Sarah: Yeah, it does, and that idea really resonates with me. I think that's definitely been a feature of the work I've done and the work that I continue to do. It's very focused on collaborative efforts and how we can make the most of what we have, so that we can ultimately improve outcomes for people with aphasia.

Janet: I do think we need to pay attention to this. We cannot just assume that if we give a test pre and post treatment, it is a meaningful outcome to a person with aphasia or to their care partners or to a third-party payer.

Sarah, you have led the ROMA group, that is Research Outcome Measurement in Aphasia. As I mentioned earlier that group published three papers describing standardized assessment measures suggested for use in aphasia rehabilitation outcome studies. Would you briefly describe the genesis of the idea for this work and the studies the group has published?

Sarah: Following on from what I mentioned earlier, this was during my Ph.D. Once I had this idea that I wanted to do something to help reduce research wastage in aphasia, I started reading more about approaches to standardizing outcome measurement and came across the work of the Comet Initiative, which is a group that brings together people who are interested in the development of standardized sets of outcomes, which they refer to as Core Outcome Sets. There's this idea that a Core Outcome Set is essentially the minimum outcomes that should be measured in treatment studies of a particular condition. And that really appealed to me. So, we went from there, we conducted a series of studies looking at different stakeholders, gathering thoughts and perspectives about what an important outcome actually is. We conducted a scoping review of outcome measurement instruments so that we could try and match those outcomes to available tools. And then we've had a number of consensus meetings, where we've tried to pair those two things together.

Janet: I think the work of the ROMA group is important, and being part of that group, it's exciting to watch the minds of people all over the world, contribute their various perspectives, and have discussions about the different measures and the value of the measures. While I think it's wonderful to work at this level, this broad level of perspective, at some point, it has to inform our daily clinical practice. How do you see that happening?

Sarah: Yeah, that's a really good question. Essentially, we conduct treatment research so that we can help clinicians and people with aphasia and their families to make informed decisions about treatments. What's going to help? What's the best treatment for a particular issue and for a particular person? To answer these questions, researchers need to measure the effects that a treatment has on a person, what we refer to as outcomes. When we're measuring different outcomes in different ways it makes it harder to compare data, to combine it across studies, and to draw strong conclusions about which treatments work best.

Core outcomes also need to be relevant, and this is the other part that has been really exciting to me. They should capture results that are important to people who live with that condition. Ultimately, I think that the clinical relevance of the ROMA Core Outcome Set lies in what it is hopefully doing - helping to produce the best evidence that we can get for aphasia treatments, so that those treatments can then be implemented into practice in order to improve the lives of people with aphasia and their families.

Janet: I think that those papers should be required reading for every speech-language pathologist dealing with people with aphasia, and also other rehabilitation professionals, because it helps if we can all be thinking in the same way, as you said, to think about treatment candidacy and does one treatment work better, or for a specific person. or someone with a particular aphasia profile, than another kind of treatment? How do we make good clinical decisions for our patients? That's exactly, I think, what you're saying.

I mentioned also earlier that with several colleagues, you recently published a paper titled Operationalizing Treatment Success in Aphasia Rehabilitation. That paper was published in the journal, Aphasiology. I am a great fan of that paper and would like to begin by asking you why it would be important, in your mind, to operationalize treatment outcomes, given the variability that we see among aphasia patients.

Sarah: Thanks, Janet. And yeah, and this is a great paper. It was led by Caterina Breitenstein and other researchers from the Collaboration of Aphasia Trialists. This paper is really trying to answer the question, “What is a successful outcome from treatment?” What are the ways in which we can actually measure that treatment success? This is such an important question because research will end in clinical practice and so much hinges on this decision? Whether a treatment is successful is going to depend on how we define success and whether we can measure that success in a way that can actually be captured.

Janet: Sarah, in light of your thinking about the different stakeholders, how might operationalizing treatment success differ for the various stakeholder groups that you've identified? That is, people with aphasia, family members, clinical and other medical professionals, medical administrators, and aphasia researchers?

Sarah: This is really that idea that different outcomes are important to different people. If we think about this from a societal perspective, or from a healthcare funders perspective, any treatment that's provided as part of clinical care needs to be cost effective. So that might be something that from a funders point of view is a really important outcome. For clinicians, the ability for someone to take part in conversations and to communicate in different settings and roles is something that, through my research, was identified as an important treatment outcome. Then from the perspective of people with aphasia, not surprisingly, it's improved communication. But it's all these other things as well. It's being able to participate in a conversation. It's having a sense of recovered normality and a feeling of autonomy and independence. So again, I really think it's the idea that it really depends on who you're asking, and the perspective that they're coming from.

Janet: Your comments make me think about work done by Jackie Hinckley and others about stakeholders being part of deciding research questions or research directions. It also makes me think about work done by Michael Biel and others about motivation and engagement. All of these, I think, have a bearing on the research or the clinical enterprise. Are people engaged? Are they willing to commit time and resources to a rehabilitation enterprise because they see value in it, and because they see that there's a likelihood of a good outcome. I believe that what you're doing in terms of thinking about operationalizing helps move us along in that direction.

Sarah: Absolutely. I think that's a really important point, that if someone can't see the relevance of what they're working on in therapy, for example, then they're not going to engage in that process. It really starts with goal setting, and really identifying, working with a person to identify, goals which are really going to be functionally relevant to them and to their day-to-day life. I think if you can get that part right, then everything else follows on from that.

Janet: In your paper, you and your colleagues describe the concept of minimal important change, as a way of determining clinically relevant improvement on an outcome measure, considering the average statistically significant change across groups, as well as statistical significance at the individual level. Can you unpack that concept for us and describe how it relates to daily clinical practice?

Sarah: Yeah, absolutely. This is a really exciting idea, I think. Basically, minimal important change, and it is called different things, but this is the term that we've chosen to use, is the smallest change score above which an outcome is experienced by someone as being relevant or meaningful. I really love this idea, because what we're essentially doing is applying qualitative meaning to quantitative change on an outcome measure. To put this in an example, what this might actually look like, what we're asking is, for example, if I do a Western Aphasia Battery, and then do it again, how many points would actually tell me that that person had experienced a level of meaningful change. So that's what we're trying to work out to determine these benchmarks for meaningful change. We've actually recently received funding for this work, which is really, really exciting. We're going to be undertaking a project, where we use an anchor-based method to establish minimal important change scores for the measures that are in the ROMA Core Outcome Set.

Janet: That makes a lot of sense, because I know in the paper, there are some formulas and statistical representations and discussions that might not be easily familiar to some of our listeners. It was a tough read in some parts of your paper, for sure.

Sarah: Yeah, it is. It's probably not the sort of paper that you sit down and read from start to finish, I think. Some of these concepts are complicated, and they are a bit dense, but I sort of see that paper almost as a reference guide. I think it's the sort of thing that you can come back to, and it does, you know, tend to make more sense over time.

Janet: You did give us one example about operationalizing outcomes with the Western Aphasia Battery and minimal important change. Are there a couple of other examples drawn from this paper that you might share, bringing it to the level of our daily clinical practice?

Janet: Sure. Well, I think, overall, one of the really nice things this paper does, is it actually explains that you can determine treatment success in a number of different ways. We go through concepts around, what approach would we take if we're trying to work out does this treatment work for this particular population, and how well does it work? Then we have different approaches where we’re looking at who does it work for, looking at individual change on outcome measures. It really walks you through approaches for group level analysis, looking at mean differences between groups in research trials, versus approaches for determining individual therapy response and outcomes, like minimal important change, and like smallest detectable change.

Janet: Is there an idea or a thought, from this paper and from your work in thinking about operationalizing outcomes that you might give to our listeners that they can put into practice on Monday morning in their clinical practice?

Sarah: Absolutely. This is something I've given a lot of thought about recently, because I think it's one thing to have a very theoretical sort of paper, and to think about the minutiae of all of these issues, but I think for clinical practice it comes down to probably about four different things. (One) I mentioned earlier, I really believe that meaningful outcome measurement starts with shared goal setting. You need to work with your clients to really set meaningful goals that are relevant to them, that they are invested in, and that are going to help them to achieve the outcomes that are important to them.

(Two) The next thing I think, is thinking about, “I have these goals.” We have Clinical Practice Guidelines, we have research evidence, and I would encourage clinicians to use those resources to then really think, “Well, which treatments do we know are effective? Which treatment is most likely to work for the person that I have sitting in my clinic?”

(Three) The next part is when we really get to the measurement part of it, which is really thinking about what you want to measure. Thinking about those goals, thinking about your treatment, where would you expect change to happen following that treatment? Are you looking for a change in function? Or in a behavior? Or is it a feeling, is it confidence that you're trying to change or, someone's emotional wellbeing or an attitude? What is it that you're actually looking to change?

(Four) Once you've determined what you want to measure, it's then thinking about what's the most appropriate way of measuring that? For something like confidence, the best way to measure that is that it really has to come from the person themselves. It's a PROM (Patient Reported Outcome Measure), it's patient reported, it's self-report. But there are many other ways that we can measure things: performance on a task; a report from a caregiver or significant other; it could be a clinician rating or report. It's really then thinking about what's the best way of measuring this? There are all these resources out there like the Shirley Ryan Ability Lab, or Stroke Engine. There are websites where they break these measures down and can give you some information about their psychometric properties. Do they measure what they say they measure? Is this tool reliable? Is it sensitive enough to actually pick up change? I think if you can consider all those things, then you're well on the way to successful measurement.

Janet: That's a tall order! But I think it's a good order. Perhaps if we started Monday morning with just one of those things, and felt comfortable implementing shared goal setting for example, and that became an easy-to-do, relevant part of our clinical work, then we might move on to the other points that you're making and gradually incorporate them.

Sarah: Absolutely. I think at a basic level make sure your goals match your outcome measures. Make sure you're measuring what you're actually trying to change, I think is the basic message.

Janet: Sarah, that sounds like a pearl of wisdom to me. What I would like to ask you as we draw this interview to a close, reflecting on your career beyond the ROMA papers and this paper that we've been talking about, operationalizing outcome measures, and reflecting on your research and clinical career, you've just dropped one pearl of wisdom. Are there any others or lessons learned that you would like to share with our listeners?

Sarah: Yeah, sure. Thinking about my career sort of in total, one of the real highlights of it has been collaboration. I think working together is my other pearl of wisdom, so to speak. I think when we work together and we collaborate, we use our efforts to the best, and in the most efficient way possible, we can reduce research wastage, and we can really put our combined efforts towards improving the lives of people with aphasia. Me personally, I'm involved in a group called the Collaboration of Aphasia Trialists and they have a brilliant website. They're a global network of aphasia researchers, with a lot of resources on their website, which are intended for clinicians to use. They have a particular emphasis on multilingual assessment and outcomes and treatment, which is relevant to all of us in the world that we live in. We're often seeing really diverse populations in the clinic. So, I think yeah, that's my other pearl.

Janet: Sarah, I am an ardent recycler and believe in reduce, reuse, recycle. You've mentioned twice now in our chat, about reducing research and clinical wastage. I think that's a great phrase I want to remember, so that we're not continuing to reinvent the wheel, or spending time and money and resources doing things over again, and wasting, I thank you for that term and that idea.

Sarah, thank you also for being my guest, and the guest of Aphasia Access, for this episode of Aphasia Conversations. I enjoyed our conversation, and I will also say, I think we could probably continue to talk for hours about several other topics, especially related to motivation and engagement and measurement, but we'll stop for now. I learned a lot of new things in reading to prepare for our discussion and also listening and talking with you. I think that your work in aphasia rehabilitation and change measurement is important, very important, not just from an academic point of view, or a third-party payer or funding point of view, but most importantly from the patient's point of view, so that we are delivering the best, most effective treatment we can in the most efficient manner. So, thank you for being my guest today.

Sarah: Thank you for having me, it's been a pleasure.

Janet: I also would like to take a moment to thank all of you, our listeners, for your continuing interest in Aphasia Access conversations. As a reminder, check the Show Notes for today's episode for any references or resources mentioned in today's podcast. For more information on Aphasia Access, and to access our growing library of materials, go to www.aphasia.access.org. If you have an idea for a future podcast topic, please email us at info at aphasia access.org. Thank you again for your ongoing support of Aphasia Access

References, Links, and Podcasts

References

Biel, M., Enclade, H, Richardson, A., Guerrero, A. & Patterson, J.P. (2022). Motivation in aphasia rehabilitation: A scoping review. American Journal of Speech-Language Pathology, 31,2421-2443. https://doi.org/10.1044/2022_AJSLP-22-00064

Breitenstein, C., Hilari, K., Menahemi-Falkov, M., L. Rose, M., Wallace, S. J., Brady, M. C., Hillis, A. E., Kiran, S., Szaflarski, J. P., Tippett, D. C., Visch-Brink, E., & Willmes, K. (2022). Operationalising treatment success in aphasia rehabilitation. Aphasiology. https://doi.org/10.1080/02687038.2021.2016594

Hinckley, J., Boyle, E., Lombard, D. & Bartels-Tobin, L. (2014) Towards a consumer-informed research agenda for aphasia: preliminary work, Disability and Rehabilitation, 36:12, 1042-1050, https://doi.org/10.3109/09638288.2013.829528

Long, A. F., Dixon, P., Hall, R., Carr-Hill, R. A., & Sheldon, T. A. (1993). The outcomes agenda: Contribution of the UK clearing house on health outcomes. Quality in Health Care, 2 49–52. https://doi.org/10.1136/qshc.2.1.49

Wallace, S. J., Worrall, L., Rose, T., Le Dorze, G., Breitenstein, C., Hilari, K., Babbitt, E.… Webster, J. (2019). A core outcome set for aphasia treatment research: The ROMA consensus statement. International journal of stroke : official journal of the International Stroke Society, 14(2), 180–185. https://doi.org/10.1177/1747493018806200

Wallace, S.J., Worrall, L. Rose, T.A., Alyahya, R.S.W., Babbitt. E., Beeke. S., de Beer, C….Le Dorze, G. (under review). Measuring communication as a core outcome in aphasia trials: Results of the ROMA-2 international core outcome set development meeting. International Journal of Language and Communication Disorders.

Links

Collaboration of Aphasia Trialists. https://www.aphasiatrials.org/

Comet Initiative. http://www.comet-initiative.org/

ROMA COS. Core outcome set for aphasia research – The Collaboration of Aphasia Trialists Shirley Ryan Ability Lab. https://www.sralab.org/

Stroke Engine. https://strokengine.ca/en/

Aphasia Access Podcasts

Episode #69: Motivation and engagement in aphasia rehabilitation: In conversation

with Michael Biel

Episode #88: Everyone’s an expert: Person-centeredness in the clinic and research - A

conversation with Jackie Hinckley