Call and Response: How Arizona and IDC Address Nonresponse Bias

A Date with Data

Resources
Response Rate Representativeness and Nonresponse Bias
Nonresponse Bias Analysis Application
Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###
00:00:01.52 >> You're listening to "A Date With Data" with your host Amy Bitterman.

00:00:07.34 >> Hey, it's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.70 >> Welcome to "A Date With Data." On this episode, I am joined by Heather Dunphy, who is the lead education programs specialist from the Arizona Department of Education, and we also have with us one of my wonderful colleagues from IDC, Tamara Nimkoff. Heather is going to be talking to us about how Arizona has been conducting their nonresponse bias analysis that all states are required to complete for SPP/APR Indicators 8 and 14, and Tamara is also going to be here to highlight an IDC tool and talk about some other support that IDC can provide states related to the nonresponse bias analysis so welcome to both of you.

00:01:08.06 >> Thank you.

00:01:09.19 >> Thank you.

00:01:10.50 >> So first off, I was hoping if each of you could just introduce yourselves briefly, say a little bit about your role and what you do. Heather, do you want to go first?

00:01:20.13 >> Sure, my name is Heather Dunphy, and I'm a lead education programs specialist at the Department of Education in Arizona. I work a lot with significant disproportionality and LEA determinations, and I also coordinate most of our federal submissions, including the state performance plan annual performance report, and I'm really happy to be here today.

00:01:42.19 >> Great, happy to have you. Tamara, do you want to talk a little about what you do on IDC?

00:01:47.26 >> Sure, thanks, Amy. So I've been with IDC as a state liaison for mostly a [Indistinct] specialist for several years now. My work has been really focused around the analysis and use of data, previously on the state systemic improvement plans, on using data on structured data meeting and over the recent years more focused on supporting states around their data collections or sampling as well as in the areas of representativeness and nonresponse bias, on the topic of today's chat.

00:02:26.75 >> Great, thank you both. So, Heather, can you start us off by kind of walking us through Arizona's data-quality journey related to nonresponse bias analysis? What does that look like?

00:02:39.97 >> Sure, so I began with the agency about 2 1/2 years ago. The FFY2020 SPP/APR was the first federal document that I was responsible for coordinating, and that was the first year that the words nonresponse bias appeared in the APR, and the question in the APR, it asked, describe the analysis of the response rate including any nonresponse bias that was identified. So I was familiar with how to analyze response rates, but I was unclear what exactly nonresponse bias was and certainly how we were going to analyze it. So in 2021 we did our best to analyze the nonresponse bias to the extent that we could, so for Indicator 8, for example, parent involvement survey, what we did was, we divided our survey window into three periods: the beginning period, the middle and the end, and the idea was that the responses might differ from people who answered this survey early compared to those who answered the survey late, and then we examined those responses that came in from parents at the end of the data-collection period as a proxy for nonresponders. And then we then compared those responders to the ones that came in during the beginning and the middle of the data collection period, so this method gave us some insight into whether or not the results might be biased. And then the other strategy we used at that time for Indicator 8, we were looking at our responses by subgroup and to see if they were representative in respect to certain demographic areas such as race and ethnicity, and then we looked at the rate of agreeableness with Indicator 8 by race and ethnicity, and we were just kind of visually trying to see if there was any nonresponse bias. If we received more survey responses from one particular race, ethnicity, and their level of agreement was different than the others, then there might be nonresponse bias. So this provided a good estimate for measuring nonresponse bias at that time. Those are kind of the tools that we had at that time.

00:04:58.87 >> Gotcha, okay, what about for 14? Were you doing something similar??

00:05:02.10 >> Yes, we were doing the same for 14 that we were doing for, or similar, to Indicator 8, yes.

00:05:09.91 >> Great, so you had kind of your start and had a sense of what kind of makes sense. And then what led you to engage with IDC in terms of supporting you around the nonresponse bias analysis? I know you attended the Hands-On Learning Academy that IDC hosted on the nonresponse bias analysis tool that IDC had developed so tell me kind of what made you kind of shift from what you had been doing initially to wanting to do something different, something more.

00:05:40.62 >> Mm-hmm, right. Well, yes, in the spring of 2023, it was last spring, I had heard about a new tool for analyzing nonresponse bias that was in the testing stage, and they were looking for states to try it out.

00:05:56.98 >> Mm-hmm.

00:05:57.72 >> I had heard that the tool was built to assist states in addressing the requirements related to response rates, representativeness and nonresponse bias and to kind of ease that burden of analyzing survey data for Indicators 8 and 14, and anything that's going to ease the burden of any work, I'm all for it.

00:06:18.89 >> Mm-hmm.

00:06:19.85 >> So myself and one of our Indicator 14 specialists in April went to Rockville, Maryland, for 2 full days of learning about the NRBA App.

00:06:31.23 >> Great, and you said you did attend that?

00:06:35.21 >> Yes, I did.

00:06:35.90 >> The OLA. Okay.

00:06:36.71 >> Mm-hmm.

00:06:37.15 >> Do you want to talk a little bit about your experience at the OLA, what you learned?

00:06:43.00 >> Oh, absolutely, yeah, at the workshop it was Tamara and Ben. They walked us through the various ways to use the app. There are several tests that we can run. They showed us how to set up our data set in the appropriate columns. It needs to be set up in a certain way, and as soon as you have your data set up in a certain way, really the app does most of the work, and I think Tamara can talk more about all the things that this app does, but it really does take the burden off of the user and puts it onto the computer to do the calculations for you.

00:07:20.76 >> Yeah, that's always a good thing in lots of ways. We can hope there's not the user error that we might experience, and it's just kind of pushing a button.

00:07:29.59 >> Right, and the nice thing about the workshop was, it was still a little bit in that testing phase, so we had a small group of representatives from several states, and we could try it out, and we saw that some things were not quite right, and then that gave them time on the developer side to work out the kinks before it went live to all of the states.

00:07:50.98 >> Great, that was, yeah, a good kind of dual opportunity there for IDC to be able to have some testers, too.

00:07:58.39 >> Mm-hmm.

00:07:59.61 >> So, Tamara, can you talk more about the NRBA tool, how it works, why we developed it, how states can use it?

00:08:08.48 >> Yeah, absolutely, and building off of what Heather has shared, really emphasizing that the development of this resource has really been directly informed by input from our colleagues and in the state agencies. We were at the beta version stage in that spring 2023 that Heather mentioned attending. We had other states, representatives from Georgia, Indiana, North Carolina and West Virginia there, and we also had earlier input on an alpha version from our friends in Montana that gave really valuable feedback to inform the development. The impetus of the tool was really ... It's really aligned with our direct support of states' capacity to meet those SPP/APR data quality requirements for Indicators 8 and 14, as Heather was sharing about the work that they were doing prior to engaging with this particular tool, and it was triggered really by those requirements of the 2020 package where it caused us to think about at that time what tools were already in the field and where the gap might be. We felt that the field could really benefit from a tool that was both powerful but also flexible and user-friendly to the extent possible, so we wanted something that would allow a user to choose among many different ways of analyzing their survey data using those best practices, but that gave them some flexibility to have as much guidance along the way as was needed. So it was also informed by some of the kind of common issues that we at IDC had observed over years of supporting states in writing their SPP/APR responses to those prompts, and we knew that we wanted something that could both sort of support states in getting that conceptual understanding of the differences between data representativeness and nonresponse bias, which Heather alluded to, as well as gave them options for digging into their data in deeper ways beyond the submission of an APR report each year if they wanted to. So a little bit about the tool itself just to give people a really high level: It's a browser-based application, online and application, and the first time that people use it, they'll install a free statistical program along with the package itself. It's the program R, which is an open-source program that's widely used across many fields, including education. A couple of things that were really important to us were to make it really flexible but secure, so users access the app within their preferred Web browser like Google Chrome or whatever while the statistical program runs the computations in the background.

00:11:44.73 >> Mm-hmm.

00:11:45.35 >> When users upload in order to use it, their data into the app for a session, it's done via a Web browsers, so no data are actually passing across the Web, so their data remains secure within their local computer.

00:12:04.58 >> Yeah, that's important.

00:12:06.17 >> Absolutely, absolutely, it's kind of built on, the application, it is built on a couple of kind of stages. It guides the user through setting up the session, which is importing their data set and then indicating how they collected those data, telling the application about their data. For example, was it from an attempted census? Or was it from a sample? Then they can choose from a whole series of, excuse me, analysis options, their questions about response rate, representativeness and nonresponse bias. So it's a tool that is not just focused on nonresponse bias but allows the user to look at all of those areas guided, for example, what are our response rates? Do they differ across subgroups? Are some subgroups in our population overrepresented or underrepresented in our data? Is looking at data representative? But then also how do our survey outcomes differ across subgroups? And understanding how those survey outcomes vary across subgroups combined with the information about representativeness of their subgroup is what informs the user about the presence of nonresponse bias in their data.

00:13:36.13 >> Mm-hmm.

00:13:37.16 >> And the tool is also powerful enough that it gives the user options for looking at, can we use some statistical adjustments to reduce nonresponse bias of the data?

00:13:47.94 >> Hmm.

00:13:48.24 >> So not a requirement of the APR but an added kind of best practice of looking at one way of assessing if there's nonresponse bias and seeing how it might be adjusted is by using weighting adjustment.

00:14:05.42 >> Mm-hmm.

00:14:06.14 >> So that's an option in the tool. There are many kind of analysis options that if a user wanted to dig more deeply into their data, they can do so.

00:14:17.62 >> So, Tamara, would the tool actually weight the data for you or [Indistinct]?

00:14:22.64 >> That's right. That's right. All of the analyses, whether it's calculating a response rate or whether it's looking at the proportional difference between representation or whether it is providing a comparison of weighted and unweighted data, all of that is done through these preprogrammed analyses that are part of the application. They're running in the background, and the user is choosing what variables to look at as well as other kind of parameters of the analysis that may need to be decided depending upon the specific statistical task that's being done.

00:15:13.13 >> Wow, that's ... And just having that all in one package, all of those pieces because I think so much of the confusion that we heard from states, especially when the nonresponse bias analysis requirement was added, was not understanding the difference between them, how they are connected, how they work together potentially, and so just having it all together like that I think is so powerful for understanding the different requirements.

00:15:41.52 >> That's right. Yeah, that was really a goal, and one of the things Heather mentioned, the input that was provided at the OLA, which, again, I'll just say was so valuable for our development process, one of the things that is also really a part of the application are the supporting resources around it, so having resources that provide detailed instructions about not just how to install the app but how to set up the data set for really honestly whether you're using the application or not, what elements of the data set are valuable to have for the particular analyses that need to be done for them as well as kind of a pretty comprehensive reference guide that gets to the conceptual pieces of along with using the app, what it means. How do you interpret the analyses?

00:16:54.75 >> Yeah.

00:16:54.84 >> So all of the pieces we want to be a part of the resources available and we're continuing to think about and get input from folks about what resources might be useful moving forward.

00:17:13.25 >> Yeah, and this is something that you don't have to have any type of statistical background to use. What would you say ... A data manager could pick this up and do it with the resources ...

00:17:25.51 >> Yeah, I ...

00:17:25.79 >> ... and support that goes along.

00:17:28.33 >> ... Yeah, absolutely, and Heather can certainly speak to this as well from our, the great engagement that we've had in collaboration around them using this tool with their data is that we really do encourage users to leverage IDC's technical assistance to make the most of the application, but the support is really intended to be flexible. There may be folks who are very experienced in statistics who might choose to use the application independently just using the guide for reference. Others may really benefit from engage one-on-one with an IDETA specialist like myself to work with the data collaboratively to get input on which analyses might be most useful to them or to discuss together, how the results might be interpreted. Folks might want to kind of gain proficiency with the tools themselves with that range of IDC support, and it really is ... The support around the tool is really quite flexible as well.

00:18:43.18 >> So, Heather, tell us about your experience using the tool and what that's been like and how maybe that's changed your analysis, your interpretation, your results in terms of the nonresponse bias and representativeness as well.

00:19:00.24 >> Sure, sure, so like I mentioned at the workshop, Tamara and Ben, they walked us through various ways to use the app, and just like anything we learn that is new, I felt a little outside of my comfort zone at the workshop but with their leadership, the process wasn't too stressful. It seemed fairly straightforward, but then when I returned to Arizona, I tried using the tool on my own. I found it far more difficult than I anticipated. I felt clumsy. I was making mistakes. I tried to follow the written directions I had gotten at the workshop, but I was stumbling with the steps, so I reached out to Tamara. I felt a little embarrassed because I needed additional support after spending 2 fulls days with me but you know what? She made me feel totally comfortable. We went through some of the exercises together. She helped me understand how to interpret those results. She made sure that my data set was organized correctly, and then after that the process was fairly simple. Just like Tamara said, the RStudio app will run all of the calculations, so it can do some of the basic calculations like calculating the response rates by subgroups and comparing subgroup percentages in respondent data to data from respondents and the nonrespondents, but what I found really helpful is learning, do we have that nonresponse bias? And so, for example, in Arizona the Indicator 8 data we have is about 92 percent of agreeableness, but we're only receiving 14 percent of responses, so that's a lot of unknown. That's 86 percent that we don't really know how they're going to respond, and so the tool was really helpful. We found that if we were to get everyone to respond, which we only got 14 percent, but if we were to extrapolate that and get everyone to respond in respect to race and ethnicity, it was extremely close to that 92 percent that we calculated. So that gave us ...

00:21:08.09 >> Good.

00:21:08.25 >> That gave us a lot of confidence that we didn't have nonresponse bias in respect to race and ethnicity for Indicator 8, and we also looked at Indicator 14, and we received quite a few responses. We have about a 74 percent response rate, but you don't still don't know if there's nonresponse going on even though we have quite a few respondents, so we ran the test there, and what we found is, we're getting about 80 percent of our responses from graduates, okay? So we get quite a few of the graduates, but we're getting fewer responses from dropouts. It's harder to get responses from the dropouts, and so we wanted to know if there's some nonresponse bias going on. And the areas of engagement differ from graduates and dropouts, and especially when we look at, for example, who's going into higher ed, what we found is that for the percentage of youths with IUPs that go into higher ed, it's about 19 percent in Arizona. But we were wondering, if we received responses from everyone, would it still be 19 percent? So that's where the tool really came in, in handy, so in a perfect world, if we had gotten all of the responses from graduates and all the responses from dropouts, the app showed us that instead of 19 percent, it would be lower. It would be about 17 percent.

00:22:31.25 >> Mm-hmm.

00:22:31.88 >> And so my next step in this process would be to learn about, is that significant? What's the level of significance, that 2 percent difference? And that's something that I think the tool can help with. I would like to learn more about that in the future.

00:22:47.34 >> Mm-hmm.

00:22:48.91 >> So I'm still kind of at the beginning stage of learning about the tool, but from what I've seen it is really, really neat and saves a lot of work.

00:22:57.37 >> Yeah, did you use it for the SPP/APR you're working on that's due in February?

00:23:02.25 >> Yes.

00:23:02.46 >> Great.

00:23:02.76 >> Yes, we did, mm-hmm.

00:23:04.11 >> That's very exciting. Well, it sounds like, yeah, you've gotten a lot out of the tool, and there's a lot it can do and a lot more still to even explore with it, so kudos to you all.

00:23:17.68 >> Yeah, thank you. I will add that any time you're working with something complicated, you try not to make it so complicated. You try to simplify it, so one thing that did help me to understand the tool better is, I made a smaller data set of fake data. So instead of looking at 10,000 responses, I just made a false data set of 100 responses to help me just understand the tool better. And so in respect to race, ethnicity, I just made 50 were Hispanic, and 50 were white, and I ran different scenarios to see how the tool worked and what if more Hispanics answered? What if more ...

00:23:57.06 >> Yeah, how would that change?

00:23:57.95 >> ... white people answered? What if the Hispanic level of agreements was very high, and the white level of agreement was really low? How does that change each of these calculations. So I would suggest to anyone learning about this tool to try that, to create a smaller data set and just look at the different statistical calculations that the app provides and what states might find valuable.

00:24:19.09 >> That's a great tip. Do you have anything else for states that are just interested in possibly using this or have starting using it that might be helpful for them?

00:24:28.87 >> Sure, I would say in addition to trying a small data set, to reach out to IDC. Like I said, I was kind of unsure, kind of feeling clumsy about it, and they are just so helpful to make sure that your data set is set up correctly. If it's not, the app is not going to work at all, and so that's the first step. And then when you finally get the results, you might need some help interpreting them, and that's where IDC can really come in handy as well.

00:24:54.46 >> Great, so, Tamara, for states that are interested in learning more about this and getting their hands on it, can you tell us how they can get the tool and maybe a little bit more about the support that IDC is able to provide?

00:25:09.45 >> Yeah, absolutely, well, the app itself, information about it, if they want to kind of self-explore is available on the IDC website along with those supporting documents. Also on the website in various places are various kind of presentations that we've done either at the SPP/APR summit, for example, that speak to both the topic as well as a bit about the application. If a state user doesn't want to kind of engage in it, then contact your IDC state liaison, and they will be able to connect you with a TA specialist. In terms of what's next, we are continually improving and looking for those opportunities to provide TA, really, that help inform the hearing from the field on what other kind of supporting resources would be useful. It's been immensely helpful to collaborate and rewarding to collaborate with Heather in Arizona and, for example, mentioning the smaller fake data set, we have small data sets that we have developed that are, with fake data, that are for Indicators 8 and for Indicators 14 that we use for demonstrations with the app, and we certainly could make those available to folks so that they could then ...

00:26:43.24 >> They ran themselves.

00:26:44.27 >> ... use those, absolutely, so we're constantly learning from each other in this process, so that's great, and as she mentioned and I mentioned, really our support is very flexible and geared towards what's going to be the most useful and meaningful for the state user in whatever phase they are in, in this process.

00:27:09.43 >> Great, and we'll put links to the resource application itself as well as some of the other presentations that might be helpful in the notes for the episode so folks can easily get to them.

00:27:21.77 >> Great.

00:27:22.56 >> Great, well, thank you both so much. I know that I learned a lot more about the app than I knew before, and it seems like something that's so useful and helpful and so appreciative that you came on and have shared your story so other states can hear about this, too.

00:27:39.92 >> Oh, you're welcome. Thank you for having us.

00:27:41.84 >> Yeah, thank you so much.

00:27:42.96 >> Of course.

00:27:44.71 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org.