Bonus: Elevating the Conversation on NAEP Scores w/ John Warner

‍

0:00:11.9 Nick Covington: Hi there. So my name's Nick Covington, I'm from the Human Restoration Project. And we're here to talk today about the NAEP results. So the results of the National Assessment of Educational Progress or N-A-E-P, NAEP, were released yesterday, September 1st, prompting a New York Times headline that read "The Pandemic Erased Two Decades of Progress in Math and Reading". The 74 headline added "Two Decades of Growth WIPED OUT by Two years of Pandemic". Peter Greene, an education policy watcher, called it NAEP Pearl Clutching Day. And I myself even tweeted a lot about this "With the release of the pandemic NAEP scores, we're about to have the worst cycle of education discourse imaginable", and man did that ring true. So everyone seems to run to their corners to abolish teachers unions, attack the remote & hybrid learning, precautions that we took, mask mandates and just relitigating every pandemic issue imaginable. The results even brought out the usual resident experts in everything, like Matt Yglesias, who called the scores "A Short-term L for the left that was more supportive of closure".

0:01:24.0 NC: So while everyone online is jumping to conclusions we thought it would be important to help provide some context, step back and take inventory of the data, the claims, the headlines, and provide some context and forecast the next steps. So what, if anything could, or should we do in response to that report? So I reached out to author and educator, John Warner, whose intuition, I tend to trust on this kind of thing. John's the author of several books, Why They Can't Write, The Writer's Practice and Sustainable. Resilient. Free.: The Future of Public Higher Education released in 2020. So thanks John for making the time to turn around and talk with me about that today.

0:02:01.2 John Warner: Oh, my pleasure. Good to see you and try to make some sense of this stuff that's going on.

0:02:07.9 NC: Yeah, let's get right into it. So let's just start with what the results say and what do they mean? And maybe we can compare that to the headlines as well. What did the results say? What should we make of them?

0:02:18.7 JW: So in contrast to the headlines, which frame this around this idea that somehow learning... The nine-year-olds. Again, we gotta remember what we're talking about, we're talking about nine-year-olds who took a single test for reading and math back sometime in the early part of the year, have scored at a level that is consistent with student scores on average from 20 years ago. The history of the NAEP test has been a kind of relatively steady growth and increase in scores. So when we say it's equivalent to 20 years ago, it's significantly lower than students who took it the last time it was administered, which was 2020. And therefore on par with students who took it in, more like students who took it in 2002. The headlines, erasing two decades of progress or two decades of growth are wiped out, are absurd. That these were allowed to be published in major national publications are, should be to their everlasting shame because that's not what this data says.

0:03:35.3 JW: What this data says is, at one particular time and place nine-year-olds scored this on this exam for reading and math. We can now take a step back and ask, So what are the factors that have gone into this score that is significantly lower than the last time it was taken, that is on par with 20 years ago, prior to what has been a sort of gradual increase? But the idea that suddenly our nation's nine-year-olds are 20 years behind [laughter] where they should be it's nuts. And even just thinking, let's assume that's true, right? That somehow our nation's nine-year-olds have been getting steadily smarter over the last 20 years, we're learning steadily more over the last 20 years. How are our 28, 29, 30-year-olds doing today? Just fine, right?

0:04:36.6 NC: Mm-hmm.

0:04:37.6 JW: The notion that these scores are dispositive, that they carry some sort of predictive, inevitable predictive weight for a future 20 years from now, 10 years from now, five years from now is a total abuse and misuse of what a single test and a single data point means. It doesn't mean it's meaningless, but it does not mean what these screaming headlines say.

0:05:03.9 NC: And I think absent from those singular pieces of data that are going around now, is all of the context of the last two years to go alongside that too. Something that you've commented on and I've commented on as well is, missing from reducing the experience of the last two years into two singular points, and particularly just for reading and math, disregarding every other subject that that kids would study in school. But kinda just disregarding the project of the last two years. Which might not have been to keep raising math and reading scores, but instead to turn schools into safer learning environments that adults and students alike can inhabit without getting each other sick, risking bringing infection home to others, scrambling for schools to be able to provide the same level of services that schools have provided in terms of free meals to students and obviously childcare then for working parents as well. So there was just a seismic cultural shift across the entire board, and one of the reasons why I think the NAEP is called The Nation's Report Card, it's the gold standard for these standardized assessments is, even though the world might change around it, NAEP always stays the same. And yet here we have this assessment result that seems incongruous with our experience of the last two years.

0:06:30.6 NC: I was just looking back at a Gallup survey from earlier this summer that said K-12 workers are the most burned out employees in the United States. Higher burnout rates than medical workers than anybody else. College faculty are on that list as well. So when you look at a five-point decline in math or reading scores, alongside teachers are the most burned out employees in the entire country. You say, what were we doing? If we were just sitting at home doing nothing, surely we wouldn't be as burned out as we were. So why does that framing matter? How could... What could a more meaningful, I would even say like conscientious, conscientious, reporting of these results do? What's missing from this discourse that we're having online about this?

0:07:20.1 JW: Well one of the things we should recognize is, as the Nation's Report Card, as the snapshot of something that... Of a test on a single day in a single context or a single time is disruption has happened and this is now reflected in the data. Whatever schooling was, in whatever form it took, under whatever circumstances was disrupted by a global pandemic. And so not only school was disrupted, the entire lives of everybody in the country and the world was disrupted. So the notion that we could have avoided this, with some kind of different school or different choices, going back in-person, sooner or anything like this, it's not reflected in any of the data. In fact, in some of the underlying data, they released, there's indications that none of that mattered. Like in reading scores, students who are in city districts, their reading scores were actually flat, the only metric that didn't go down. And these were schools that were much more likely to be closed for longer. Regional differences, don't tell us much of anything in terms of schools in the South, which were more likely to go back to school in-person sooner had decreases equivalent to and in some cases, greater than schools in the Northeast, which we likely to have gone back in-person later.

0:09:02.0 JW: So what we have really is a picture like a big thing happened, and this big thing has affected how students perform on this test, and it could be any number of reasons. It could be something like actual specific test prep, is the thing that was sacrificed. And if that was the case, that would be a really interesting finding. It would tell us that these sorts of tests when they're prepped for, maybe don't tell us all that much about what students are learning. I don't know how big a factor that is or if it's a factor at all. It could be that students were grieving the loss of a caretaker. It could mean that students who are, and there is some data to suggest this. Students who are on the lower end of the scale in terms of performance were more significantly affected by these disruptions so their scores decreased by a larger percentage. So it's like these marginal students often in situations that, where they don't have sufficient resources for their educations or even their lives, their scores fell off a cliff while others were more modest. These are all things that we can dig in deeper and try to figure out over time, not necessarily from the NAEP score, but other tests and other indicators.

0:10:34.7 JW: But we don't know what many people are claiming to know, that in-person school would have made a difference, or removing masks would have made a difference or that remote learning cannot work under any circumstance. We do know that students who tend to have less access to good educational resources did even worse than those with better access, but beyond that, we just don't know all that much. We know something really big happened and it's going to take time to recover from it. I used this analogy yesterday in a piece I am writing and working on. Imagine we're talking about domestic air travel, where over 800,000 people took a flight in 2019, and that dropped to 300,000 in 2020, and it's up over 650,000 or something like that in 2021. If you compared 2021 to 2019, you would say that the air travel industry is collapsing because we're down from 800,000 to 650,000. The reality is, it's in the midst of a recovery after a large shock that has caused significant change.

0:11:53.2 JW: I think that's a much more plausible explanation of what's happening with students. Then the idea that these students will be permanently harmed or that they cannot recover from this, I think is totally unfounded, counter-productive. And really anybody who wants to say that is pressing an agenda that does not have much to do with trying to understand at a deeper level and more practical and actionable level, what can we do to help students in schools today, tomorrow, next year. Who are gonna continue to experience some level of disruption from this pandemic, who are gonna continue to have their own issues around burnout, just as their teachers are and are not out of the woods in terms of the effects of this disruption.

0:12:47.3 NC: Certainly. And I think what is so interesting is this just is such a microcosm of what it says about how we use data in education, of what data counts, and kinda how we communicate the success of schools and school systems in the general public. A couple points on this, I think, one of my concerns that is, just understanding how reactive our political systems are to something like the NAEP Score to a New York Times headline. We're gonna be making some really dramatic changes to education in response to, in the short term, in response to this singular piece of data that is just going to serve that data. So the conversations that I've seen are around accelerated learning, extending school hours or the school year, even though the United States already has students in schools, and American teachers already work more hours and teach... Have more contact time with students than their counterparts in the rest of the world, all of these potential solutions.

0:13:52.5 NC: And one thing that I think is particularly interesting too, is that I don't think it was... These results are a surprise, right? I don't mean to sound callous at all in this kind of sense too. To your point, we faced a global disruption. We have a million dead Americans. We have hundreds of thousands of orphaned children, and right, to expect that schools were going to be immune from those changes, we really were just going to be... It was a question of, what is the drop gonna be in these test scores, right? Because schools were focused on other priorities. Now, the other part of this that I think is interesting, and again not surprising, is that those... The score drops tracked alongside the groups in society who would have been most heavily impacted by the various waves of the pandemic, right? You said people in the Northeast, the South, poor communities, Black and Brown communities that are underserved in so many of the other aspects of public life, and access to healthcare, and all of that as well. And I think what is particularly frustrating, is our response is not really gonna meet the needs of those communities, in the sense that we know that it's gonna take more resources to sort of... But not just resolve the tensions that are simmering socially, but there's also this leap into the pedagogical side of things too.

0:15:17.5 NC: So we wanna try and solve these systemic social issues of lack of access to healthcare and poverty, etcetera, with these pedagogical programs. They're gonna try to solve it when they get to the schoolhouse door. So kind of to respond to this, I'm curious what your response is here, but what do we say to those people who are going to argue that this data is important, that it's helpful for us in order to address those equity concerns? So it pretty demonstrably shows that people in underserved communities were hit harder than the people with more resources. That's not a surprise to anyone who's been in education for longer than five minutes. They would argue, "We have the data. Here's the data, here's how it shows how kids who are behind fell behind even further."

0:16:08.3 NC: Again, probably referring to poor kids, Black and Brown kids, etcetera, that this data's gonna help us target those interventions, accelerations, etcetera, right? There's a lot of sincere people who believe that that's going to be doing what is best for kids, and what is the response to that? Is there a response? What's the conversation?

0:16:26.4 JW: Well, I think if there's any clear response to this data, and really, this is a response to the NAEP data every single year. It's that under-resourced schools need more resources. And schools by themselves cannot fix larger social ills, and lack of access to stable, healthy existences. The data says, as we've been talking, if you were in a worse situation in terms of access to high speed internet, or your own computer, or a quiet place to study, or some adult coming to help, your score dropped more. These are all things that are still in place, even if we go back to school.

0:17:28.0 JW: And school really, this test and schooling in general, is really often a by-product of the larger societal atmosphere and the things that are going on around students. The notion that this is gonna be solved by a kind of accelerated schooling, or sometimes what sounds just like a harder cracking of the whip over the heads of students as we try to spur them on to achievement, seems misguided to me. Because as we know, the loss has not only been to their academics, it's been a loss to their overall experience of their humanity, as it has been for all of us, right?

0:18:20.3 JW: Teachers and others aren't burned out only because they've been working under difficult circumstances, either through remote learning, or the challenges of trying to teach in-person while mitigating the virus, it's that there has been a lot of other stuff going on in our lives around the virus, not the least, loved ones getting sick, or and potentially dying. So there's a larger recovery to this, and a longer recovery to this. One that...

0:18:53.5 JW: Points I try to make with people is, a year from now, we're gonna get next year's NAEP scores, and they're gonna go up. And people are gonna be asking about the NAEP miracle, how did we return to the previous status quo or close to it in just a year's time, when we are 20 years behind last year. And my answer right now... And it will be the same next year, is time. We need time to process, we need time to quite literally heal, we need time also... And this is the thing that I wish more of these conversations were about, to figure out what school can and should be going forward. The tremendous hunger to return to the status quo among many corners of the discussion, baffles me. Because there were a lot of criticisms, justified criticisms of the status quo from various all parts of the spectrum, from people who I would fundamentally disagree with, they were also critical of the status quo, I was critical of the status quo. Why are we rushing to go back to something that nobody thought was ideal? Just because it's more comfortable, just because it seems better than what's going on currently.

0:20:22.8 JW: I think those are all natural responses. But I think it's a shame if we don't use this as an opportunity to have better, deeper conversations about what students would benefit from in our schools. So we can use these scores as a way to have those conversations. Unfortunately, that's not what's really been going on, at least not in the first 36 hours since the scores release.

0:20:49.4 NC: Yeah, this is definitely the instant react kind of idea. But I really doubt that the discourse is gonna get a lot more... A lot healthier after that.

0:21:00.3 JW: It's become this tug of war, right?

0:21:02.3 NC: Yes.

0:21:02.5 JW: Of the various forces that want to bend schools to their idea of what school should be, and so seething on this data point, in order to do that. It's wholly predictable, it's dismaying, it's very difficult to counteract, because the first impulse is to simply grab your end of the rope and start tugging it in the direction you want. But if we can at least advocate for a smarter conversation, maybe we can make some collective progress. I try to take the good faith of those who see these things differently, seriously. That they also want students to thrive and do well in school. So if that's the case, we should be able to have a better, deeper conversation, rather than litigating the past, rather than trying to get in our time machine and decide if students should have been in-person sooner, or should have been masked or whatever. All that is done. People try to make the best decision they could have based on what we knew at the time, that's all we can do right now going forward. So let's have a decent look at what's happening, so we can make some better decisions.

0:22:27.4 NC: I think that's a great place to end it, John. So thank you so much for spending some time unpacking that with me today.

0:22:32.5 JW: My pleasure. Thanks.

[music]

0:22:36.0 NC: Thanks again to John Warner for joining me for that conversation. You can follow him on Twitter @biblioracle. You can also keep the conversation going by following us @HumResPro and by visiting our website, humanrestorationproject.org. We have an entire Learning Loss handbook that is over 50 pages of history and insight into these tests, and the learning loss narrative, particularly surrounding the last two years of pandemic schooling. In addition, we have a podcast on the topic of Learning Loss with Akil Bello and highlights of that same episode, available on our YouTube channel, if you search for Human Restoration Project. Thanks for listening.

[music]