Unpacking Learning Loss: The Who, the What, and the Why

Speaker 1 Hello, and welcome to episode 96 of our podcast at Human Restoration Project. My name is Chris McNutt, and I'm a high school digital media instructor from Ohio. Before we get started, I want to let you know that this was brought to you by our supporters, three of whom are Kate Rowbottom, Kimberly Baker, and Rachel Lawrence. Thank you for your ongoing support. You can learn more about the Human Restoration Project on our website, humanrestorationproject.org, or find us on Twitter, Instagram, or Facebook. Today, I am joined by Nick, and we are going to break down Human Restoration Project's recently released Learning Loss Handbook, providing an overview of the ideas that we present in the book, the phone narrative being created surrounding learning loss, and offering an alternative for educators to push for and pursue. You can find a free download at humanrestorationproject.org/materials. Nick, we are here, and we are talking about the handbook. And we were just saying, hey, let's just go through the entire thing, talk about what's in here, what we learned, the struggles along the way of trying to comprehend this, and basically, what we're looking at here. And we had said before that basically, Learning Loss Handbook, in in our view, is the best resource as of right now to understand what's going on. Because what we did is we we tried to combine a bunch of folks that were, I guess, way smarter than us and all assemble them into one place so that we can really tease this apart.

Speaker 2 That was, like, the genesis of this idea is that throughout the pandemic, going back as far as panic, like the moral panic over schools in the spring of twenty twenty when, you know, COVID was just getting started, I I I don't recall necessarily the learning loss narrative there. Maybe it started to pick up over that summer twenty twenty and and into the fall as schools mold over hybrid online remote options. But, you know, by the spring of twenty twenty one, it really had kind of solidified into into that solid narrative of, like, something profound is being is being lost here, but it didn't seem to be focused on what I think we felt were the right things that had that had been lost in those now nineteen, twenty months of the pandemic. So we didn't see anything out there that had addressed it, I think, from that perspective. But I was just going back through and kinda seeing some of the things that led us to release it when we did because it was it had been a project that had been in place for months. And it it was really curious that that one of the things that kinda caused us to, you know, put in the last the last kind of pull an all nighter and get that thing out there before the end of August was an article that came out, an op ed in the New York Times, the school kids are not alright. And the whole piece was focused on this data that had been extrapolated out from from the learning loss narrative, not just to suggest that there would be academic academic gaps or academic learning loss that would need to be recovered, but actually that this could be extrapolated into the loss of future potential incomes, even as far as to say the impact on The US economy could range from 128,000,000,000 to 188,000,000,000 every year as the cohort enters the workforce. So just this this huge moral panic, not around, you know, the the sort of the socio emotional impact on kids, not the impact on the the lives that were lost and kids that we know were orphaned or and are now in worse life situations because of the ongoing impact of COVID. But this asinine idea that because because kids had a an adapted school experience or, you know, maybe didn't finish their twenty twenty school year as they would normally have done, that that is gonna lead to, like, a a drop in lifetime earnings and and US GDP felt like that was the wrong thing to do. So for us, I think it was trying to refocus on on what matters and what had mattered during that time and then try to try to dispel a lot of those myths that had popped up around learning loss, where that data was coming from, and try to give some context to that because it kind of seemed like the media portion of it was sort of meant to to circumvent that frontal cortex where you think about things and just go right to the amygdala and just just turn on that fight or flight response and demand, you know, unsafe conditions for school, which I think we're seeing now

Speaker 1 has has backfired in a big way. Before we move on and talk about more about what we have in this handbook, I do wanna reiterate the fact that we're not claiming that learning loss didn't occur in terms of a major disaster didn't occur in our lives. I guess there's no issue going on, that something wasn't lost. What we are claiming is that the narrative surrounding learning loss, as in the one that's been propped up by the McKinsey report, which we'll talk about in a second, as well as via testing companies in general, serves to earn profits for testing companies. It serves to have districts and therefore students take a bunch of additional tests, purchase a bunch of test remediation software, and do many things to increase their learning loss, quote, unquote, from the pandemic. That is not the same as reestablishing connections. I guess the the TLDR of the handbook is stop focusing on purchasing products that will increase test scores because that will lead to a lot more problems than it might attempt to solve. And instead, focus on connections, rebuilding community, and social emotional learning. I mean, it was a global pandemic. It's still going on, and it's it's a major problem. The handbook itself is divided into two parts. The first part, which we'll start with is, okay. What is learning loss? Where does that come from? How is it being used? How are we making these claims? What's its connection to the standardized testing industry? And part two is, okay. Knowing all that, what can we do instead? What other connections exist? How can we reimagine systems? How can we build back better? And I think that, really, the the narrative is sadly timeless or it's cyclical. This is something that's not new. Even though we're talking about learning loss in a pandemic context, so twenty nineteen, twenty twenty school year, twenty twenty, twenty one school year, hopefully not longer than that. It's still going on with summer slide, the quote unquote learning crisis, even things like nation at risk and and previous reports where test scores have been hyper low. And as a result, testing companies have said, hey. We need to buy all this stuff. We need to fix this. Have your kids do all these things. But I'm sure most teachers listening in have been told at one point or other to use some kind of testing software or test prep company's materials, had a kid sit in front of a computer or do some worksheet and say, hey. We're gonna increase test scores. And, usually, it's absolutely miserable. Our claim is it's not really rooted in anything that's relevant, and, hopefully, we prove that here today. We're just gonna start with it. Nick and I were just talking about how this is kind of like the, like the the audio narrative over, the DVD special edition. We're gonna talk about everything about what happened in this learning loss handbook, where it came from, and hopefully give you all a good idea about what it is that we're trying to talk about here. So the very first thing that we started talking about were just terminology and history. The beginning is just breaking down whatever the heck is going on when it comes to this. And I I still remember in our first meeting when we were just trying to dissect the reporting around learning loss, It took a solid hour or two just to understand what exactly this is. It's it's very confusing, and I wouldn't be surprised if there is at least one error in this learning loss handbook because it a lot of the stuff doesn't make sense. Perhaps it doesn't make sense by design. Like like, if it's kinda like just a bunch of people working around something that just doesn't work, so therefore it it just doesn't make sense. Or maybe it's just I don't know. I don't I don't really know how else to put it. And that's the thing I think that is that is really interesting

Speaker 2 too in the way that, you know, we we have continued to kind of seed a lot more power into those those groups that claim to be generating the data that we would make our data driven decision making from. But but a lot of times, people in education who work in the classroom or are closer to the classroom, right, aren't really equipped with the toolkit to understand what that data is saying. So as we, as classroom teachers, began to dig into that data, it came with a whole new terminology, a whole new jargon that we really had to dig into to ask not just what this is saying, but then is this thing significant? So we had to understand the language of the of the of the the standardized tests in order to be able to interpret whether, you know, a one to 3% difference is significant or if if that's even important. Right? Are these things worth spending millions and millions and millions of dollars to try and recover? Like, what is this even measuring? So, yeah, so that's why we start this thing off with with basically a glossary because a lot of the terminology is fly by night. Like, it's new. Like, what is accelerated learning versus learning loss or learning recovery? Are those just synonymous for things? There's been a lot of a lot of Internet debate about how what the appropriate use is of those terms, but they all they all you know, they're they're the tip of the iceberg of that same kind of idea that that goes back not just to the summer slide and those notions that in the the seat time away from school, something is lost if we're not constantly cramming or pouring things into kids' brains. It's just gonna to ooze out of one side or the other. But really then, I don't know if it's because we're addicted to context because, like, we're we're we were we were social studies teachers, you know, or something. But but we felt like we had to go all the way back to, like, what, the early nineteen hundreds and talk about, like, basically, the twentieth century the the industrial revolution in the twentieth century and the advent of of of management and this concept of tailorism basically trace that tailorist idea, you know, that was used to fine tune factories back in the day, clear up through what? Clear up through the Cold War and Sputnik, and then, of course, in the nineteen eighties through A Nation at Risk, and then up through then, of course, No Child Left Behind in the in the early two thousands, a bipartisan, you know, push to a push towards standardization and the punitive use of of assessments and the new accountability movement. And then, of course, president Obama's own version of that with the race to the top, kinda putting, you know, putting financial incentives in place for for schools and districts and states and stuff who who could who could adopt these test score delivering programs clear through, gosh, clear through today, right, with with McKinsey and company, and they're I think they are probably the most cited in this whole learning loss narrative. So then, yeah, a little bit later on, we go a little bit more into into Mackenzie. But the whole premise of this thing is, like, measured on this Taylorist notion of seat time as sort of a method of inputting information. And the lack of that seat time means that there must be a loss of that information. And being able then to, what, be able to break these things down into, well, x number of days missed means you are this many units of learning, you know, behind or what you have lost from where you were previously. So I mean, we we trace it back to Taylorism,

Speaker 1 but also the the fact that McKinsey let's let's talk about McKinsey right now. McKinsey is a global consulting firm. Like, they are rooted in foreign policy, but specifically, their interests lie in, like, restructuring, usually management, firms or companies. They're they're the folks that, go into companies and will fire a bunch of people in order to try to increase their, stock prices. Kinda like old, I guess, bit or I guess, more modern day. Pete Pudiesch worked for McKinsey. McKinsey has been tied to the the opioid crisis. They've been tied to some shady arms deals and work in Afghanistan. They have a pretty seedy history, in general. And they've also worked at just basically spreading the idea of different business or or neoliberal style practices to every facet of America and where America is, A K A, the world. This this connection to learning loss is based off of the idea of global competitiveness through testing. Whenever we do testing in The United States, we center our work on, are we keeping up with Japan or with France or whoever, with our math and reading scores? And if our math and reading scores are not, McKinsey, as well as the folks who draw upon McKinsey's work, believe that that by using Taylor's principles or by just standardizing, this model and and putting more inputs in and treating school like a corporation that you will see increased results. But sadly, as you and I both know, and I think most people listening, the school is not a corporation. It doesn't work like that. You do not learn through neoadjuvantic ways. You can't just hire a new manager and then just say, okay. Everything's better now with all of the different problems that face the American school system. And it's not the teacher's fault per se. It's all of the underlying systems, including things like funding, solving issues of inequity in society, etcetera. So fast forward over about fifty years since McKinsey was founded, you have, as you just said, no child left behind, race to the top, nation at risk. These are all just different reports that say American kids are behind. There's a giant achievement gap, etcetera, etcetera. So put more money into schools to address standardized, testing, so we can figure out what's going on and try to make things better. I think that that's rooted in perhaps good ideas. I I I don't think that everyone that believes in those things is necessarily doing it for nefarious purposes,

Speaker 2 but there's certainly a very strong lobbying wing that invests in this. That Taylorism, right, is rooted in the efficiency movement, you know, and and and that notion that, you know, what could work to, either in induce workers to work faster or churn out more widgets, or however we could, we could speed up the assembly line, kind of practices there. That that's what that kind of mechanical, management efficiency movement is rooted in. And it's just that that mindset, but applied to schools, kind of like as that as a conveyor belt. And so right. The notion that someone is at a third grade level or at a sixth grade level or at a whatever level kids are supposed to be at is I mean, it's completely arbitrary, right, and determined by by whom, you know, by the makers of this particular test, by is is it determined by community standards? Right? Is it determined by national standards? Is it by global standards? By whose standards are we judging where these kids are at? So, you know, I imagine my seniors now that I have who have had, right, two, three years now, gosh, it it seems like forever, but, you know, they're the tail end of their sophomore year and then their entire junior year and now heading into their senior year disrupted, by by the pandemic. Right? It it doesn't fit into that efficiency conveyor belt mentality because if if this pandemic has shown anything, it's that human beings and human life is messy and and inefficient, particularly in times of in times of crisis. So why would education be anything but, you know, messy and inefficient in this time as well? Right? The notion that we would have stayed solid and steady on this conveyor belt or should have stayed solid and steady on this conveyor belt is something that I think these the the the folks pushing this learning loss narrative haven't have never addressed. Right? What would be the acceptable level of disruption given the miss the malevolent mishandling and mismanagement of this pandemic in The United States Of America? Right? We we can point to international examples, right, and say that, oh, such and such kids in such and such country are know, going back to schools as, quote, unquote, normal and pushing for those same ideas here when those other countries have just done they've done the other public health work to make those things possible, and just we have it. So there's a there's a lot else tied up in in this notion, but the idea that we can distill children's experience of this pandemic and and the community's experience of this pandemic because it does not fall on every single community equally down into an assessment score, and that's what we're gonna use to rebuild upon and and to build the future upon. That's rebuilding our future on McKinsey's values. That's rebuilding the future on the college board's values. That's right? And and Pearson's values and the people, again, who are giving us the data that we're using for data driven decision making and then turning around and buying their products, to resolve this crisis. So, you know, it's gonna sound conspiratorial, but it's really at if you just peel the thing away, it's it's just about, you know, power and money, is really what it is. And it's about, you know, them providing marketing marketing materials to solve a problem that they have the assessment data in their minds to to resolve. And so it's a financial incentive for them to to draw this into, you know, into a hysteria so that way they can, right, get schools to buy their accelerated computer software or to fund, you know, their summer academies.

Speaker 1 I have a feeling that we're probably gonna spend the most amount of time just talking about that. But I do wanna make sure that we start off with square one or step one, which is just does does any of it matter to begin with? So if we pull back the curtain and we just say, let's imagine that these reports never came out. Does the standardized test actually measure anything of use? Like, if we just look at the questions and we look at what standardized tests do and we look at the the ramifications of issuing them, does it even matter if, hypothetically, there were points that were lost or students are falling behind? Like, what exactly are we measuring? And we started this by simply looking at a series of different questions from a variety of different tests. We we pulled from the map, the NAEP, the STAR, a couple state tests, the PISA All the acronyms. Yeah. All the different acronyms. All the different tests that that kids have to take. Freshman year, our our students last year, I believe, had to complete 17 different tests, in the last two weeks of school. So it's just brutal. Like, it is brutal how many tests pop up. And we didn't I I was the one that that threw these questions in the in this book, and we'll read a few of them here in a second. I didn't cherry pick these. Like, I didn't Google, like, hardest test questions or, like, search through and browse a bunch of test questions before I chose one. It was usually, like, one of the first three, and I kinda chose them at random just because they're all absurd. Like, we'll start with the ones that are supposed to be the easy here. We'll do a few. So there was there's a test question on here, from the map. It's adaptive test. So you could get this question at any point in elementary school depending on your, quote, unquote, reading level. And it's two paragraphs about swimming. And it's like, says swimming is really fun. It's really easy to learn how to swim. My favorite place to swim is the lake near my home. I do not like swimming in pools. Pools are too crowded. And then Carl says, swimming is hard hard work. It took me all summer to learn how to swim, but I'm glad I learned. I like going to pool to swim. I have fun swimming with my friends. And then it has you sorting between on the things that Carl and Alex agree on and things that they disagree on. You had to, like, pull them into the right, sections. And what we were curious about is if you were reading this prompt, first off, could you answer it? I think most adults probably could answer this question. But I I think the better question for this one is, like, why might a student answer this question correctly or incorrectly? What barriers

Speaker 2 could fall in there? This is on page 13 if you're if you're looking in the in the book at the at the same time that we are. But, I think the thing that that happens when I see this is that there is just there's just cognitively so much going on on the page for this question. Like, if I'm if I'm a child and taking taking this test, it's just really hard to figure out the formatting and then to figure out what the question is asking and then how and then processing, how does it want you to answer it? So something that could be very easily resolved, you know, or assessed between students and teachers just by having making this a conversation. You know? And then you can pretty easily sort out what kids, you know, get and what they don't get turns it into, I don't know, kinda kinda like a cognitive nightmare, honestly, just just trying to sort through this piece. And, yeah, clicking and dragging and moving around, so there's, like, the technological elements of this thing too. Yeah. It's just

Speaker 1 it's just kind of a it's just kind of a nightmare question, honestly. Well, also, like, the there's a a cultural component of this. This has been heavily critiqued. The the the narrative surrounding this typically brings up cultural critique and the fact that there is a sizable population of people that don't swim in pools, that don't have a pool near them. That if this is aimed at elementary school students, they might not have any idea what this prompt is really even talking about. They might hypothetically know what these things are, but it's it's really hard, especially when you're in first or second grade, to visualize and answer a question like this without the the experience of knowing what these things are. I could see students struggling with this. But to me, the the the bigger part is that's just really boring. Like, if I've read, like, 10 of these prompts, I don't care anymore. Who cares if swimming is really fun at what Alex thinks? What what bothers me about this is that is the data useful to know? Yeah. Of course, it's useful to know if, like, a kid can't read and they're in elementary school. Like, we wanna make sure kids can read. But this is not motivating kids to want to read to begin with. Like, you could easily issue a test like this between teacher and student, if you're a support specialist or something, where you can figure out this exact same data, but you could do it in a way where the kid actually enjoys doing the thing that they're being measured on, and you'd be helping their way through it. Right. This is this is not something, again, that could not just be assessed in the moment by a a teacher and a student. That that would take an inordinate amount of time to sort out. Right?

Speaker 2 Take a day to look at the data if you need to and then be actionable like the next day. I don't know what why this has to be on a map question. Now, when you brought up that cultural competency piece, it made me think of some questions that didn't actually make it in here because I think we had far more pages than ended up being in here and one was actually about a tollbooth. Do you remember this tollbooth question? It's on page 15. Oh, it? Oh, Well, that's great. There it is. So that's the same kind of idea. Right? The toll rates for crossing a bridge are $6.50 for a car and 10 for a truck. Now, you know, in Iowa, where we don't have a toll for anything, right, you might get to tenth grade and not have an understanding of what toll roads are if that's not something that you've experienced or or something that you know about. So so then in that question, it's not even asking you it's it's a, it's asking you, like, to do hardcore algebra with this. So so then, which of the following systems of equations yields the number of cars and the number of trucks that cross the bridge during the two hours? So you have to turn this concept that you don't understand into this mathematical equation. But meanwhile, you're wondering, what the hell is a toll rate? What's what's a toll bridge? I've never driven a truck across that. Why why is that there? So That question so that that question's from the SAT.

Speaker 1 For someone that can't see it, it's like it it's asking you to input these values into x and y to make the equation that you would use to solve this. So for example, like, x plus y equals 187, 6.5 x plus 10 y equals 1,338 divided by two. And it's one of those things where I do not know a person alive who would ever solve this question like this. Like, this might be mentally what you're doing, like, with a calculator in your head, but no one would write this out as x plus y and put it into some it feels like it's forcing this to make up for the fact that you had to learn this very specific standard to meet this test. It's not rooted in anything realistic. And, again, just like, if this is my tenth test in two weeks, and and and not to mention, this is the SAT, so I'm super anxious about it. Who cares? Like, who in their right mind would do who cares how many trucks and cars pass a toll road? Even if you had toll roads were, all I care about is do I have enough money to pay for the toll? I don't know. It's just it it it's a very silly question. What I wanted to bring up is the one right before that on page 14 from the star. I think this is the the fan favorite for us. I still can't pronounce this word. I still can't I'm gonna embarrass myself because I I don't remember either to bring it up. And we've looked it up several times. The the star is another one of the the plethora of tests that's used to gauge student understanding, for multiple times during the year, kinda like the map test, from Renaissance Learning. So this is a twelfth grade test. So this is supposed to be the hardest test that they issue. It's from English. Vic, do you know how to pronounce it? Did you look it up? I do because I just googled it. So the how is the word actually pronounced? Synecdoche. Okay. Synecdoche, which to be fair, Thomas, who is an English teacher on our staff, knew what this word was, Kind of. However, it took all three of us studying this question to figure out what the right answer is. So quiz for you at home. See if anyone knows this. It says synecdoche is a figure of speech in which a part of something is used to mean the whole of or conversely the whole for the part. One example would be if someone referred to her car as her wheels. Which is another example of synecdoche? One, the captain called all hands on deck. Two, the pen is mightier than the sword. Or three, the dawn raised her rosy red fingers. I feel like I'm on a game show. It it legitimately sounds it what's wild to me is this one gives you an example of what it is, which in theory is supposed to supply context.

Speaker 2 But I'm not gonna I know we discussed this before. I still do not know what the right answer is. I do not know which of those three is the right answer. And there's only three. That's the thing. We we it's not even like the traditional four, like, on some of them. Yeah. So, like, the first hurdle, obviously, is can you even pronounce this word correctly? And if you can't, is it that important knowing anyway?

Speaker 1 I mean, if you were pronouncing this, would literally read. That's just, like, how you would phonetically pronounce that. Or. Yeah.

Speaker 2 We're embarrassing ourselves now. But we we have college degrees. We are college educated, successful. Right? Like, successful in big scare quotes. Right? But, like, functional maybe is the correct word here. Right? Like like, have have functional adult lives. Like, I've got children. You know? Like, we both have homes and cars and, like, you know, we we manage these things in our lives. And, like, at no point in my life has has there been a gate that has been kept close to me because I have not understood synecdoche. You know? So I think, like, really, that's kind of the lens then that that we were thinking about with these questions. Somewhat of the absurdity of them aside, I mean, you can kinda quiz yourself at home or or or do do it yourself if you'd like to. But but we were really asking, yeah, that that what is this really assessing? You know, for students who might be unmotivated or disinterested, what would cause them just to you know, if they couldn't fifty fifty it and guess, just look at the prompt, feel overwhelmed, fill in a bubble, and move on. And then how accurate then is the information that we're getting? Is that actionable in the first place? Because if we're not acting on whether or not kids know, we're acting on whether or not they care to answer the question in the first place. So, you know, we're we're responding to the wrong thing. Right. Also, we were acting on if kids knew, then we wouldn't receive almost the vast majority of these test scores

Speaker 1 partway through the following year. Almost all of these tests, you don't get the data anyway. And even when you do, so much time has passed and that usually teachers have changed that there's no way for you to even act on the data. The data just serves to showcase typically the problem, more than it does to actually offer any kind of solution for you outside of purchasing more test prep materials. And before maybe we wrap up this section, I'll kind of meet with an anecdote here for my But really quick, I I'm pretty confident, by the way, the answer is three. Dawn raised her rosy red fingers because the trivia side of me wants to have that answer right,

Speaker 2 but I'm not positive. That's okay. So somebody at home listening right now, send us an email, chris@humanrestorationproject.org. Don't send it to me. Send it to Chris. But in my in my own experience, I remember, I don't know, I had been in my district maybe three years, maybe four years. So I was I was fairly, you know, new to the district, and at that point, fairly new to the profession as well. And and I can just remember us being called into a meeting where we were meeting with somebody from the Iowa assessment team. Like, they were part of the test development team or part of the research team, something like that. And they were gonna come and talk to us about the social studies portion of the, like, then the Iowa assessment in in its form maybe back in the mid twenty teens or something like that. Right? And I can just remember sitting in that meeting and kind of just being excited at first because it's like, oh, we're gonna pull back the curtain and kinda see, you know, how the sausage gets made on on this kind of stuff, and maybe that'll help us with the responses. I mean, our our scores weren't bad to begin with anyway, but the more that we went into the actual test, which as educators was the first time that I had actually gotten to see the test. They handed us the test booklet. I still have it. They handed us the test booklet, and we went through the whole exam. So, like, this is what students take. Right? Like, what are what's your response? How can we improve our teaching kind of in response to what we know now? And the answer was, we're not gonna change a damn thing because what we're doing is head and shoulders above anything that this test expects our kids to do. Right? We're getting kids to, you know, write, write letters to to local elected officials, right, and and kind of learn about that process. We're having them go to school board meetings. We're, right, having them do, experiential kind of project based learning stuff. We're getting them to think and read, like a historian. You know? We're getting them to, you know, understand, the the impact of economic thinking on everyday decision making and all these other kinds of things. And the test wanted to try and reduce that whole experience into a series of the most inane, the most, irrelevant, questions that I could ever imagine. So I left the meeting just being disgusted. And the decision that we made as a department was, well, we're not gonna do anything in response to this because it's it's not worth responding to. And we we never have. We've never been responsive to these tests, and we're not like a big tested subject, so it hasn't mattered. But that's at the end of the day, it's like, does is what you're doing in your classroom right by students or in front of you, or are we trying to impress, right, the people who are furthest from students with the researchers for the for your state assessment and to try and, you know, to try and juke the stats a little bit to show show progress in those areas? Or are you teaching the kids in front of me? There's so much cognitive dissonance here because

Speaker 1 what a test manufacturer and district will tell you is that in theory, you are not supposed to be preparing students for the test. These tests are meant to just see whether or not teaching is effective, and you're not supposed to cater your content towards the test. You're supposed cater your content towards the standards which are then addressed by the test. However, at the exact same time, every district does this. Every district cares about their test scores because typically your funding is associated with your test scores. And every test company, either themselves or through some kind of partner, offers resources that do increase traditionally test scores. And the way that they do that is by teaching students exactly how to answer the questions, not because they necessarily understand the content, but because they know these very specific clues and hints and types of questions that this content asked. And the same reason why I'm sure everyone had a friend in high school who was great at taking tests, but ultimately had no idea what was going on. Because the tests only measure a very, very, very specific subset of knowledge and don't accurately measure learners at all. I myself am a terrible test taker, but I think I can produce things and do things quite well. And and and and also, we we highlight on here. I don't wanna get too caught up on it, but we we highlight on here. I mean, we there's a lot of data about standardized testing and the problems that cause. There's anxiety, sleep deprivation, fear, assumed cultural knowledge. It it attracts from creativity. Mean, the modern day tests that are created by AI in order to speed up the process, not only are the multiple choice test questions, okay, whatever. I mean, obviously, they are quickly graded. But now they do writing through AI, and there have been multiple circumstances where folks have just written nonsense. And the test is like, okay. It's a five out of five because you use this certain amount of words and these certain sentences. No one actually cares what you wrote. So if you have an original idea or you're a creative writer, who cares? Did you use the word synecdoche in your your paper? In which case, that's a five out of five. And just the whole legal apparatus that surrounds this, even pulling these questions and putting them on this document, how do you I had to use previous versions or things from test press to prep companies. I can't actually pull up the test myself and take screenshots of it because that would be illegal. Test companies own the the rights and don't allow folks to share the data. I mean, I'm sure everyone has probably given or been around, like, giving, the SAT, ACT, etcetera, where you have to read the ridiculous prompt. That's like, if you, share this data on on social media, we're gonna come after you, and you could be, like, sued. Like, what or, like, your test results will be deemed inaccurate. It's like, why are we making tests that we can't prepare for by looking at the test first? That meant to me makes no sense. The irony

Speaker 2 of that leaving that meeting in this you're you you saying teach the test prompted this. The irony of that meeting, and this has kind of become a a running joke in our department for some time, is we were advised by the Iowa assessment person to teach oh, what was their like? Yes. Teach to the test because that's gonna, you know, get you the results that you want. Teach to the stuff that's on the test. You just can't teach the test. So so we're like, doesn't isn't that just more difficult? Like, if I know what's gonna be on the test, why not just give them the same kinds of questions and things that are gonna be on there? Like, if that's what matters, why wouldn't you just spend all of your time doing the things that are gonna improve those test scores instead of, you know, to your bigger point, you know, if you have some standards that you can draw from a broad range of of skills, abilities, content, time periods, like thinking from a social studies background. Why would you not do that? But those are are gonna be things that may or may not reap rewards for you on the test later on. I mean, I teach an AP class. This is the world that I live in is, you know, this constant tension between, you know, giving what giving my students what I think they they deserve out of that classroom experience and teaching to a single three and a half hour exam in May. You know? And that's that's the tension that we're at at a systemic level. You know? K through 12 is teachers. Right? Are you teaching kids to three hundred and sixty days of the of the year, you know, or are you teaching to those five days that they're gonna be, you know, trapped in map tests or whatever the case might be. Yeah. And and also just the fact that the way that you fix this isn't by attempting to improve the test.

Speaker 1 We've tried that for decades. Or, honestly, even going back to, like, the old Thorndike Dewey debates back in the day. Back in the, what, that, thirties? People were debating whether or not multiple choice tests could be valid measures of intelligence. Honestly, I I don't think it's worth the effort. We have tried to make tests more fair, more equitable, make them make more sense, make them more culturally responsive, and it doesn't work. Because ultimately, you can't take an entire year's worth of learning in an entirely different context, in entirely different communities and say, alright. Take this test and then figure out whether or not they learned something. It doesn't make any sense. There are other ways of measuring student growth. I I think both of us would probably advocate for, like, portfolios or showcasing of learning, and you could have perhaps a state board that reviews those things to see if teachers are on the right track and if kids are still reading and writing. It's not like they're we should get rid of all oversight. It's just that that oversight should be working in a different way to support learners and support teachers instead of paying them against each other in a competitive testing environment, especially when well, one, the testing, obviously, hurts students from low income backgrounds because, typically, funding is attached to this, and students from low income backgrounds face so many different challenges that they typically don't do well in standardized testing. But two, the the overall narrative here, if we were to really go back, is that these tests are rooted in the idea of racist systems of practices. Like, they're they're doing what they were designed to do. The whole idea of meritocracy through a multiple choice test, through the SAT and its origins, was designed to keep students of color and poor students out of higher education and out of potential opportunities for them. And that's documented. That's that's evidenced.

Speaker 2 That is that is a real thing. We we know this. That's in the history. I mean, yeah, like like, it it goes back, I mean, to to Alfred Binet via Carl Brigham, who, I mean, was was a eugenicist at his core. And Alfred Binet, you know, of course, with the with those IQ tests and the and those those military proficiency exams around the turn of the century too. And then, of course, I think lockstep with that eugenics movement with, you know, the development of racist science that validated in in their minds Jim Crow and segregation and and every other discriminatory policy imaginable. And, you know, at the risk of of Poe's law here, right, those things got exported to certain countries, and they abused them in in horrific ways, of course, in the nineteen thirties and forties. And we still use those exams at the same time. Right? They were they were deliberately designed to keep, you know, Jews out of Ivy League schools back in back in the day. And in the nineteen fifties and sixties and stuff, there actually was, like, huge as as desegregation led to more college entrance and stuff for for, you know, people of color in in in marginalized communities. The the the score percent the score averages went down, right, as as people from disproportionately underserved communities took these exams. And, right, this led to a a little crisis in in the s eighteen and the college board and, like, led to recentering has led to a lot of things over the years to basically right. Again, from their perspective, if it to juke the stats so you can keep this certain groups of people, out under the a more objective basis than just discriminate against them based on race, religion, socioeconomic status. But, we we know that these things are just proxies for all of those. One of the I I don't know if it's a Jack Schneider, quote, something that he maybe tweeted out sometime during the pandemic, but, he says, like, well, we know if the argument is that we need these standardized test data to know which communities are underserved so that way we can better serve them, we already have that information. It's called their ZIP code. And and there's no new information that these tests can provide that isn't already delivered somewhere else. So save the money that we would be having on be using to, what, like, take the tests, analyze the tests, make these things actionable, buy the the accelerated learning software, and just give it to schools, right, to serve, to to to provide free free, lunches, and breakfasts and meals and to make before and after school programs and to, build better schools in better conditions, like and build air conditioning, build, right, the things that are gonna serve them in this pandemic. So yeah. So so there is, like, a huge controversial history with that that that, for some reason, just gets thrown down the memory hole,

Speaker 1 and and we never really think about it. I mean, it's it's goes back to even, like, picking up, like, a Jonathan Kozol book from the the seventies where we see this today. Anyone who's even vaguely familiar with public education in The United States knows that if you go to a school that typically serves students of low socioeconomic status, those schools tend to be more prison like. They are hypervigilant about preparing students for tests. They tend to be more worksheet driven, more lockstep. They sadly do things like lock the bathrooms and let you make you be silent in the hallways. They're they're super gross and dehumanizing. Meanwhile, a lot a lot of politicians and some rich folks send their kids to schools where they do whatever they want, and there's, like, a public beautiful library and everything's experiential. And their test scores still end up being higher than the students who spent the entire year just preparing for that test. Because, ultimately, test preparation and the way that we do test prep and the way that we focus on tests is a better indicator of how much wealth and how much access you have to, I guess, like, quality living circumstance. I don't if that's the best way to put it. I I think you know what I'm saying. And I I think the best way to highlight that, it's in our booklet we wrote about wrote about on page 19, is but what happens if you get rid of the test? As of the 2122 school year, more than 1,500 US four year colleges are not going to require ACT and SAT results. That's a lot. That's 65% of all four year institutions in The United States. Because during the pandemic, people dropped tests and society did not collapse. And what's interesting to me is that in 2018, so this was before the pandemic, this was study there was a research study that went out, and they looked at certain certain colleges and those that were test optional, and they, saw, well, what ramifications does this has does this have if schools go test optional? To be clear, test optional means you can still submit test scores, but you can choose not to submit test scores. And I'll get to some caveats here in a second, but in a study, a fourth of all applicants didn't submit any scores. And as a result, obviously, more people applied. However, interestingly enough, you saw higher numbers of black and Latinx students apply and be admitted. What they found was is that students students of color were more likely not to submit scores. So 35% of black students did not submit scores in comparison to 18% of white students. And as you saw higher numbers of black and Latinx students be admitted, first year grades were slightly lower for those who did not submit scores, but everyone graduated at the exact same rate or higher rates than their test submitting peers, as in over time it all equated out and the graduation rate was roughly the same or higher than the folks that did submit test scores. There is a caveat because test optional still doesn't work perfectly because those that submit good test scores are still given preferential treatment over those who choose not to submit test scores. But I think in practice, this shows that I I mean, if I was a college admissions counselor, I'd be like, well, what the heck? If if students can get in and have the exact same graduation rate or higher graduation rate by not submitting a score, what does that mean for the test scores in general? Or a better question, you know, last year, during the pandemic, a ton of schools were test optional. If all of those schools and all of those students ultimately end up having the same graduation rate or higher and perform the same as or better their peers, why do any of this? Like, what's the purpose of all this? It it doesn't make any sense to me. I I think especially if you consider

Speaker 2 you know, we focus so much on that admissions process, you know, and and getting kids to, you know, to the the drive out to the parking lot at at the college campus. And then we we then we forget about them, and we and we don't care. We don't realize, like, to what extent there actually is, a dropout crisis anyway. And if you look at the cause the reasons that cause students to drop out, it's it's it's overwhelmingly not because they it's not because of scores. It's not because they didn't have background knowledge. It's not because, you know, they weren't well prepared or whatever. It's because of cost. It's because it's expensive. It's because they have to, you know, take care of family members. So so it's really about, like, those community support aspects too. So, again, like, the test score as college prep conversation is like another non sequitur like squirrel because then it gets us focusing on content and standards and, you know, the the the the things that we need to micromanage in the classroom rather than, like, building an off ramp from kids to move from one environment successfully into another and then and then stay there, right, and and be successful in those environments. One thing that's really interesting that that I wrote about just pre pandemic, it was, like, March early March twenty twenty while this is all coming out, was was about GREXIT. And this was the GRE exit where where increasingly, schools, particularly, like, in in what we call, like, the the hard sciences, you know, like, physics and and some of those things too, they they were going through hundreds of programs had been dropping, GRE requirements. And what they found was that, that GRE didn't correlate at all to the quality or the and the success and the completion of, like, their doctoral programs in these areas, but that GRE requirements actually did select against underrepresented groups. So it's like, okay. There's no correlation between the quality, but it keeps people from underrepresented groups out. What function does it serve other than, like, as a huge gatekeeper to, you know, people who who traditionally have not been able to to to be entered into those programs? It just perpetuates those same biases.

Speaker 1 It it's almost like there is a system in The United States that could be analyzed about how different legal practices and practices of testing and meritocracy exclude certain groups intentionally. I wish there I wish there was a thing that folks learned about that that dealt with that. You should you should develop a theory around that perhaps. Maybe about critical race or something. I don't know. Speaking speaking of research too, I I I I'm not gonna go through this whole page. We have a page in the handbook that goes through just the amount of research that does not support standardized testing that there's studies that come from the eighties, the nineties, 2000, two thousand tens, even this year about standardized testing. And I think almost every educator knows some of these, but I did wanna highlight a few just because I forget just how big of an impact it is. Like, there's obvious things that I just like, oh, right. This is a thing. We we should probably care about this. Like, fact that our high school does not have a gym class because of a focus on college readiness. The students just don't have gym, so they don't or phys ed or any kind of outdoor activity outside of what we integrate into our own classrooms. The fact that a lot of teachers will do, like, the quote, unquote real curriculum after the test occurs. All the fun stuff happens after the test. The fact that educators and students just face so much pressure from this. You see a lot of teachers getting in a serious amount of trouble because they're worried that they're gonna lose their jobs or have a salary cut, so they manipulate test scores. I I I don't understand. Like, why in the world would you develop a system where fee people feel like they need to cheat when it's just a measure of their knowledge? It's not meant to be competitive. The the fact that the results don't add up, we'll talk about that here in just a second with the the learning loss narrative and the McKinsey report. Engagement, it it is linked to pushing students out of school. It's hyper boring. No one likes doing test prep. There are very few students who enjoy taking tests. And if they enjoy taking tests, they can still take them, but there's different ways of doing that. They can get really good at trivia. Enjoy it there. The equity issue, I think we've exhausted that conversation just the the way that this selects against students of color is overwhelming. The data is so clear that it's disgusting to me that the conversation even has to occur. Again, it's so obvious. And then finally, just when you focus on preparing for the test, you lose out on so much. I think about all the different times where, like, students have wanted to explore something that I thought was super cool, even in design because I have a standardized test and design. And I'm like, well, I'd really like to do that. But if I focus on that right now, that's gonna take a month, and there's no way you'll be able to get that part of that test done. Like, we we really do have to cover this to make sure that test is there. And always in the back of your mind, you can't lead a classroom that is truly based off a student interest if constantly you have to shift and pull towards meeting ultra specific narratives and ultra specific questions that will appear on a test. It's not the the tests are never broad enough to encompass just generalized learning during the course of a year. Yeah. Right right when you right when you've gotten past, like, the the initial

Speaker 2 sort of bite. Right as kids begin to, like, grasp a concept or begin to get excited about it and wanna, like, dig and dive in as deep as they can on that thing, you're moving right along. So, you know, the the even the concept of of learning loss kinda takes the content coverage narrative of this thing too, right, and says that, like, you know, we can break these into these discrete bits of of information that need to be delivered on x, y, and z day. And so if you miss so many days, right, you're missing these discrete bits of information instead of, right, losing out on the process, right, and not being able to practice, you know, the the kinds of thinking skills that, you know, otherwise competent, successful adults might be able to practice. No. You're missing the discrete bits of information that are gonna be used against you and your and your schooling institutions later on. Exactly.

Speaker 1 So I realize this is gonna end up being a real long podcast, but that's alright. Because I honestly think that every part is just so hyper valid. So hopefully people are right at the pause and take a break through this thing. Because this next part to me is the most interesting. I feel like everything that we've covered up until this point, if someone's been involved in education, they probably realize, yeah, this is a problem. And, hopefully, we're providing more evidence and more tools to communicate that. But the next section deals with the connection of the standardized testing industry and this new learning loss narrative, the one that we're currently involved in right now. This was the part that was done first. Before we did anything else, this this one was mostly me, sitting down, and it's the the Charlie Day meme from Always Sunny. What's the the the the woman's name? Who is Pepe Sylvia? Yeah. Who is Pepe Sylvia? It's that that that meme with a conspiracy theory. So I I literally made a graphic showcasing all of the weird connections between the testing industry, testing companies, and the reports that highlight this idea of learning loss. And here's here's the gist of it. So you have, like, Pearson, Savas, College Board, NWEA, Curriculum Associates, I Ready, Illuminate. These are all testing companies. Some of them are huge scare quotes, nonprofits, and some of them are corporations that generate an absurd amount of revenue. College Board alone generates a billion dollars a year, over a billion. Technically, could make a billion dollars and still be a nonprofit somehow. Some I don't know. I'm not really sure how that works. And pay your executives a lot of money, FYI. You have all just an absurd amount of money. And pretty much every single one of these organizations sells services that address the concept of learning loss. So obviously, all these testing companies offer textbooks, resources, etcetera, that you pay for in order to do better at their test. But now all of these testing companies are selling explicit services to address learning loss, that this is this major problem where, hey. You need to buy this special curriculum. I did this at the beginning of the year. Our school has done this. In fact, the the federal government has required that 20% of school budgets be dedicated towards addressing what was lost in the COVID year, which is highly connected to this learning loss narrative. So I would not be surprised if pretty much everyone listening has had to purchase some kind of testing software, or do increased testing as a result of this learning loss narrative. So they're all offering these services, making them even more money. But where this comes from so, like, the way that we know that we need to address learning loss is mostly through three reports, the big one being the McKinsey report. So tying it back together with that that global consulting for McKinsey. McKinsey published a report in 2020. They've actually established multiple reports now saying that, hey. We have this major problem. As a result of students leaving school during the pandemic year and schools shutting down and going virtual, even though they don't always say that, students have a lot of academic needs. So we need to make sure that we address this. So McKinsey pulls data from, Illuminate Education, I Ready, and a couple others to say, hey. Look. Test scores are going down. Then all of the testing companies point to the McKinsey report and say, man, this is terrible. Look at this. McKinsey says that all the test scores are going down. They analyzed all of our data. So therefore, you need to spend $25,000 this year to purchase something to address this. But here's where it gets interesting. The actual data, even at taken at face value, if we ignore the fact that the test itself doesn't mean anything, that it's irrelevant, that the the the test, questions that are being asked are absurd. The data doesn't add up. So we dove into the actual reports. We looked at what these reports said, how much learning was lost, quote unquote, And here's a few examples. So, you have the McKinsey report, the Annenberg report, and the Illuminate education report. Often in articles, you'll see the McKinsey report cited, and now they started citing each other. So, like, Annenberg will cite McKinsey. McKinsey will cite Illuminate, etcetera. Those are the only three major ones to to our knowledge. In the Illuminate report, there is this chart, which is screenshot. It's on page, well, it's in the learning loss section. Actually just not page numbers. I think it's page 24. Here, there is a figure which shows grade one a math scores, which is an adaptive test that shows students' math level. The y axis is the predicted a math score. The x axis is a student's free and reduced lunch amount, and the lines represent the 2018, 2019. And then the other one is the 2019 or 2020 school year. So for example, if you had a student who was on entirely free and reduced lunch in an urban school environment and they received a predicted score of, let's say, a 197, the previous year, they received a 200, which if if anyone is having trouble visualizing that, that's a three point difference. So from a quote, unquote normal school year, the score was 200. To the year where school shut down and a global pandemic shut us all down, their score was a one ninety seven. That, when you look at all the data on this entire report, is the largest gap. As in the largest gap that is used on this entire report is three points, and that is three points on an over 200 scale. So it's not even 3%. We're talking, like, minute percentage points. In fact, if you were using an adaptive test model, we're talking about probably one or two questions, difference. There's a similar, one in here where it's it's third grade aide reading. It's a wider scale. So the the scale is actually 510 points roughly. There was a one to two point drop year to year. So one to two points on a roughly 510 scale. That's it. And the the most interesting part about this whole thing, this is the one that blew my mind. This is the one that I screenshot. I tossed in our Discord. I'm like, yo. Can you believe what this is written in here? I'll just read it adverbatim. So this company, which is looking at this data saying, look how terrible it says. Look at this data. They also say, like, hey. We wanna bring up the fact that, MAP and STAR tests maybe didn't find the same thing that we did, in their reports. They write, in the comparison with MAP math results, the drop in the median national percentile is very similar to FastBridge results, with differences ranging from zero to three percentile points. It is worth noting that when analyzing learning loss in FastBridge by using changes in mean achievement scale scores across years, the estimated learning loss is consistently similar than the loss derived from ROI differences. This may imply that MAP results underestimate true learning loss. As in, they looked at another report from the MAP and the NWEA and saw they said that they might have not lost any points at all. So, actually, they did some advanced formula wrong, and we found that, actually, there might have been a three point drop. And it's like, wait. What? Like, this this to me is so wild. Like, it's still it's still, riles me up. Like, we're we're freaking out about three points on a 200 scale,

Speaker 2 aka one or two questions on an adaptive test. That's what all this money is about. This this entire thing is rooted on this. This is the basis. What is amazing, if you're looking at that chart of the AMATH scores, it is so deceptively designed because that y axis you you would think that the spread in that y axis would be 50 points or something like that. Right? But it goes from one ninety to one ninety five to 200, two zero five, and two ten over about the space of two inches to so that way, they can get the little decline on their curve so you can actually visualize that. If that was a scale that went from zero to two ten, it would be imperceptible. Imperceptible decline in that thing. Now, like, if we were talking about if we were talking about getting a rocket to the moon, right, like, 3% would be probably meaningful. That would be that would be valid. That would be, like, reliable information that we would need to act to correct. But given all of the other, right, doubts about the utility of those three points and what those one to two questions are asking, how they're asking them, are they things that are worth responding to in the first place? Like, since that is not an exact science, right, since that's very much a value driven proposition, and and oftentimes, we don't have access to what those those questions are, how they're, you know, phrased, what it is exactly that they're asking, and in what way or what might be confusing for students. We just have to take it on face value from, from, wherever, from FastBridge that this this drop is not only, meaningful, but it's important and worth acting on. And again, and I'm gonna keep ringing this bell, right, the fact is that we're seeding, like, we're giving up agency. We're giving up power to the people furthest away from the classrooms to make decisions, right, on on our behalf based on data that they control and that we have to pay for, but that they expect us in the classroom to enact, changes in curriculum and assessments to answer, not to the kids in front of us, not to parents in the community, right, not to the goals of the school district or whatever else, but to their,

Speaker 1 their test. In my opinion, this is intentionally hyper complex. I feel like this is all very much buried, both from the design of the graphs to how it's written, to how you find this data. Because if you open up this report and you read the summary, the very beginning of the Illuminate Education report starts talking about how we need to make up multiple months of lost time. They are translating this percentile drop to literally one to three months of learning, which is not rooted on anything. That is not based off an equation. That is not based off of some in-depth study. That is Illuminate Education's opinion of what they describe as negligible to moderate loss, which there's something very funny to me about the idea that one month of learning was considered negligible. Like, to me, one month of learning is a lot if we were gonna measure it in time, which is also absurd. Even if, again, we take the data at face value, you're telling me that in the face of a global pandemic that shut down schools where kids learned online and teacher were scrambling to figure out what the heck to do. I'm sure everyone remembers, like, March slash April when school shut down, you're like, what the heck is this? That in that whole time period, kids maybe missed one or two more questions, and that that's the problem. Like, that would be assumed. That doesn't necessarily mean that, like, we need to be frantic. Speaking of data that is very questionable in in how it's presented, I also wanted to call to attention the McKinsey report, which is if you read an article about learning loss, again, you will if you, we've we've been joking about viewing, like, your control f and searching McKinsey, It will be in there. Everyone references McKinsey about how terrible learning loss is. Their report pulls from the map, which is the one that most students take, the NWEA map. Essentially, they highlight this one graph in here that shows significant learning loss or risk for significant learning loss. And it's a chart that shows four different lines going across. And the lines are meant to predict a scenario where it's just normal learning, where students are just in class and then, like, the the chart goes up and then summer hits and then starts to to fall a little bit. The second scenario is 52% learning growth through quality remote instruction. So someone who wrote this report was like, if you do high quality virtual instruction, then students learn at 52% starting in March, which, again, that that's not based off of anything. That is just an assumption. It's an estimate an estimation. So then on that little chart, it's a few points lower. It goes up a little bit and then goes back down. The third line is that no growth or loss occurs at all. So this is low quality remote instruction as in you're treading water. Students aren't learning anything, but they're not losing anything either, which I'm assuming is, like, worksheets provided online. I don't know that these these terms, high quality and low quality instruction are never defined. And then finally, no instruction occurs at all. So there is a line on here that says, hey. School's closed, and they never reopened. And the line just goes straight down, and no one learns anything. Ignoring the fact that that assumes that you learn everything in school, so so much of this is just an entire assumption. Like, who why 52%? Like, why? Like, where why would we think that it's 52%? And, again, even taken at face value, I would argue that the vast majority of schools in The United States either taught, let's say, with high quality or low quality materials. I don't know what that means again virtually, but the difference is a two thirty RIT score, which is like an adaptive score, learning occurring. We define what RIT scores is. I don't wanna go into that. Two thirty. And the lowest one on here is, like, roughly a two nineteen. So if we average that, let's say, or even did, a low average, at most, they lost 10 points on a over 200 scale, because the writ, again, is adaptive. We're talking, again, a difference between, let's say, three or four questions, again, based off entirely unknown data. That is being translated into saying students lost multiple months of learning because they missed a few questions on the test that they didn't normally before.

Speaker 2 What is what is super interesting too is is you're you're speaking of the data, of course, that goes from, like, typical in person and then looks at the the various qualities of remote instruction. Right? Within and that's, like, within a 10 range. But even even the lowest one that assumes that there is no instruction, like kids a kid just falls off the grid, right, shuts off their brain for for what? Like, thirteen months is what this chart basically goes through, right, or longer than that. Even then, it goes from what looks like about a two twenty two to, like, a two zero eight. So it's, again, assuming that you've just straight up been gone for a dozen, you know, months of of instruction, your scores are gonna go down, you know, 15 points. And that, I think, you know, I may I don't wanna jump the gun on things, but it's like, man, if the only thing that we're doing in schools is is is valued by the way that it's measured on tests, that would you know, this would be frightening in in how little it actually impacts these scores. But, right, as we begin to ask that question about, like, what are we actually losing? Like, what function should schools serve in growing like little human beings? You know, is it is it the case that those things can be accurately distilled down into that assessment score? Or if you miss a month of school, right, are you missing out on something else that is just not being captured? Right? That's the value of school. Those things are not being captured on, on those assessment scores. And and we haven't yet really figured out a way to and I'm not trying to give anyone any ideas, but evaluate, right, like, kind of systematize those things. And this is this has been Susan Engel's point, you know, a a developmental psychologist and and former HRP board member for the quick plug, doctor Susan Engel. She would make this point in a heartbeat. Right? Like, we have ways of of evaluating curiosity. Right? We of of figuring out affective, you know, ways that students feel about school, right, and feel about their time in those buildings and how we can how can we measure those qualitatively and quantitatively and use that data to improve those measures. But that's not data that the learning loss folks have access to. Right? That's not anything that McKinsey cares about. All they're doing is looking through this very narrow slit at the entire functions of, of the eight hour school day plus, right, plus or minus two hours on either end of that thing, or more, and then saying, nope. 10 points on on your on your MAP test using RIT scores. So, yeah, like, what are we actually losing in that time is is gonna have to be a conversation that we have as part of this too. Yeah. And the only other report I really wanted to draw data on because this was the the funniest one to me,

Speaker 1 is so McKinsey pulls from the NWEA data. NWEA, the map creators, also have their own, like, publication, which features the exact same data. It it's where the chart comes from the McKinsey report. What's interesting about this one is the one that we link to basically shows, like, again, like, very, very, very minor differences in scores over time, basically, the exact same data. But since we published this, there have been updated, charts. So a lot of these folks have walked back, previous statements. They've also rebranded, so it's no longer learning loss. It's unfinished learning, which sounds a lot nicer, but it's really rooted on the exact same thing. And the data didn't end up being nearly as bad as they predicted it was going to be. It actually was much higher. In some cases, we looked at one where the scores were higher than the previous year. So, even even in the state of a global pandemic, students actually had higher test scores than they did the previous year, which really makes you question, like, why are we focused so much on this? The the whole point that we're trying to make here is that there's there's much ado about nothing. Right? It's we're talking about differences that are so small, but yet have massive ramifications because of how this has been both, narratively driven and, to a sense, propagandized. The Department of Education has been highly rooted in this. Again, not to sound conspiratorial, but most of these testing companies lobby, the government and say, hey. You know, this is an equity problem. Students need to do better. The achievement gap is getting worse, and that it's sold on this idea of we need to help students that are most vulnerable. But as we know as educators, focusing on standardized testing does not help students who are most vulnerable, that it actually hurts students that are there. And we have a a large research database that that demonstrates that in my opinion. So earlier, I had mentioned operation reverse the loss, which is the Department of Education's new initiatives to invest in solutions to the the the learning loss problem. Essentially, order to get money through the American Rescue Act, states have to invest 20% of their funding to explicitly address issues of academic loss. And you can find the link in our handbook. It's from future ed. They've documented what every single state has done and where their money has gone. You'll find things in there like summer reading and math camps, intensive programs, a lot of adaptive learning software, promoting concepts like learning acceleration, creating new screening measures, so taking even more tests. I know we're taking an additional MAP test this year, so now the kids have to take even more tests. PD programs focused on addressing learning loss. So although there are states investing in social emotional well-being and trauma informed education, unsurprisingly, it seems like the vast majority of funding is going towards this explicit problem. And I I can't say this enough because I feel like like I'm losing my mind, but we're talking about one, two, or three questions on a test. It it would maybe be different if we were saying, like, these kids failed half the test and it's, like, a a national panic, but it's not. This is a very minor problem. And I think, honestly, it's just rude to making more money for testing companies. So so, basically, I I think what we're we're getting at as human restoration project when it gets to this is, one, we're connecting the financial industry to the learning loss narrative. We are trying to provide educators with the knowledge and the tools to recognize that they are not they are not going crazy. This is this is not rude, in fact. This is not nearly as meaningful as we say it is. The data itself, that we're pulling from doesn't even make any sense. But when we're focused on things that were lost during the pandemic I mean, at the time of writing this report, we were at six hundred thousand lives. I'm I believe we're much higher now, due to COVID. People have died. You know, that affects personal relationships, our mental health. Learning online is for some an improvement, but for many, it was a struggle, especially when it comes to social connections and and all those things that kids like to do. We did our a hundred days of conversations initiative, where we collected and we're about to publish the data of a 100 and, I think, a 107 different conversations with young people and adults. And almost universally, students said, hey. What I didn't like about the pandemic was I didn't get to go to school dances. I didn't get to hang off my friends. I didn't get to, like, take my driver's test to get my license. I didn't get to eat out anymore. Like, things that are the social experience of being someone in school. That is the thing that students struggled with. They didn't really talk about, hey. It was really hard to take the test this year, or I really don't get math anymore. Math is just an enigma to me. It's probably the same as it always has been. So if we are focusing instead on at the start of the school year, on saying, oh my goodness. Test scores have gone down. We need to fix this learning loss. Here's the PD I just did. Here's all this test prep material. We are ironically causing the same problem that that students are saying that they have, which is loss of social connection because those activities destroy that.

Speaker 2 Yeah. I think one thing that I have definitely noticed in my students this year, especially my seniors, you know, my sophomores, I I don't know if it's a resilience thing or or whatever. They've just they've been as disrupted. Right? But my seniors, I think, heading into their senior year, they're gonna be grateful to be done with the herky jerky, you know, all of this stuff. They're they're kind of they're kind of fed up with it. And and the thing that I see in them is not like like they don't know as much, right, or they weren't able to cover as much in their US history class last year, and now they're unprepared. But their perspective on how school should be and their their role as learners, I think, has been changed for the worse. I I think that this the class of seniors that I have, I I I love them, but they have a very transactional view of how education should work in part because we really failed to invest in, you know, call it high quality distance learning or digital learning or whatever. We've had digital learnings forever. Right? And it has been, it can be high quality if you pair it with, a coherent digital pedagogy. And and that was the failure. Okay? We tried to we tried we tried to, reinvent school online and then bring it into kids' bedrooms. You know? And and those traditional power structures, those, you know, those behaviorist systems, etcetera, just just didn't translate. And so the ways that we tried to mitigate it not working was to double down on the transactional kind of model. Right? Like school as this checklist. You check-in. You get these points. You, you know, you do whatever. And and kids, I I think they still have that view a little bit because we didn't invest in, right, the digital pedagogy that meets them in their bedroom and sees that as an opportunity, right, and doesn't see it as a barrier to to schooling, but views, you know, what can we do today with you that moves the ball forward with something in your life. Right? And then maybe how can we make math a part of that? Right? But start with students first. Now when I I I think I was at least marginally successful in doing this with my remote students who now I see in person in the building this year. And it was pretty incredible because it's the first time I've been able to see a lot of them face to face. But, you know, those are some of the strongest relationships that I was able to build last year because the in person environment came with so many barriers and hurdles and sanitation policies and so much anxiety around it already that the online our my online cohort just didn't have. So so those are kids that that feel free, you know, even now coming to me in person and talking about things, talking about their experiences last year, really as a meaningful and impactful thing and how we we grew as a community of people all experiencing this trauma together. And that's a thing too that I think we failed to really meet in this transition back into school is that that learning loss narrative can end up being a self fulfilling prophecy. If all students end up hearing is that, you know, that that online learning was bad, that I didn't learn what I I didn't do what I should have done, that, you know, school failed, that etcetera, etcetera, etcetera. Like, they they adapt that. But, again, if we had if we had flipped that to be something more positive and we could say, like, hey. How did we, you know, how did we do how did we survive during this time when a lot of people did not? How can we come back and, again, you know, borrow Yong Zhao's framing of this, like, how can we take the lessons of the pandemic and build back better instead of instead of, you know, beat kids over to the head with this deficit narrative from day one? I mean, the purpose of this is to

Speaker 1 to to contextualize this, to understand what's going on because there is an ability to push back against it. We'll talk about the how in a lot more detail in the next podcast, but understanding this stuff gives you the tools to recognize that you can push back and there is a plausible argument on why we're doing this. It's not just like gut instinct. It's real. Just like all the other things involving progressive education, like ungrading and restorative justice, etcetera. It's not just it feels bad. There is something actually wrong with the current system. That's what we're trying to contextualize here. So the last two things I want I wanted to highlight is how the narrative and framing can affect how we treat students and how they feel. The first one I wanted to call out is the stereotype threat, which is typically attributed to Claude Steele. He wrote the book, Whistling Pivaldi, which is an incredible book. The stereotype threat is essentially the risk of confirming your own negative stereotype because you're perceived to have a lack of ability. So for example, one of the first studies in that book talks about, that they brought in, many students, I think it's 54 students with strong ACT, SAT math scores. They were in really high calculus course. These are students who test well in math. That is their skill set. They had two groups. They had a group that they told the students, hey, this might show gender differences in math ability. Then they told the other group, hey, this test is explicitly designed not to show gender differences in math ability. And for students who were told, hey, there's a there might be a gender difference here, overwhelmingly, male students did better. But when the group was told explicitly, hey, this test does not show any gender differences designed not to, it was roughly the same between men and women. And they repeated this experiment over and over and came to the conclusion that female students, when told this measured gender, difference, there's a perceived cultural norm, that men are better at some subjects than women, which is obviously not true. But when that cultural norm is, kind of pitted against you, you start to feel a lot of anxiety. You feel like you might need to overthink questions. You feel like you're not gonna perform well. You feel like, oh, this test might be, like, not designed for me, etcetera, and you psychologically do worse. And that's shown when you when you get rid of that that explicit narrative. They did the exact same thing. This book is so interesting, by the way. I love this book. They did the exact same thing with white and black athletes in terms of this concept of sports intelligence through golf. I'll just briefly summarize this one. They they basically brought in different golf players, like high level golf players who were black and white, and they gave them different ways of taking this test. And they they told, like, one group, hey. This measures natural ability. Hey. This measures sports intelligence. They had one group take a, like, a a demographics question before they did the test. They had another group that it was just, like, general sports performance, and they didn't collect the demographic data until after. People mirrored the cultural stereotype. So, like, when, for example, they were told that this measure sports intelligence, white athletes had better scores than black athletes, because of, like, the perceived verbiage of sports intelligence, which is defined as the ability to think strategically during an athletic performance. Or, for example, when, folks were given a survey beforehand about demographics and being a golf player. When people took that demographic quiz beforehand, white athletes did better. When people took the demographics question afterwards, it was about the same. It's all based around this idea that this threat of, hey. If I, know that this stereotype exists about me and I feel like it's gonna be a thing, then I might underperform because of that anxiety and narrative and centering that idea. This connects to the stereo this connects to the the learning loss, narrative because in media framing of learning loss, there's a stereotype threat against those of marginalized backgrounds. This whole narrative is primarily centered on schools that already were struggling through school report cards, etcetera, and that typically affects those of marginalized backgrounds the most. Teaching your class and you are centering this idea of learning loss and saying about how much was lost and you you constantly reference this and you say, hey. You know, we gotta make sure we improve test scores this year. They weren't great. Everybody lost learning. You might, I guess, inexplicitly or unintentionally, end up making those students perform worse, especially if, you you bring in, I I hope you wouldn't do this, but if you were bringing in the data itself, it's gonna talk about, marginalized communities. It's gonna talk about absenteeism rates. It's gonna talk about, inequality and inequity. So recognizing the fact that narrative, can have threats. The other one that we highlighted was the Pygmalion effect or also called the Rosenthal effect, which is basically where they took a group of elementary school teachers, and they, it was through Harvard University. These Harvard University researchers came in. They told these elementary school teachers, hey. We designed this cool new test, and it's gonna identify who the Spurders are. So the students that are really go getters that are academically talented, socially talented, they're these are just great kids. And they had the all these kids take this fake test. And the researchers took all this data, and they chose 20% of those students at random to be Spurders. There was absolutely no data involved in this. They are just like, these 20% are Spurders. And they took it back to the teachers and said, yeah. These are the Spurders. These these are the successful ones. These guys are great. They did so well, on the test. And they checked in with these teachers throughout the year. There were two things that they found. First off, the students that were identified as Sperger's, especially at younger, like, age levels, like first grade, second grade, drastically outperformed students on tests. Like, unbelievably, like, doubled their score on tests versus students who were identified as non Spurgers. And they did, like, narrative. They would go around and talk to the teachers and say, like, you know, what's the behavior like of these Spurgers? And the teachers were talking about how they were gonna be more successful. They were curious. They were interesting people and just how awesome they were. And then the, researchers would ask them about, the students who were not Spurgers. And then they would say, oh, you know, I'm that kid. You know, I don't know if they're gonna make it. They're not really that great. And then, trust me, if of the nonspers was doing really well, they would they would describe them in positive terms. They would say, like, oh, it was really unexpected. They're really unique. It's weird how they're able to do this without, you know, being identified as such. Like, it was, like, almost like a negative. And the researchers attributed that to the fact that when teachers saw that these certain students were Spurs, they maybe gave them more one on one attention. They were a little more friendly. They were more affirming. They felt like maybe the reason why younger learners were more susceptible is that when you're younger, you tend to be a little more, like, malleable. So it's like, hey. The teacher's paying attention to me. This is cool. And because you got paid more attention to, you tend to learn more for the test, and you tend to be like a happier kid because the teacher's paying attention to you and working with you and doing cool stuff. Whereas everyone else feels left out. I don't and that to me is just so fascinating because there's so many different ways that you could go with that. There is there is such an interesting,

Speaker 2 again, like that affective domain of this thing. Because in the bottom of that page 31, it says the researchers asked teachers about the behavior of their pupils. They consistently noted the Spurders had a better chance of being successful and happy later in life, as well as being more curious and interesting than other children. So so that is it it's kind of interesting to pit at the beginning of, of this down ninety minute episode. I I had mentioned, you know, the apocalyptic tones in which, you know, the New York Times, the Fordham Institute or whatever, talks about the the impact of learning loss on incomes over the lifetime of the child and, you know, of the hit in billions of dollars to national GDP, etcetera. And and that's based off of extrapolation from, you know, the two or three points in the in the score data that we've been talking about, etcetera. But here, it's it's the case too that's like, just the way that we treat kids can have a a a immeasurable impact on what? Being successful and happy, more curious and interesting, or just the way that, I guess, you perceive those things. And and, god, isn't that so interesting too to think that there's there are ways that we can impact the future growth of kids in a very positive way that isn't tied at all to what is getting measured on those assessment scores, and it it is instead tied to the ways that we structure our classroom to value groups of learners. And if we can, you know, put a value on those kids that are perhaps in struggling or underserved, you know, communities, then I I wonder what, you know, the the emphasis on putting the money into the things that are gonna value those socio emotional those qualitative things could do instead of just going towards the that test prep software. Again, right, there's the opportunity cost. There's that trade off associated with what we do with those funds and the outcomes for these kids. Are we gonna use that money to pursue narrow test scores? Are we gonna use them to, right, expand the child's experience of this return to school and deal with the trauma of the last eighteen months? Exactly. I I think it it centers the idea. I mean, I I guess

Speaker 1 credit where credit's due, using the words unfinished learning instead of learning loss is a more positive framing of this. And it it's gonna affect how people view the concept, but there are better ways that we can view learners about the previous year that approach this from an even more positive attitude. And this isn't to be confused with, like, like, toxic positivity. We're not saying that, like, if you just treat all students super well, that they're all gonna be, you know, they're all gonna be hyper successful. There's there's a lot more to it than just that. And there are students that need additional help that, like, you're gonna need to identify, like, students, for example, on an IEP that are are really struggling, that need additional supports. That's fine. But it's really ultimately just if we're approaching the school year saying, hey. We need to catch up, and that's all we ever talk about, we shouldn't be surprised when our students struggle more. Instead of acknowledging the adversity and how they overcame it and how valiant they are and how they pushed through it, how they made this work, and and really celebrating the fact that school continued during a global pandemic. All these teachers banded together and made it work. And for the most part, even though it was quite messy originally, I I didn't see a lot of just complete chaos. People got through it, over time, and there were a lot of positives. A lot of kids actually did better online. Like, they preferred socializing online. They liked the asynchronous elements. There are things that schools have adopted from this that they wanna keep. Like, some of them have developed, like, half days or more virtual days. They they've established better PD for online learning to reach more learners. So instead of going, oh my goodness. Look at all this learning loss. We need to dial it back and go, quote, unquote, back to basics, which we've repeated that ad nauseam for every single decade we've gone back to basic, which is code for go back to hyper standardized, quote, unquote, rigorous learning. Instead, we can make a school year that starts with hope and wonder and reimagination, reinvention, learning from what we just went through, reestablishing those connections and changing education for the better. It's not it's not a lost narrative. It's something that we're actually

Speaker 2 gaining and changing and capturing. So if if if this is gonna be the the the first part, right, where we're kind of talking about the the what and the why part, in in the next conversation that we'll have about this, we'll actually get to the latter half of the of the learning loss handbook where we're we are actually gonna address, like, how do we change that narrative and how do we, you know, grab onto those assets that we do have? How do we recognize our losses, and how do we push forward to change education for the better? Because we have some tools and some some templates and some activities that you can do both with yourself for, like, a personal reflection, you know, on on what we've lost, what we've missed out on, what we'll do differently, but then how to bring those conversations in with your kids too. So, again, like, refocus on values and refocus on on what kids miss missed out on and and how you can work to, right, bridge that gap and and address, again, the trauma that we've all been through of this last twenty months that has fallen disproportionately, you know, upon some people more than others for a variety of reasons. But thinking about how we can listen to kids, how can we actually start to address what is lost, how can we practice creative noncompliance and and push back against the the inhumane structures that may be built up in place instead of this? And then we'll kinda talk about some resources too that you can use to extend above and beyond all of this. So if you've made it this far, I mean, thank you so much for the time. It's not very often that we we do sort of a a loosely formatted thing like this. But if you like it, certainly let us know. This might be something we can we can do more of. If you hate it on the other hand, you want the a tighter, more more scripted kind of thing, let us know that too. But thanks thanks for sticking around. And One more thing too.

Speaker 1 If you stayed along this long, we are in the middle of our funding drive. So if you are listening to this in September 2021, you could give to us right now, get a bunch of donor gifts. We're trying to to reach a goal to continually produce these kind of resources because, honestly, it's a it's a lot of work. There are a lot of, sleepless nights trying to look at graphs and charts. So if you want us to keep doing that, whether or not you like us and you think this is really interesting analysis, maybe you don't like us and want us to keep looking at graphs and data. That's up to you. But, we we would really would appreciate your support. Goes a lot. Thank you again for listening to Human Restoration Projects podcast. I hope this conversation leaves you inspired and ready to push the progressive envelope of education. You can learn more about progressive education, support our cause, and stay tuned to this podcast and other updates on our website at humanrestorationproject.org.