The Reality-Check Rating
Anticipating the challenges to implementing policies and curricula that seem like a good idea
Do you know how many literacy lessons an early elementary school teacher needs to get through before the end of the school year?
One literacy curriculum has 160 lessons and there are 180 days mandated for schooling. OK, 20 days with no lesson. That doesn’t sound too bad!
But wait — there are field trips, school assemblies, fire drills, and more. And what if the lessons are very long, with lots of parts and transition time? Should the teacher rush through and try to do more than one lesson in one day? Are some lessons more crucial than others? Can parts be dropped or compressed?
Every day, teachers will be grappling with these issues and making these choices, likely with minimal guidance.
“Do you think this will go well?”
Dan Meyer, who writes a good deal at Mathworlds about how excellent teaching looks in the classroom, recently asked teachers to rate AI lesson ideas, using the question “Do you think this will go well?”
This question seems broadly applicable to many education interventions — and should be asked more often.
I spent my early career days working with states implementing huge plans to improve education in their states (e.g. New York). While there were successes, I saw that certain policy ideas didn’t work that well in practice, or had other effects that were not positive. Tim Daly at the Education Daly has done a great job documenting some of the challenges with implementation of policies such as more rigorous teacher evaluations. Now others are finding that high-dosage tutoring, despite promising findings from places such as Chicago and Washington, DC, may not achieve the academic gains expected.
A recent study from Matthew Kraft at Brown University and his team examined the impact of tutoring on Nashville students’ academic progress and found small to medium impacts on reading test scores and no impacts on math. There are many potential reasons for that, as the Hechinger Report notes, including computer and software issues and where the kids actually took part in the tutoring — school hallways are loud!
The study reminded me of some other implementation gaps that can take place. These are all real-life examples.
A class doesn’t get to the end of the lesson because it takes too long for students to transition from carpet to their seats and back again as proscribed by the literacy lesson.
A student sits for 30 minutes watching another student’s screen because the student can’t log into the math computer program and has been told not to interrupt the teacher while the teacher is leading a small group lesson.
A lesson is missed because the classroom was “evacuated” due to a student who was having trouble managing in class and was posing a safety issue to others.
A school bus is late due to an obstructed street and a set of students miss the first part of the lesson.
Play “Where will the barrier be?”
These implementation issues don’t mean that we should have a low bar. It’s appropriate to expect that on most days, high-quality instruction will be taking place in classrooms. But remembering what happens in the classroom on a day-to-day basis points us toward a few findings:
Anything new needs a “reality-check” rating BEFORE it is scaled. Many things are possible in a pilot when there are only a few students and teachers involved, everyone is very bought in, intensive training is offered, and there are additional resources provided. When programs scale, many of those components are dropped for reasons of expedience or expense.
Leaders need to proactively identify common barriers and develop systems to address them. They need to provide guidance for teachers on how to handle common scenarios that will arise.
E.g., how do we make sure each kid has a headset and that there are spares to replace those that are lost or broken? How do students alert a teacher or staff member if their computer isn’t working?
Rate-O-Rama
My family has a rating system for recipes that we make at home for dinner: (a) is it good? (b) How hard is it to make? That combines our appreciation for tasty food with the reality that if a recipe is really complex or tricky in some way, we are not likely to make it again.
For school systems, the rating would be (a) Is there proof of impact? (b) How hard is it to implement? This reality-check rating would equip administrators and teachers to go into a new initiative with eyes-wide-open about the potential challenges.
We can do hard things in schools. But we should be fully aware of how hard things are to do and try to make them easier to do. Researchers sometimes express concerns about lack of “implementation with fidelity.” Perhaps something is not being implemented with fidelity because it’s just really hard to do. Possibly there are ways to make it easier that won’t affect the efficacy of what we are planning to implement. To bring it back to literacy curriculum implementation, possibly kids can turn-and-talk on the carpet rather than going back to their tables once again. We can make it easier for teachers to implement the lessons.
The best ideas, poorly implemented, will not have the impact we’re seeking. Let’s focus on implementation in reality.