Podcast Programs 1-8 Podcast Home Podcast #1: An introduction to basing our practice on better evidence Podcast #2: Changing the nature of the education conversation Podcast #3: Trends in using measurement to improve learning Podcast #4: Monitoring what gets taught: insuring an adequate opportunity to learn Podcast #5: Making science usable - engineering evidence-based knowledge into lessons and learning Podcast #6: Comparing ourselves - honestly - with the best Additional Audio Podcast #7: Using shared assessments to unpack standards and compare instruction Podcast #8: Making Mistakes and Moving Beyond Them is in the Scientific Spirit

Podcast Program #6

Comparing ourselves - honestly - with the best

Summary: This program looks at the movement to develop fair comparisons among schools. These so called "matched comparison groups" help educators judge whether their own efforts are as successful as those of their peers. Once educators see that others with similar students can perform better, it is hoped that they will learn from these better practices and improve their own classrooms as a result.

Introduction: When we make an appointment to see our doctor, we like to believe that she knows and understands the best and most current findings about what treatment works and what doesn't. We assume that our doctor has access to data and uses it and that she doesn't rely just on her own experience of what has worked for her patients.

In reality, though, the systemic application of outcome data to individual medical decisions and practice is a relatively new phenomenon. Doctors, like everyone else, are subject to what might be called the "particularistic bias;" they resist giving credence to data that comes from outside their own personal experience. When large scale statistical studies are done comparing actual outcomes, doctors are too often found to be in "violation" of those practices that are most effective — the best practices available.

In education, as in medicine, practitioners often have their fingers on the scale when it comes to assigning weight to conclusions based on their own experiences. They downplay or disregard objective data that does not support their decisions.

Host

Welcome to another of our podcasts on improving education through applying more rigorous ways of knowing. I’m Ed Janus.

One of the fundamental, if not the fundamental way science gets closer to the truth, is by finding ways to compare important things. To matter, these comparisons must be between like things. And, these comparisons should be made using the neutral language of numbers.

Comparing like things objectively in order to discover which is better suited for the job at hand helps us create better treatments for disease for example. Today there is a movement to create fair comparisons between methods of doing things—between practices. It’s called the “best practices movement,” and it’s based on the increasingly wide-spread collection of accurate data on outcomes.

Today, comparing the outcomes that result from different practices is being applied to a very wide range of endeavors. From baseball to medicine and now to education. It is a way to test the effectiveness of our efforts. A very good thing.

Our two guests are Megan McAndrew Cooper, the editor of the Dartmouth Atlas of Healthcare. Ms. Cooper is well aware of the best practices movement in medicine – and the resistance to it from doctors, by the way.

Next is Brad Duggan, the former president of the National Center for Educational Accountability, which has created a web site that allows schools to fairly compare themselves with schools with similar student populations. Schools are ranked by performance. The best performing schools are studied extensively to establish which practices they use that are different than their less well-performing peers.

Welcome Ms. Cooper. I take it that people interested in comparing outcomes first need accurate measures of important outcomes.

Cooper

Well, that's what people who are interested in outcomes research try to do, is say, "What we need to be able to do is to measure those things accurately." To use good, scientific methods to say, "What are the actual outcomes, and what are the outcomes over time," not just, you know, 30 days after discharge but after a year, after five years. The surgeon who does the open heart surgery is not who's treating you a year later.

There's a group in northern New England that did open heart surgery in the early '90s. And they started collecting data about, you know, numbers of procedures and that sort of patient sickness. And the best performing hospital had a mortality rate of two percent, and the worst had a mortality rate of about nine percent.

And so people who thought, you know, they were identified or thought they might have higher rates said, "Oh, you know, it's because our patients are older and sicker. And what they did, then did was start collecting information in, you know, a scientifically valid way, and over time they got enough information to say, "It evens out." Everybody had just about the same distribution of very sick to not so sick patients.

And you need data to do that. You, you know, you're talking to scientists, and you need to go back to them. They're trained in the scientific method, theoretically, and you go back to them and say, "No. We did this right. And we know that they're, these differences are not attributable to differences in the patients, they're attributable to something that's going on, that's different at your hospital and from this surgeon to that surgeon." And the doctors get, are very, they're very threatened. They're not used to being looked at that way.

Host:

I take it Mr. Duggan that you've been applying this same logic of fair comparison to schools and even classrooms?

Duggan

The purpose of investigating best performing schools is to define are there other schools with equal or more challenging students to educate where achievement is higher? Which is why you go on the web site and you get to see how many limited English-speaking kids are in each of these 10 schools. How many low income? All the other characteristics. So you can ask yourself, "Well, are they spending a lot more money than me? Do they have lower class size than me? Do they have X, Y, and Z different than me?" Until you finally sit back and say, you know, that school has similar kinds of challenges and in fact, they've even got more challenges.

Then go and investigate what's the instructional strategies that are different on the high performing schools, versus similar student population campuses that are performing average or below average? And that is called what we call the "opportunity gap."

Cooper

One big effort in the '80s and '90s that really bombed was to establish guidelines. They said, "This is how this is done at the hospital with the best record. This is what we, everybody agrees, you know, is the way to go about doing whatever this is." And doctors threw a hissy-fit, and said, you know, "We're highly trained, we're skilled, we're smart. And you're trying to give us a cookbook and tell us that we're just, you know, working our way down the checklist.

Even though that checklist, if followed properly, might result in better outcomes.

Exactly.

Host:

Do high performing schools tend to develop "best ways" of doing things? Starting with collecting consistent data on performance?

Duggan

And what we found was the high performing schools really spent more time having a conversation about who was the most successful in teaching the children the objectives of the last two weeks. Because it is based upon a set, common assessments of all of the students that the teachers have developed. So then, you had a common way on a Friday afternoon, when the teachers get together to say, "Well, how did your kids do?"

Cause there's got to be something that allows teachers to know how to compare the effectiveness of instructional strategies, classroom to classroom. And then we can start looking at, does one strategy work more successfully than another?

Then constantly there has to be this monitoring of student achievement. In which you are constantly checking through these kinds of assessments and a lot of other measures, how well this program is working and teaching the kids and there is a lot of dis-aggregation and breaking down of information. And the more specific, the more in detail they have done that, the higher achieving they are as a rule, the more they develop specific strategies to do that, the better.

And so, the principal can now sit there and, and say, "Okay, in your classroom, you are able to get more results in certain areas than the other teachers in the 6th grade. Let's have a conversation, let's schedule time where we can dig a little deeper and figure out what is happening."

Host:

How do hospitals uncover better practices?

Cooper

The doctors in this northern New England cardiovascular group went to each other's hospitals, we, they went in each other's operating rooms and watched what they did. And said, "This hospital, the perfusionist sits on the other side of the patient; this hospital, the perfusionist is behind the surgeon. These guys operate with cold room temperature, these guys operate with warmer room temperature. How can you figure out what in the system works and what doesn't work, and implement best practices? You know I mean, if you have an infection rate that's twice as high as my infection rate, let's look at what I'm doing that works better than what you're doing. Who is getting the best results, and how do you emulate those results?

And they could, you know, later could come back and say, "Look, there's really evidence that this doesn't do what you thought it did.? And people are, you know, surgeons are able to change their practice. Or, you know, good, the good ones are willing to look at real clear scientific evidence and say, "Okay," you know, "I - I am convinced and I will change." But in the absence of really good data about what works and what doesn't, everybody is working on theory.

Host:

So, comparing your school against a better performing but comparable school is the beginning of improvement?

Duggan

Well, it goes back to why, I think, educators have so embraced this voluntary organization of data. And why the schools that look at data in regard to this opportunity gap and set goals based on the best that is out there and go and investigate best performing schools, why the data shows that those schools increase in achievement 3-8% more than similar schools that don't do that. And the reason is you first have to create a belief in educators that other people are able to get higher results than what you are able to do with similar kinds of students working similar kinds of hours, still having families, still doing everything else that is required. And so, the heart of building that belief has to be, are there other schools with similar kinds of kids, can you build a fair benchmark model, and if so I want to know who those schools are.

So, you do that and then you identify people that they can call so that they can then say, "Well, how did you get from where you were 5 years ago to today?" It is getting over the disbelief, in a sense, and it allows you to go and visit schools that have been consistently high performing and that is what the goal of this is to empower educators with the knowledge of the profession. Not in an abstract way, not in a philosophical way, not even in a research way. What are the best performing schools doing and do I like those schools and do I believe in those results? And if you can get educators to walk down that path, then almost all the educators will sit back and say, "If they can do it, we can do it." And that is the moment when you realize the school is on an effort to improve.

And so, data is starting the conversation to build this profession so that everybody can investigate best practice.

Host

Until our next podcast, this is Ed Janus for the Center for Comprehensive School Reform and Improvement at Learning Point Associates.