Audio Webcast Program #6
Additional Audio from Audio Webcast Program #6:
Comparing ourselves - honestly - with the best
Additional Audio 1 of 4: Megan Cooper
Title: The "particularistic bias" in medicine.
Listen Online: (2:58)
Download Audio Webcast MP3 File: cooper1.mp3 ~2.7 MB
Summary: Using data to look beyond our own, limited experiences, is important if we are to open ourselves to better ways of doing things. Knowing the outcomes achieved by others with students similar to our own sets us on this path.
Megan Cooper is the editor of the Dartmouth Atlas of Healthcare and a student of the best practices movement in medicine.
Introduction: Doctors argue, just as educators often do when referring to students, that their patients are more difficult to treat than another doctor's patients and that it is wrong to compare how rates of success in treating them. Proving to doctors and educators that it is possible to impartially and fairly compare their patients or students with those of other practitioners is the first step to getting them to willingly compare the outcomes they achieve. In many cases it turns out it is not the patient's -- or student's -- condition that determines the outcome, but the quality of the treatment the patient -- or student -- receives.
When educators begin to look at outcome data, they often find that another teacher or school is having better results with students with the same demographic and learning profiles. This discovery raises the questions, "What are they doing that I'm not doing?" and "What are they doing better?" Pursuing the answers to these questions is where the search for better practices begins.
Transcript:
Ed Janus
Using
data to look beyond our own, limited experiences, is important if we
are to open ourselves to better ways of doing things. Knowing the
outcomes achieved by others with students similar to our own sets us on
this path.
Megan Cooper is the editor of the Dartmouth Atlas of Healthcare and a student of the best practices movement in medicine.
Megan Cooper
Well, I think part of the, part of the problem is a question of small numbers. That
doctors don't see, they don't see thousands of patients. And the
effect, you know, in order to measure some of these effects, we've
looked at every single Medicare claim for every single beneficiary
over, you know, the course of ten years.
And that's how you, you
get statistically enough to make a judgment about whether something
works or not. And that's how clinical trials are operated is you have
to have enough people enrolled and randomized and, to come to valid
conclusions. Well, an average doctor doing, say, a hundred surgical
procedures a year, if his mortality rate is one percent, he's going to
have one death. If the mortality rate is two percent, you know, he'll
figure, "Gees, it was just two patients." And the difference, you know,
I mean it's a hundred percent difference, statistically.
But it doesn't really look that way to the person who's doing it,
and they see everything as being a special circumstance. And everyone
assumes, I mean it's just a universally shared assumption that our
patients are older and sicker.
I mean, so keeping up with medicine and understanding what's going
on is challenging enough without having to say, every time someone
comes out with a kind of broad study saying, "This works," or, "This
doesn't work," well, your experience has been pretty, you know, it
might have been pretty good, and it seems to you that it works. And
mostly it looks like you're having pretty good outcomes because almost
nobody tracks their own outcomes past 30 days after they're discharged
from, patients are discharged from the hospital. The surgeon who does
the open heart surgery is not who's treating you a year later.
Well, that's what people who are interested in outcomes research try
to do, is say, "What we need to be able to do is to measure those
things accurately." To use good, scientific methods to say, "What are
the actual outcomes, and what are the outcomes over time," not just,
you know, 30 days after discharge but after a year, after five years.
New York State started, sometime in the '90s, started publishing
mortality statistics by surgeon and by hospital for open-heart surgery.
I bet that was easy to get through, huh?
Oh, man. And the doctors get, are very, they're very threatened.
They're not used to being looked at that way. They're not used to being
ex-, you know I mean, they sort of see, you know, it's the "My patients
are older and sicker and I take riskier cases than other people do. And
if you expose what's going on I will be operating in a climate of fear
and I won't take people who are risky."
Additional Audio 2 of 4: Brad Duggan
Title: Investigating best practices begins with fair comparisons among schools.
Listen Online: (2:45)
Download Audio Webcast MP3 File: duggan1.mp3 ~2.5 MB
About this audio clip: The National Center for Educational Accountability has created an extensive web site where data is presented allowing educators to compare their achievements with those of schools just like theirs. The Center encourages schools to learn from those that perform best.
Brad Duggan was president of the Center.
Context: Although publishing and tracking many educational outcomes is now the law, actively pursuing ways to learn from these outcomes is a relatively new trend in education. Collecting data is important; but fearlessly looking at these data and outcomes, and comparing them honestly, teacher-to-teacher, school-to-school, district-to-district, state-to-state, and even country-to-country is fast becoming one of the most important "tools" an educator can wield. Learning how to learn from our more successful peers must be placed high on the to-do list of today's teachers and administrators.
Transcript:
Ed Janus
The National Center for Educational Accountability has created an extensive
web site where data is presented allowing educators to compare their
achievements with those of schools just like theirs. The Center
encourages schools to learn from those that perform best.
Brad Duggan was president of the Center.
Brad Duggan
Well,
the purpose of investigating best performing schools is to define what
is a best performing school and the goal of analyzing how a school
needs to improve is to define, are there other schools with equal or
more challenging students to educate where achievement is higher? It is
building a benchmark model that has the support of educators, and
that's important. So we analyze the performance of kids on the standard
that represents the level of achievement that educators believe
represent quality. We look at those kids that have been continuously
enrolled, and then we search the state to determine, are there other
schools that have as many low income kids (or more) than the school we
are talking about? As many limited English-speaking? Is at least 40%
the size of the school? Does it put more kids in special education?
And educators help us formulate that in each state and then we look at the
10 highest performing schools at the elementary level that meet all of
those criteria, average their achievement, and then look at the
difference between the school we are talking about (such as 5th grade
mathematics) and those 10 high performing schools in 5th grade
mathematics. And that is called what we call the "opportunity gap." And
we can do that for every grade and every subject where information is
available. Now that is step one.
Step two is to then using that data, analyze where schools have been consistently high
performing over time, over multiple subjects.
And the third thing is to then go and investigate what's the instructional
strategies that are different on the high performing schools, versus
similar student population campuses that are performing average or
below average?
Additional Audio 3 of 4: Megan Cooper
Title: Changing practice by looking at outcome data in medicine.
Listen Online: (2:25)
Download Audio Webcast MP3 File: cooper2.mp3 ~2.2 MB
About this audio clip: Getting doctors together to discuss the best ways of doing things has always been an important way to improve practice. Using data showing the various outcomes based on these ways of doing things is now driving medical practice much further much faster.
Megan Cooper, with the Dartmouth Atlas of Healthcare.
Context: N/A
Transcript:
Ed Janus
Getting doctors together to discuss the best ways of doing things has always
been an important way to improve practice. Using data showing the
various outcomes based on these ways of doing things is now driving
medical practice much further much faster.
Megan Cooper, with the Dartmouth Atlas of Healthcare.
Megan Cooper
Peer review is very important to doctors. And it's very, you know if you get, if you
convene doctors and say, "Look, your rates of surgery for benign
prostate disease are three times as high as his rates, why do you think
you do so much more surgery?" And then they would talk through why that
guy thought he did so much more surgery and what the sort of basis for
their decision-making was. Since you didn't have really good outcomes
evidence about that, it's all being done on the kind of theoretical,
"Well, if I do it now I won't have to do it later when he's older and
less able to tolerate surgery," that kind of difference of opinion. And
what that kind of group discussion of peers tended to do was drive
everybody towards the mean.
And they could, you know, later could come back and say, "Look,
there's really evidence that this doesn't do what you thought it did."
And people are, you know, surgeons are able to change their practice.
Or, you know, good, the good ones are willing to look at real clear
scientific evidence and say, "Okay," you know, "I - I am convinced and
I will change."
Would you like to apply what you know about the way medicine is
practiced to the way public schools practice using a school as sort of
the hospital, or a district as the hospital analogy?
Yeah, I think it, it's very analogous. I think there are lots of
ways in which it's the same thing. I mean, it has the same sort of
priesthood; you know, I teach or I'm a good person. I'm a doctor, I'm a
good person. And I'm smart, and you know, if your outcomes are bad,
it's "cause your mother didn't do it right."
The kid comes to school not ready to learn.
There must be some external variable that's causing this, not that
we're not producing this correctly or we're not doing the right thing.
And I guess the other thing is just that everything is, or, I don't
know how much, you know, sort of outcomes research there is in
education, but in the absence of really good data about what works and
what doesn't, everybody is working on theory.
Additional Audio 4 of 4: Brad Duggan
Title: Changing practice by looking at outcome data in medicine.
Listen Online: (1:52)
Download Audio Webcast MP3 File: duggan2.mp3 ~1.7 MB
About this audio clip: Teachers comparing the outcomes they have achieved after each lesson is one of the ways best performing schools are bringing teachers together for real, self-generated improvement. It is strong profession development.
Brad Duggan, an expert on developing best practices in schools.
Context: Every classroom and every school in American can become a vital experimental laboratory for what works -- and what doesn't. Using such tools as value-added measurements, statistical analysis, and professional insight, it is possible to uncover critical information about the many components of teaching and learning. The most effective practices can be imitated and new habits of teaching and classroom and school management established. Learning how to learn from the laboratories of every classroom and school puts the power of science into the hands of every teacher and administrator. And it will definitely improve the futures of our students.
Transcript:
Ed Janus
Teachers
comparing the outcomes they have achieved after each lesson is one of
the ways best performing schools are bringing teachers together for
real, self-generated improvement. It is strong profession development.
Brad Duggan, an expert on developing best practices in schools.
Duggan
And what we found was the high performing schools really spent more time
having a conversation about who was the most successful in teaching the
children the objectives of the last two weeks, what materials did you
use, and how do we then use them the next time we teach?
Cause there's got to be something that allows teachers to know how to compare
the effectiveness of instructional strategies, classroom to classroom.
So what the principal becomes, and what I have defined it, I watch the
principal sort of become from the super-person more to the person in
charge of quality control. And so, the principal can now sit there and,
and say, "Okay, in your classroom, you are able to get more results in
certain areas than the other teachers in the 6th grade. Let's have a
conversation, let's schedule time where we can dig a little deeper and
figure out what is happening." You can start sharing more activity
because you are monitoring now. You've got the ability to say, "Okay,
after these two weeks, we now have 7 kids who aren't performing at the
level they ought to, that we expect in the area of writing. What are we
going to do for these kids?" So the principal becomes the one who
monitors and adjusts the system and demands the resources for the
quality.
|
