The Center for Comprehensive School Reform and Improvement Center for CSRI Home
Audio Webcast Series: Harnessing the Scientific Spirit to Improve Learning

Audio Webcast Program #2

Additional Audio from Audio Webcast Program #2:
Changing the nature of the education conversation: What do we mean by "scientific," and how can the spirit of science help educators become better professionals?

  • Additional Audio 1 of 4: Howard Bloom

    Title: How doctors became "scientific." It wasn't so long ago.

    Listen Online: (3:31)

    You need version 7 or higher of the Flash plug-in to use the online MP3 Player. Click here to download.

    Download Audio Webcast MP3 File: bloom.mp3 ~3.3 MB

    Summary: Listen to a very short history of the movement in medicine to move away from practice based solely on doctors' opinions to practice more and more strongly influenced by the evidence derived from reliable science. Howard Bloom is our guest.

    Medicine is often held up as being the model of a scientific discipline; are doctors themselves actually "scientific?"

    Introduction: Testing beliefs about what works is fast becoming the preferred way to operate in areas of life that were once the sole domain of personal and professional experience. For example, our National Pastime, that ancient domain of managerial genius and foolishness, is itself becoming scientific. Baseball team managers are finally using the vast trove of statistics at their fingertips to test core questions of practice. Is it really more effective to bring in a left handed reliever against a right handed batter in all cases as common wisdom would have it? Are first round draft picks worth the incremental costs over 10th round picks? Does it make sense to "play for the tie on the road?" Some baseball managers are finding that common wisdom isn't all that wise after all.

    Sixty years ago, the federal government began to take the lead in basing medical decisions on evidence. Before that time there was no FDA to assess the validity and efficacy of medical practices or drugs. One simply trusted one's doctor who, as highly trained and well-paid professionals, were the accepted authorities on what made you well. Nowadays, of course, it is hard to imagine that a doctor could perform an untested surgery on us or prescribe medication that hadn't been tested through large scale random assignment trials.

    Today, many educators and policy makers believe that we are at the point medical science was 60 years ago: in need of greater scrutiny of the practices and "remedies" that are being prescribed for our children to ensure that they are effective.

    Transcript: Well I'm not an expert on medicine, okay? But the field of medical research has definitely become more scientific. There's been an, a very interesting question about: How does research get reflected in practice? Which is at the heart of your question. But there's no question that the science -- in my mind -- that the scientific research in medicine has definitely become more scientific.

    This stuff does get reflected in practice, but it takes time; it has to become absorbed over a period of time. It gets absorbed by journals that doctors that read, journals that doctors read, by conferences they go to, by conversations with their associates.

    Let's talk about authority. What's the source of authority in a profession? Well, authority used to be the docs. The doctors could tell you what worked. Put this in quotations: "They could tell you what worked." They didn't feel they needed statistics, in general, and experiments in particular to learn what works or not. They could tell you what worked. And they reigned supreme until the end of World War II. When I say they reigned supreme, you know, they were the ultimate -- you know, I'm putting all this in quotations, okay.

    They were the ultimate source of authority on what works and what doesn't. Well then the question became, "Should their opinion be the -- and only their opinion -- be the ultimate source of authority? What role should science play as a source of authority?" You see what I'm saying? The source of authority shifted over time from personal opinion, personal, professional opinion, almost exclusively. To some mix, which I wouldn't try to characterize, of personal, professional opinion and science. And it's that mix that' really at work. And it's that mix that determines what the doctor actually does for you when you go to visit.

    Did the incentive structure of medicine change, do you think, to force this adoption?

    Absolutely. And there are some fascinating parallels going on right now in other fields. There was no Food and Drug Administration much before I was born, which was at the end of World War II. You didn't have drug companies -- you didn't have the kind of laws we have now to monitor the implementation of new medical procedures and new drugs. There were some particularly high profile problems that occurred, that kind of raised the salience of the whole issue of how does one screen new medical products and procedures.

    There, the Federal Government indeed did step in and created a Food and Drug Administration, you know, a government institution that had authority over the ability to use new medical procedures and so on and so forth. You sort of feel like, "Well, that was always there." You know, and it sure as heck was not always there. It had to be brought into being.

    Yeah. And this bringing into being is where we're at in education today, hopefully.

    That's kind of what I see.

  • Additional Audio 2 of 4: Howard Bloom

    Title: Asking the "outcomes" question

    Listen Online: (1:33)

    You need version 7 or higher of the Flash plug-in to use the online MP3 Player. Click here to download.

    Download Audio Webcast MP3 File: bloom2.mp3 ~1.4 MB

    About this audio clip: To really test whether an intervention works, whether it is a new drug or a reading program, science must ask what is called the "counter-factual question." What would happen if I do not intervene? Would the outcome be better or worse? Even-handedly comparing the two outcomes gives us credible confidence that our intervention makes a difference or not. Howard Bloom continues with a lesson on basic experimental science.

    What, for example, would educators want to know about a new reading program that they were thinking of adopting?

    Context: NCLB is now the educational equivalent of the act of Congress that created the FDA. That is, rigorously examining the core practices of schools and choosing only those that meet the highest level of effectiveness is now the law of the land. But understanding what is effective is by no means easy, especially for educators who have no formal training in the scientific method or scientific analysis. Help is on the way, however, as the research community, federal and state governments and, yes, vendors gear up to put more scientifically validated practices into districts, schools, and classrooms. While this effort will ultimately pay off in better ways of doing things for students, it will require educators in districts and schools to work and think differently.

    Transcript: They should want to know how credible it is to say that, you know, this curriculum or this way of teaching kids or this way of dealing with kids' families enables them to read more, better than they otherwise would have. Well the first question is: Well, how do they find out how they would have read it otherwise? Did they pick another group of kids to compare? And if so, how did they pick those kids? And is there any reason to believe that those kids are really reading the same way that the kids in the program would have read had they not been in the program? I mean that's always the problem.

    And with the random assignment experiment, when you can do it -- and by the way, I mean nobody is saying that you should do random assignment experiments every time you want to learn something, and for every question you do an experiment. That just doesn't make sense. That's not possible. That's a very simple-minded notion. But in order to ultimately get solid answers to the question, what was the impact of this procedure, this way of teaching, this way of running schools? We haven't been able to find, as a science, a more credible way than that.

  • Additional Audio 3 of 4: Rebecca Herman

    Title: N/A

    Listen Online: (1:33)

    You need version 7 or higher of the Flash plug-in to use the online MP3 Player. Click here to download.

    Download Audio Webcast MP3 File: herman.mp3 (~1.6 MB)

    About this audio clip: As part of the federal government's efforts to promote good evidence as a force in education, the Department of Education has established the What Works Clearinghouse. The Clearinghouse will help educators judge the validity of claims that programs and interventions make that they are "evidence based." The Clearinghouse will make the science behind these claims transparent to educators not necessarily trained in the scientific method. Rebecca Herman is project director at the Clearinghouse. Here she describes the work of the Clearinghouse.

    Context: N/A

    Transcript: What we're doing is we're testing the claim that an intervention makes, that it works to improve a student outcome. So we first determine whether the study, how strong the study is. And we, from the very strongest of the studies, the ones that are the most valid, we then will report out what those studies find. So we will be telling you what the studies find but only from the studies that really, that have some validity, some credibility.

    The What Works Clearinghouse is unique in several features, and one is that our standards for saying that a study or a set of studies are credible and valid are transparent. So it's not a case where we get together a number of experts and they sit in a room and they talk and then they come out and say good or bad. So that's a unique feature of how the What Works Clearinghouse reviews the research and, and sets the standards.

    And I want to caution against saying that we, we stamp something or approve something because that's really not what we're doing. We're really trying to create a decision support tool where we provide information and we provide enough information in an easily digested, partially digested way, that, that education decision makers can decide, make their own decisions based on this information.

  • Additional Audio 4 of 4: Lisa Towne

    Title: N/A

    Listen Online: (3:17)

    You need version 7 or higher of the Flash plug-in to use the online MP3 Player. Click here to download.

    Download Audio Webcast MP3 File: towne.mp3 (~3.0 MB)

    About this audio clip: For Lisa Towne, empowering educators to have deeper conversations about their practice will be the most important effect of the movement towards evidence-based practice. Ms. Towne is a study director with the National Research Council.

    Context: By funding The Center for Comprehensive School Reform and Improvement, the Institute for Education Sciences, and the What Works Clearinghouse, the US Department of Education is trying to change the nature of the education conversation from arguments driven solely by personal and professional opinion and ideology, to those driven by scientifically validated evidence. These three pillars are education's FDA; a source to which educators can turn to judge the scientific strength of claims made by programs and interventions. These are the resources that can move educators from intuition to knowledge.

    Transcript: And so, part of what's going on, you know, there's examples, you'll be talking to people that are involved in the What Works Clearinghouse. That's an example of the kind of tool that's going to be really important, because they're taking very seriously this idea of systematically reviewing a set of studies about a certain topic.

    You can see a mile away the vendor coming a-calling, with their product that's got the scientifically based research stamp right on top of it. And so what we've got to do is help to put in place these kinds of resources so that superintendents aren't caught flat-footed and they're going to be able to look at the features of the, the so-called study that's behind that stamp, and make a judgment as to whether it's legitimate.

    Do you see this, our ideal superintendent of the future, having a somewhat different, shall we say, mentality or ethic intellectually about his or her work, in the light of being part of science?

    I would hope so. I think one of the, I think one of the challenges for that is, is the nature of the job of the superintendency. Right now it, they, at the moment really don't have the time to reflect on what good research suggests about what to do in their schools. You know for them truly to be instructional leaders, a knowledge of relevant research needs to be part and parcel of that leadership.

    And I think that change in mindset is more likely to occur if, if we don't try and shove scientifically based research down their throat, in a way, you know? As, so long as we, as we try and, and convey to schools and districts that this is really empowering for them, and that this really has very strong potential to help their kids, that will make this kind of a culture change more likely to occur in a positive way.

    Do you think that the incentive structure of education has to change to some degree to accommodate this new objectivity? Pragmatism?

    Sure. I mean I think the fact that, that these requirements are now in Federal law helps the incentive structure. You know? You know states and districts now do have to pay attention to this. Now of course the way in which they approach doing that matters a lot. If they go about this, from the perspective of just doing what's minimally required to comply with the law, then that's not really going to advance what I think is the, is the spirit of scientifically based research provisions in the law and the larger evidence based education movement.

    I think part of what's compelling about that case can be just engaging people in a conversation about how they think they know what they know. How do you know that what you're doing in schools is the most effective thing you can be doing right now to get your kids to proficiency?

    So at any one given point in time, this is the role of the research community, I think, that the research community needs to help to try to, to try to say, "Okay, our professional consensus, our best judgment at this point in time, given the evidence before us, suggests that this is what the research base says."