Audio Webcast Program #1
Additional Audio from Program #1:
An introduction to basing our practice on better evidence
Additional Audio 1 of 5: Robert Slavin
Title: Better evidence can help us answer many of the core questions of practice
Listen Online: (3:07)
Download Audio in MP3 format: slavin.mp3 ~2.6 MB
Summary: Better, more evidence-based research can help us answer many of the core questions of practice — if we are willing to accept the verdict of science rather than our own deeply held beliefs. Robert Slavin, co-founder of Success for All is a believer in the power of science to improve education.
Introduction: It goes without saying that we would never allow a NASA engineer, for example to decide the trajectory of the space shuttle based upon his own personal experience of the trajectories of stones he skims along the shore. We demand "hard" evidence based upon extensive trials, and errors, of a large number of alternative trajectories. Our engineer would be guilty of criminal malpractice if he pushed the "launch" button without having highly predictive data about what works.
Likewise, as we're being prepped for brain surgery, most of us would be quite grateful for the countless hours spent by researchers analyzing data from trials of alternative methods of brain surgery. I think most of us would agree with the late teachers' union leader Al Shanker, that we would want our brain surgeon to follow proven methods rather than her own, idiosyncratic methods, no matter how creative. We would surely insist that our brain surgeon perform sufficient diagnostic tests to determine the exact nature and location of our problem and that she use ongoing diagnostics to keep her on target while she's operating. We would insist that she be "data-driven," not opinion-driven. In other words, we'd want brain surgery that worked.
Modern science is, rather simply, a social and intellectual system we can use to gain better and better knowledge of our world and how it works. Science articulates a set of effective rules and procedures by which we can test our ideas of how we think things work against the judgment of the real world. In scientifically-based knowledge, reality always trumps personal judgment, no matter how strongly or sincerely we believe something. To practice science is to make an almost moral commitment to seeing the world as it is, no matter how much it conflicts with how we'd like it to be. Science demands intellectual objectivity. And objectivity works! Think antibiotics, nanotubes, micro-surgery or 250 bushels an acre corn yields.
Transcript:
If you're asking a question like, "Should I use this program or that program?" Should we use full day K or half day K? Should we do ability grouping or heterogeneous grouping? Should we do cooperative learning or, or have kids sitting in rows? Those are all questions that lend themselves to experiments. And I think that we have, as a profession, done far too few experiments of that kind and that when the experiments are done, they're often not of the quality that you'd like. Not that they're, you know, that they're necessarily, you know, shoddy or dishonest, but oftentimes they're very brief, or they use a very small sample, or you know, they do other things that make the experiment less valuable than you might want to answer an important question like, you know, what happens if we use this program versus something else, over an entire year, or over several years; you really need an experiment that actually does that.
And I don't think it's going to happen by haranguing school board members or, or superintendents; I think it's going to happen only when government or researchers themselves or others are able to, first off, provide more useful research on topics that people really care about, but also when there's an opportunity for the results of research to be synthesized in a compelling way that enables people to use the results with more confidence; it's not just one person's opinion or what have you. And that's why I'm very excited about the What Works Clearinghouse, which is an attempt to do just that, to try to say, you know, "Here's what the evidence says. Here's where the evidence is lacking." But you know, here, this is really a guide to practice that I'm hoping will be, will help change the discussion a great deal, to a situation in which a, you know, somebody who is having to make a very difficult policy decision will have something much firmer to use as a basis for that decision.
I think certainly people need to be focused on objective data, both data that, from the outside, as well as data within the school system, so that things can be held up and examined as objectively as possible before you're making decisions that are consequential for children. And that's not an easy process because, you know, education is, has a tradition of being very ideological and being very much driven by belief systems rather than by evidence. And, but I do think that with, you know, with consistent leadership, that could change.
I think that eventually it's got to become acceptable. It's got to become mandatory for people to begin to look more at the evidence, and we need to have -- and I think we will have -- you know, more investment in high quality research in the first place, so that we can have more and more confidence in the results of the research.
Additional Audio 2 of 5: Howard Bloom
Title: Asking the basic scientific question
Listen Online: (3:14)
Download Audio in MP3 format: bloom.mp3 ~2.3 MB
About this audio clip: The basic question asked by science is simple: What effect does something have; does it change an outcome for the better. The best way to answer these "effect" questions is through rigorous, controlled experiments as Howard Bloom will discuss in this audio segment.
However, we ourselves can become more "scientific" in our day-to-day decisions if we ask ourselves, "what actually accounts for the effect I've observed." and if we gather objective data about effects. And, if we allow the chips to fall where they may.
Howard Bloom is chief research scientist at MDRC and a pioneer in applying rigorous science to social questions.
Context: Today, without objectivity and the practice of testing ideas against reality we cannot be said, to really "know" something. We may have good insights, intuition and much useful practical experience, but without testing these in a more unbiased way; without comparing alternatives, we are only guessing and hoping. But the practice of intellectually rigorous objectivity is not, shall we say, our normal default way of knowing the world. We human beings give an awful lot of weight to our own quite limited experience of what has, more or less, worked for us in the past. None of us exactly embraces the idea that we should actually test whether we are right or not. And most of us would consider it a total waste of time to apply, say regression analysis, in order to compare the effectiveness of our decisions to those of thousands of other people. Good enough is, well, good enough.
Most of us assume, for example, that our doctors are not doing much guessing and hoping when they treat us. But even though the scientifically-derived knowledge base of what works in medicine has grown incredibly, unbelievable in fact; even though this science saves countless lives and alleviates much suffering, its application to professional judgment and practice is not as wide spread as one would think, or hope. So even though doctors have access to vast amounts of real, valid and useful science and even thought they have many powerful diagnostic assessments, all too often our treatment is left to a "pre-scientific" reliance on limited data and understanding derived unsystematically.
However, today the gaps between science and practice in medicine are being addressed. Data-driven best practices are coming to the doctor's office. Doctors' judgments are becoming more and more tempered by objectivity. And, the demand that practice become more objective is also coming to the classroom and schoolhouse!
Transcript:
The fundamental problem in trying to figure out, in medicine, whether a procedure makes a positive difference or not, or whether a medicine improves things or not, is the same problem in education when you ask the question, does a new curriculum improve things or not? Or does a new institutional structure improve things for students or not? The question always is: What do you mean by improve things? What do you mean by impact? Or effect?
Well what you have to mean by effect is, does it cause -- if you're talking about medicine -- people's health to be better than it would have been otherwise. In education, it, did it cause, you know, student achievement to be higher than it would be otherwise? Can students read better than they otherwise could have? Can students do math more proficiently than they otherwise could have?
The problematic part of getting an answer to that question -- and that's what we'd call the impact question or the program effect question -- is: Well, what would they have done otherwise? How do you measure what the very same people would have done or experienced otherwise, absent the intervention, be it a medical intervention, an employment intervention, an education intervention, a housing intervention, criminal justice intervention. The question is always the same, you know: How much difference did it make, and therefore, what's the difference between the outcome that was experienced relative to the outcome that would have been experienced without the intervention?
What, for example, would educators want to know about a new reading program that they were thinking of adopting?
They should want to know how credible it is to say that, you know, this curriculum or this way of teaching kids or this way of dealing with kids' families enables them to read more, better than they otherwise would have. Well what is it, how did they, you know, the first question is: Well, how do they find out how they would have read it otherwise? Did they pick another group of kids to compare? And if so, how did they pick those kids? And is there any reason to believe that those kids are really reading the same way that the kids in the program would have read had they not been in the program? I mean that's always the problem. And with the random assignment experiment, when you can do it -- and by the way, I mean nobody is saying that you should do random assignment experiments every time you want to learn something, and for every question you do an experiment. That just doesn't make sense. That's not possible. That's a very simple-minded notion. But in order to ultimately get solid answers to the question, what was the impact of this procedure, this way of teaching, this way of running schools? We haven't been able to find, as a science, a more credible way than that.
Additional Audio 3 of 5: Howard Bloom
Title: Perfecting the art of teaching with objective evidence
Listen Online: (2:00)
Download Audio in MP3 format: bloom2.mp3 (~1.7 MB)
About this audio clip: Educators today are being called upon to inform their art with the more-objective knowledge uncovered by science, as Howard Bloom explains.
Context: Creating credible knowledge of things-that-work-better-than-others is on the agenda of those concerned that too much effort is being expended on too few academic achievement gains. Today, the use of rigorous scientific methods to establish a body of credible knowledge about how children learn, how teachers might best teach different kinds of children, and how schools might best be organized is emerging. Educators are being ushered into the Age of Objectivity. The standards movement demands it! And No Child Left Behind requires it!
On these Web pages and especially through the audio webcasts and audio discussions that accompany them, we 'll look at the emerging movement to put education on a more credible scientific footing -- to harness objectivity in the service of learning. Our goal is to introduce practitioners -- principals and teachers -- to the opportunities that this new movement offers them to improve the effectiveness of their practice. After all, science and the practice of objectivity have improved our lives immeasurably in nearly every area of life. By embracing the scientific revolution in education we educators will be able to bring a new and very powerful force to bear on the work we hold dear -- the improvement of the lives of our students.
This scientific revolution, as we 'll hear, includes a very wide range of objective information and powerful methods of systematic analysis; from the knowledge we gain in our classrooms from student and instructor performance data, to objective evaluations of program effectiveness, all the way to very sophisticated research on how humans learn using the most rigorous scientific methods. We 'll hear about the efforts now underway to "engineer" large scale science into useful tools for teachers. And perhaps most importantly, we 'll spend a good deal of time and attention on discussions of how teachers can use the objective data derived from thoughtful programs of assessments. For it is these data, made available and useful to teachers, that hold perhaps the greatest potential for learning gains. After all, good education, like good medical care begins with clear-sighted assessment and accurate diagnosis. Then it can proceed to a "cure."
Transcript:
Teachers now I believe, think of themselves often as kind of intuitive artists, that there's a craft to teaching.
Which I believe there is.
There must be. But in a sense, if we're going to give them tools that are scientifically based, we're going to ask them to be, in a sense, kind of more technicians, yeah?
Well, we're going to say, "Look, there's an art and a science to teaching." That, you're not operating in a vacuum; there are, I believe, certain fundamentals and certain near universals -- I hate to say universals -- but there are some forces at work there that are important to know about, and that hopefully you can, you know, influence things by understanding these forces, and that there are better and worse ways of doing things.
Now, clearly that interacts with who you are, and some people can use some methods better than others, so there isn't necessarily a universally best way to proceed in all, you know, in all respects. But I truly believe there are better and worse ways to proceed for almost any profession, and a willingness to understand that and to seek out that, I think is important in a professional.
I mean, it -- you know if, if in graduate school you're not taught to think about how to process the information you get, and you're not taught to think that science has anything to say to you that's worthwhile, you won't walk away from graduate school thinking that, you won't care. But if instead you are taught about, you know, the role of science in progress in any field, and how then to use that science to achieve your own personal goals, if you're taught that, then you're more likely to think about it, you're more likely to use it, you're more likely to let it affect the art of teaching that you develop, I think.
Additional Audio 4 of 5: Rebecca Herman
Title: Education's new FDA
Listen Online: (2:00)
Download Audio in MP3 format: herman.mp3 (~2.0 MB)
About this audio clip: Sixty years ago we had no FDA to certify the claims made by drug companies about the effectiveness of their drugs. Many a dollar was wasted and many a patient suffered as a result. Today the federal government is attempting to help "education intervention consumers" make better judgments so that they too don't waste dollars, or the lives of their students with educational snake oil. The What Works Clearinghouse will attempt to rate the science behind interventions. Rebecca Herman is the project director at the Clearinghouse.
Context: N/A
Transcript:
We are right now in, at a high stakes environment where schools and districts are being expected to deliver results for the, to demonstrate that they're improving student learning. And to do that, they're looking and being guided to look at research that shows evidence that the programs and practices that they're using will improve student learning, and the, the current catch phrase is scientifically based research for that.
But unfortunately, there is a lot of evidence out there, and it's very difficult for an incredibly busy school superintendent or teacher or principal to be able to sort through all this evidence and come up with, identify which are, is the strongest evidence, which is the evidence that one might consider scientifically based research, and then to summarize the findings from that evidence and shape their policies and their decisions based on that evidence.
It's something that, there's a movement toward doing this, using this scientifically based research in educational decision-making, but it's, it's a huge task to add on top of already overloaded practitioners and policymakers. One of the issues is it's very hard for somebody who's not steeped in research methodology to sort out an anecdote, somebody's report that this thing worked for them at one time, from a, a study that might have a little more validity in a number of different situations.
And so there was a need to create an independent, credible resource that would do this task and bring together the information and provide information on scientifically based research on educational interventions to education decision makers so that they could then make decisions based on sound evidence.
I see educators making decisions based on research or factoring the research into their decisions. I see that the level that educators will be critical consumers of the research, not simply accepting a claim that something works, like castor oil improves your digestion or whatever. But, but saying, you know, "If you make this claim, let me see the studies, let me see the research that supports it, and let me see that it is research that has a sound methodology."
-
Additional Audio 5 of 5: James Pellegrino
Title: N/A
Listen Online: (2:00)
Download Audio in MP3 format: pellegrino.mp3 (~1.5 MB)
About this audio clip: Responding quickly to information is important for survival. Is that a bear or a shadow? If it's a bear, it matters! So too does it matter that a student does or does not get a concept or skill before it is too late to do something about it. Responding quickly, before it is too late is the principle of today's revolution in assessment. James Pellegrino is an expert on measuring human performance.
Context: N/A
Transcript:
N/A
|
