Measure for measures: What do standardized tests really tell us about students and schools?
HGSE Professor Daniel Koretz

Video Clip (Part 3)

Daniel Koretz: Well, scores are often inflated. What’s particularly frustrating is we don’t know when they will be. But when they are, the inflation is sometimes enormous. It’s not uncommon to see gains on tests that are three to five times as large as they ought to be. And there are a number of cases in the research literature where large gains, really large gains on accountability tests have been accompanied by no improvement whatever than any other measure.

So we don’t know when it will appear. Even within a district, we don’t know which schools are going to show the problem, but we know it can be very, very severe. And I think it stems from two things. One is a lack of understanding of what tests are. What you hear in the field over and over and over again is, if the test has good material on it, why can’t we coach kids for this test? It’s sort of analogous to saying, “If Zogby has picked the right people for the poll, what’s wrong with our spending all our time trying to convince them?” It misses the point that the test is a very small sample that needs to represent something much larger.

The other problem is that the current political system is very, very narrow. What you get credit for (this is a bit of an oversimplification, but not much)-- What you get credit for under No Child Left Behind is raising test scores, period. It doesn’t matter how you do it. It doesn’t matter what else you do or don’t do. So if the question is, should you as a teacher choose something that will raise scores very rapidly or something that might raise them a little more slowly, but really get kids engaged, what would you do?

Well, under No Child Left Behind, there is nothing in place to look at whether your kids are engaged, whether they’re motivated, whether they leave school loving subjects or hating them, whether they are motivated to do work on their own, anything else. All that counts is one numerical measure. And what we’ve seen in field after field after field, not just in education, is that’s the wrong way to hold professionals accountable. You wouldn’t want to take your doctor and say, “We’re going to look at one measure of your performance. Nothing else you do with Joe Blatt matters,” right, “…only that one numerical measure.”

What you want is something that balances the incentives that teachers get and takes into account a wide array of the goals of education. That’s not what we have now.

Getting better: an ethical challenge

Daniel Koretz: Well, I think it’s at least partly fixable. If you look at work and incentives across fields, what you see in research that looks at professional jobs is there’s no perfect solution. You always have to be alert to ways that your accountability system isn’t working right. You have to keep changing things as people find ways around the system. But we could do vastly better than we are doing.

One of the problems, to be fair, is that if you compare, say, education to healthcare, efforts to hold doctors and hospitals accountable over the last several decades has spawned an enormous amount of research. And there is widespread free access to data by the people who are doing that research who can then say, “This method does not work as well as that method.”

One of the problems we have in education (and I think it’s an ethical problem) is that there is, in this country, no view that what people are doing in schools has to be evaluated. So it’s very common, for instance, for states and districts to assume that their data is their proprietary sandbox, basically, that they aren’t-- as long as scores are going up, they don’t owe kids a serious evaluation of what’s happening in schools. I think that’s appalling.

And it means-- First of all, it’s appalling on ethical grounds. But it’s appalling also because it means we can’t get better. The only way where we’ll get better is to monitor what happens in schools as we try out different things. And that’s, I hope, one of the things that will come in the next iteration when we sit back and think, “What has No Child Left Behind done right and wrong? And how can we do it better?” I’m hoping that serious evaluation of effects in schools will be on the table finally.

<view video> (requires RealPlayer)