Measure for measures: What do standardized tests really tell us about students and schools?
HGSE Professor Daniel Koretz
Video Clip (Part 1)
What tests can tell us
Daniel Koretz: There are a lot of things that we can learn from tests, though the irony is that as we’ve begun testing more and more over the past decade or two, tests in some cases tell us less, for a reason that I’ll get to.
But we know something about, for instance, how American kids compared to kids in other countries now that we didn’t know decades ago. We have tools that allow teachers to reliably compare how their-- I should say accurately compare how their kids performed to kids in other schools that they’d never met. We have tests that allow people to unpack performance in an area like mathematics and pinpoint some of the things that their kids can and cannot do to improve instruction.
The problem is that as people have become increasingly focused on the tests that matter, the tests for which people are held accountable, scores on those tests have often become misleading, sometimes wildly misleading. And that’s ironically undermined what we can say with confidence about how much kids actually know and can do.
Daniel Koretz: Well, the international comparisons have for us, but not for people in all countries, the same advantage that our national assessment has, which is that nobody is spending a lot of time prepping kids specifically for that test. So when scores go up on one of those tests, we have a fair degree of confidence that kids really know more. So that’s a tremendous advantage.
And I think it’s useful to know how we compare with other countries. The picture, if you zoom out enough and look at it in broad brush, is quite consistent. We score generally with a lot of similar-- similarly to a lot of similar countries, England for example, Australia. We always score much lower than countries like Singapore and Japan. And those patterns have appeared so many times, that there’s-- on so many tests, that there’s no question about them. They’re really there.
The problem is that the small differences are meaningless. And if you look at, for instance, the two major international tests now, TIMSS, the Third -- or now it’s called Trends in International Math and Science Study, and the PISA study run by OECD in Europe, they don’t always give you the same answers. They will always tell you that East Asia out-scores everybody else, or almost everybody else.
But a lot of countries bounce around between those two tests because the tests are quite different. So my advice to people is, look for the big picture. Make comparisons to countries that make sense to compare us to, but don’t pay attention to small differences, because you can’t trust them. They’re not going to necessarily stick around the next time.
And that advice unfortunately is not followed. There’s another bias, by the way, or a distortion in people's understanding of this. The countries that participate in these studies vary from year to year. Depends on which governments want to give access to schools and pony up the money. So you’ll see newspapers and sometimes our government reporting whether or not The United States scored above or below the international average. But there is no international average, other than the average of the countries that happened to participate that time.
And so we can be, and have been sometimes well above average in one year because of who happened to be tested, and well below average the next year because other countries jumped into the pot. So the sensible thing is, figure out to whom we ought to be compared, look at those comparisons, and pay attention to big differences that show up again and again. And with that, I think it’s very useful.