Frank’s Case Book

“…they ran their heads very hard against wrong ideas, and persisted in trying to fit the circumstances to the ideas, instead of trying to extract ideas from circumstances.” — Charles Dickens, Great Expectations.

Please forgive the citation of yet one more poll in an already poll-saturated season. Pew Research recently asked a presumably representative cross section of 515 Americans “Who do you think most newspaper reporters and TV journalists¹ want to see win the presidential election: George W. Bush or Al Gore?” By a margin of 47% to 23% Americans believe that Gore was the favorite of the national media.

Americans need not be a perceptive group to divine this conclusion. CBS News, in particular, has been conspicuous in its bias. CBS belatedly concluded that it should have extended the coverage of the first night of the Republican Convention by one hour. To compensate for this oversight they extended the first night of coverage at the Democratic convention. CBS News no longer even feels the necessity to appear evenhanded.

The latest example of TV news partisanship is the coverage of the recent “issue paper” released by the RAND Corporation two weeks before the election. The short issue paper questioned the extent of the “Texas Miracle” in educational achievement, especially among minority students.

The evening news and the morning news programs of the major networks led off with the story of the RAND issue paper with the implication that, contrary to conventional wisdom, Texas education had not recently improved. The recent release from RAND was not put into context and little mention was made of the far more extensive and pro-Texas RAND report by different researchers. TV news programs had largely ignored this previous report when it was released in July. On the other hand, the new issue paper was offered as dramatic new evidence that devastates Governor George Bush’s education credentials.

Of course the truth is far more complex. The validity of a report is not measured by its length, but even the authors of the recent issue paper from RAND Corporation, would concede that the earlier 200-plus page report by Grissmer et al. is far more extensive and complete than the 14-page issue paper recently released by Klein et al. just last week. Indeed, Klein et al. warn that their issue paper is based on scores from 20 schools from “one part of Texas.” The schools “were not selected to be representative of this region let alone Texas as a whole.” Hence, the TV news programs were focusing on a speculative report that was not even based on a representative sample of students.

The July report, by contrast, was based on analysis of data from the National Assessment of Education Progress (NAEP) tests between 1990 and 1996. In that report, Grissmer et al. of RAND concluded that “…some states are doing far better than others in making achievement gains and in elevating their students’ performance compared with students of similar racial and socioeconomic background in other states. Texas and Indiana are high performers on both these counts. One group of states led by North Carolina and Texas … boasts gains twice as great as the national average.”

In fairness, the July report observed that gains were made because of bipartisan emphasis on making the necessary reforms. Part of the period studied covered the terms of the previous Governor Ann Richards and the current Governor Bush. Moreover, the bipartisan educational effort included important work by the Texas legislature. Apparently a lot can be achieved if one does not worry about assigning credit. This is the sort of bipartisan approach that Bush claims he is embracing.

What has happened since 1996? The NAEP test can not yet reveal conclusive information. Although the NAEP tests are widely regarded as good measures of academic performance, they do suffer from some important limitations. The NAEP test are not conducted every year, nor in every grade, nor at every school. Therefore, they cannot be used as a means to hold specific schools accountable for specific results. They are much better at measuring long-term trends than in making up-to-date assessments.

With bipartisan support, Texas instituted the Texas Assessment of Academic Skills (TAAS) tests to monitor student achievement and to provide accountability. These tests measure reading and mathematics in grades 3, 4, 5, 6, 7, 8, and 10.

It is in these TAAS tests that students have demonstrated the greatest improvement in test scores, the “Texas miracle.” So what’s the problem? The problem is that although NAEP test given in 1998 for 4th and 8th graders shows significant improvement, the improvements were not as dramatic as in the TAAS test. Hence, the recent issue paper authored by Klein et al. speculates that perhaps there is something wrong with the TAAS tests. Klein et al. go on to speculate that since the TAAS tests are used for high-stakes school accountability, the test results may reflect an emphasis on teaching to the test.

In some sense, teaching to the test is not necessarily a bad thing. If the test measures learned skills, teaching to the test means learning those skills. Even more importantly, teachers may make an extra effort to remind students to get a good night’s sleep before the test. Teachers may give sample problems in a test-like environment to relieve the anxiety of students through de-sensitization. There are many thing teachers can do to help students prepare for test taking, strategies that middle class students may have learned earlier from parents.

Any school administrator will tell you that the best way to make your school system look bad is to have tests that measure skills and knowledge that do not match the schools’ specific curriculum. The greater success of students, especially minority students, on the TAAS tests may simply mean the Texas curriculum more closely matches the TAAS test than the NAEP test.

The Scholastic Aptitude Tests (SATs) are no less immune than the TAAS test from teachers and school systems placing strong emphasis on test preparation and focusing on the skills and knowledge covered by the tests. Given the attention paid to these scores by parents and school administrators, SATs also provide high-stakes accountability. However, if SAT scores in other school districts increased as much as the TAAS scores did in Texas, it would represent a cause for celebration not consternation.

As Chester Finn, former assistant US Secretary of Education points out, “NAEP alone shows commendable gains by Texas kids and schools — and Texas minority kids at or near the front of the pack among minority kids nationwide. This is solid accomplishment that would deserve praise even if Texas had no state test of its own.”² Indeed, when compared to California, a state that faces similar challenges with a large immigrant population, the Texas results look even more impressive.

Perhaps most disturbing is the suggestion by Klein et al. in the recent issue paper that perhaps low-income students did too well on the TAAS test. Privacy considerations make it impossible to know the precise economic status of students. However, there is too often a correlation between lower academic achievement at schools and the fraction of students on the free/or reduced lunch program. The more students on the program, the lower student achievement typically is.

The results of the TAAS tests did not conform to this “soft bigotry of low expectations” and were, therefore, in the minds of Klein et al. not credible. The test scores for economically poor students were higher than Klein expected. It is undoubted true that teachers at previously low-performing schools feel the most pressure to help their students improve their achievement. The most plausible explanation for higher performance of lower-income students is that demanding accountability from schools improves student performance. Klein et al. were not quite willing to consider this explanation.

As a scholarly effort, the work of Klein et al. reaches, at best, the level of an internal report summarizing incomplete work in progress. The Grissmer et al. report released in July, by contrast, represents a serious professional, independently peer-reviewed publication. The release of the Klein et al. issue paper two weeks before the election was at best premature and at worst will prove to be an embarrassment for RAND. To avoid the appearance of partisanship next time, the non-partisan RAND Corporation ought to make sure there is at least one non-Democrat participant³ in a controversial study criticizing the programs of a Republican, especially a study who’s timing is so close to an election.

The question of how much academic achievement has improved in Texas is a very legitimate and important area of academic inquiry and of news coverage. However, the excessive attention devoted to the speculations of this issue paper based on an admittedly non-representative data sample in the waning days of this presidential campaign is a clear measure of the desperation TV journalists feel in anticipation of potential of a Bush victory.

¹ TV journalist is someting of an oxymoron.
² Interview in National Review Online, 2000.
³ The National Review Online (A Conservative Magazine) reports that the registrar of voters in California has confirmed that all the authors of this report are registered Democrats.