Leonard Mlodinow’s The Drunkard’s Walk: How Randomness Rules Our Lives (NY Times review) takes on the absurdity of the grading process below.
For instance, a group of researchers at Clarion University of Pennsylvania collected 120 term papers and treated them with a degree of scrutiny you can be certain your own child’s work will never receive: each term paper was scored independently by eight faculty members. The resulting grades, on a scale from A to F, sometimes varied by two or more grades. On average they differed by nearly one grade.
Since a student’s future often depends on such judgments, the imprecision is unfortunate. Yet it is understandable given that, in their approach and philosophy, the professors in any given college department often run the gamut from Karl Marx to Groucho Marx. But what if we control for that—that is, if the graders are given, and instructed to follow, certain fixed grading criteria?
A researcher at Iowa State University presented about 100 students’ essays to a group of doctoral students in rhetoric and professional communication whom he had trained extensively according to such criteria. Two independent assessors graded each essay on a scale of 1 to 4. When the scores were compared, the assessors agreed in only about half the cases. Similar results were found by the University of Texas in an analysis of its scores on college-entrance essays. Even the venerable College Board expects only that, when assessed by two raters, “92% of all scored essays will receive ratings within ± 1 point of each other on the 6-point SAT essay scale.
Now, remember that many professors grade assignments with even more “precision,” handing out 86s, 93s, etc., and bask in the absurdity of false precision.