For the Course Instructor
For the Scorers
· Look at the 2 or 3 different aspects listed under each criterion. The degree to which each one of them is properly addressed in the respective part of the student's work determines whether this work "meets," "exceeds," or "fails" to meet expectations.
· These different aspects allow to make qualitative distinctions between different performance levels related to a specific criterion. Their presence or absence provides the evidence on which inter-rater reliability is based.
· Some of our current rubrics add a fourth performance level by distinguishing between categories such as "some proficiency" and "no/limited proficiency." This can be helpful for assigning student grades on a paper or presentation, but for the sake of program assessment, three performance levels are enough to decide whether the program has done its job.
· Each student product (other than simple tests) should be scored by two instructors. The course instructor, whose students' work is evaluated, can be one of the scorers. However, if the two scorers disagree on 25% or more of their scores, a third scorer should be consulted and break the tie on cases of significant disagreement between the original scorers.
· Scoring benchmarks should be created by having both instructors score a handful of papers first and then discuss any discrepancies they encounter. This helps establish common understanding of the rubric's criteria.
· In combining the scores for each criterion into a final assessment, the overall rating of a student's product is computed as follows:
o Exceeds Expectations, if student receives top scores on at least half of all criteria
o Meets Expectations, if student receives at least medium scores on all of the criteria
o Fails to Meet Expectations, if student receives less than medium scores on one or more criteria and cannot compensate the poor score with a high score on another criterion.
For more on Inter-Rater Reliability and Managing the Review, see M.J. Allen (2004), Assessing Academic Programs in Higher Education. (p. 145-152 and 158-159).