Direct and indirect measures of learning
Broadly speaking, measures of student learning can be characterized as either direct or indirect.
A direct measure of student learning evaluates what students know and can do. It usually takes the form of a content test (multiple choice, essay, etc.) or assignment (e.g., a paper).
An indirect measure of student learning might take the form of survey data or other feedback from students, employers, graduate schools, etc. regarding how well students were prepared by a course or program of study.
It is important that direct measures of student learning be reliable. A reliable measure will yield consistent results no matter who applies it. For example, a scoring rubric is reliable if multiple assessors, applying it to a particular piece of student work, arrive at the same score for that piece of work.
One way to establish that an assessment method is probably reliable is for the assessors who will be using it to hold a practice session in which they apply the method to examples of student work, compare their scores, and discuss whatever differences they find. This activity is likely in itself to increase the reliability of the assessment method because the comparison and discussion of scores leads assessors to be a more fully shared understanding of the standards being applied.
Another way to increase the reliability of an assessment method is to use teams of readers, rather than individuals, to score each piece of student work. The members of a team score the work independently, discuss their results, and settle together on a single score.
In assessment, the term validity refers to the correspondence between assessment results and student learning. Suppose that a means of assessment is designed to measure whether a student knows x. The means of assessment is valid if results indicate that the student knows x only when the student really does know x.
Since it is impossible to know whether a student knows x without relying on some means of assessment, the only way to demonstrate that a means of assessment is valid is to compare its results with those from other means of assessment. If several means of assessment agree in indicating that the student knows x, it is likely that the student indeed knows x. In this way, one means of assessment serves to check the validity of another means.
Hence the advantage of using more than one direct measure to assess student learning relative to a particular outcome.