The methodology used to count the ballots is crucial to the reliability and credibility of the project findings. The fundamental problem facing any ballot counting exercise is that every method of ballot tabulation produces slightly different results with each pass through the ballots. The range of these differences varies from ballot system to ballot system. NORC uses the variation from pass to pass to measure the reliability of the particular ballot system used. Under this measurement scheme, one ballot system can be judged superior to another system to the extent that its variation from pass to pass is smaller.

NORC makes its assessment of reliability by calibrating the variation from pass to pass of the various ballot systems against the results of a careful hand examination. This examination of the appearance of every ballot is done by highly trained observers. These observers note in detail all aspects of the ballot, including hand notations, that might help identify voter intent.

A hurdle to be overcome when using this approach to measuring reliability is that, although hand classification is widely recognized as having the least variation from pass to pass, even hand classifications have some variability from pass to pass, as some people will see some ballots differently. The NORC project minimizes this variation by using three-person teams of highly trained observers, each team member working independently, to classify each ballot into categories based on the varying interpretations Florida canvassing boards have confronted in manual recounts of machine-readable ballots. On punch-card ballots, for example, NORC’s three coders recorded independently whether a ballot’s key chads (the pieces of the punch card that must be clearly punched through for the vote to be registered) were missing, hanging by one or more corners, or dimpled; whether light is visible around any dimple; and whether the rest of the ballot is consistently marked. Three independent codings guard against partisan biases and helps judge the degree of difficulty canvassing boards encounter in trying to assess voter intent on machine ballots.