On the Reliability of Judgments Made in Policy Capturing Measures: A Meta-analytic Review and Experimental Analysis

Alan Tomassetti

Advisor: Reeshad S. Dalal, PhD, Department of Psychology

Committee Members: Jose Cortina, Stephen Zaccaro

David J. King Hall, #2006
May 09, 2017, 10:00 AM to 07:30 AM

Abstract:

Despite being used ever more frequently over the last half-century, policy-capturing is still treated apart from other social science methodologies. Nowhere is this more apparent than in the fact that the majority of policy-capturing studies do not report an appropriate reliability estimate. Advice from Aiman-Smith, Scullen, and Barr (2002) and Karren and Barringer (2002) has failed to effectively persuade researchers to report reliability and has failed to lead reviewers/editors to require estimates of reliability for policy-capturing studies. Through a meta-analysis and a primary study, this paper will also expand on their work by providing recommendations for best practice in designing policy-capturing studies, measuring test-retest reliability, and reporting factors that are critical for proper evaluation of a study’s reliability.