No Data, No Problem: Harnessing the Power of Large Language Models for Psychological Scale Development

Zihao Jia

Advisor: Philseok Lee, PhD, Department of Psychology

Committee Members: Seth Kaplan, Reeshad Dalal

Johnson Center, #333, Meeting Room D
April 25, 2025, 11:00 AM to 01:00 PM

Abstract:

This research demonstrates the potential of NLP techniques and modern LLMs to streamline scale development, reducing reliance on large-scale data collection and expert evaluation. By implementing NLP-driven item selection and synthetic response generation, the current research explores how LLMs can enhance and support the psychological scale development process. Study 1 shows that LLM-based scale development produces psychometrically valid measures. Study 2 finds that LLMs can generate human-like responses to Likert-type items with minimal training data, though response quality is influenced by factors such as training item quantity and item key direction. These findings highlight the promise of LLM-assisted psychometric research, offering a scalable, cost-effective supplementary to the traditional scale development process.