Performance Improvement Journal: Ceiling Effects Associated With Response Scales
Chyung, S.Y., Hutchinson, D., Shamsy, J.A. (2020). Evidence-based survey design: Ceiling effects associated with response scales. Performance Improvement, 59(6), 6-13. https://doi.org/10.1002/pfi.21920
The Challenge
A ceiling effect is present in survey response scales when they can not accurately measure a survey respondent’s true response – fact or perception/opinion. For example, a survey may use a closed-ended survey item containing a statement (I found this workshop relevant to my work) and a five-point Likert scale (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree). If the result is consistently a 4.5 (agree to strongly agree) response, this response scale may not be measuring relevance with high fidelity. The response scale has limited (put a ceiling on) the participant’s ability to express their opinion on relevance negatively impacting survey analysis and conclusions. This can easily lead to a survey over or underestimating the value of an intervention. What can survey designers do to reduce the possibility of inaccurate survey results due to ceiling effects?
Work Description
The authors used a collaborative approach to document current literature in the area of survey response scale ceiling effects. Over 25 sources were used to identify when, why, and how a ceiling effect may occur in survey responses. The team engaged in multiple drafts, reviews, and proofreading iterations to arrive at the final article. This article is the final in a series of several articles presenting evidence-based research and practical tips for survey designers. The article presents several evidence-based recommendations for survey designers to reduce ceiling effects including using an increased number of response options, using positively-packed unbalanced response scales, and fully labeling response scale options.
What I Learned
You can have too much of a good thing!
If you are obtaining a majority of responses that sit on the high end of a response scale such as 5s on a 5 point Likert response scale or Excellent/Very Good on a Likert-type response scale, your training program may be great – no improvement needed! However, it is worth exploring the idea that your response scale may not be providing enough appropriate response types to identify your survey participant’s true opinion. Identifying and truly understanding your survey participants to better identify the likely responses they may provide are helpful in using appropriate response scale options in your survey to improve the accuracy of responses and findings. This will improve your survey accuracy and how those results inform your decisions about interventions.