Tips to help organizations avoid the common pitfalls in question design.
Writing Unbiased Questions (and Avoid Leading Questions)
To get honest answers, you need to ask honest questions. To have the results stand up to scrutiny, make your questions as neutral as possible. When you write a question, think about whether it is as easy for your respondent to agree as it is to disagree, and make sure that you are not signaling a correct or preferred answer.
Three Tips for Good Questions
1. Be neutral in your wording.
Biased: How strongly do you agree with the following statements:
(Implies that they should agree)
Neutral: How strongly do you agree or disagree with the following statements:
Biased: How often do you attend (city name) City Council meetings?
(Implies that they should be participating)
Neutral: How often, if at all, do you attend (city name) City Council meetings?
2. Context is important; however, if you provide background information, don’t overemphasize the good, and leave out tradeoffs.
Vague: How strongly do you support or oppose building a new aquatic center?
(The respondent might wonder where it will be and at what cost to them. Answers to a question like this might give you a general up/down on the concept, but not a good idea of specific support or opposition)
Biased: A new aquatic center would improve the lives of children and the economic vitality of the downtown area. How strongly would you support or oppose building a new aquatic center?
(You might as well ask, “Do you hate children or a good economy?” How could a respondent disagree?)
Neutral: The city is exploring building a new aquatic center downtown on the site that was vacated by Department Store X. Its construction and maintenance would be funded by an increase in property taxes of approximately $30 annually for a median-priced home ($250,000). How strongly would you support or oppose building this new aquatic center?
3. Skewing has its place. To address social desirability bias, sometimes you want to help make negative responses be OK.
In your wording, you might want to imply that it is OK to answer “negatively” when a respondent might feel that one behavior is more socially acceptable.
Neutral: Did you vote in the last election? Yes/No
Mitigated: Sometimes things come up that can prevent people from voting. Did you manage to vote in the last election? Yes/No
Alternatively, in the response categories, you can give a positive skew to your scale to make it easier to give a negative answer—people will be more comfortable saying that the service was “3=Fair” rather than “4=Poor” and “3=Poor” rather than “4=Very poor.” By the same token, more people will choose “3=Fair” than “3=Poor” (note the example scales below). Allowing people a more positive skew on negative feedback will give a more honest assessment of the item.
Balanced: 1=Very good, 2=Good, 3=Poor, 4=Very poor
Positive skew: 1=Excellent, 2=Good, 3=Fair, 4=Poor