A topic often discussed and a question often asked, "How should we order our rating scales?" is an incredibly important question as it can directly impact your overall scores. Does one way or the other swing data more positively? More negatively? 

MemberXP's best practice and recommendation is to place negative responses to the left and positive responses to the right. This is also shown on our standard survey templates to further promote this best practice as industry research shows there is "less bias" when ordering rating scales in this way. In other words, ordering rating scales positive-to-negative were shown to have more bias tied to the data, therefore inflating scores. 

Below are two examples of the same question, but its rating scale is ordered differently. Try it for yourself! Which question order do you find rating more positively versus more "true" to your experience? 

Since MemberXP codes the values from 1 to 5 for the first scale and 5 to 1 on the second scale you may have a very slightly higher average score on the second response option. 

How Large is the Left-Side Bias?

It’s important to keep in mind that this, and many other effects you get from changing question wording, question direction, labeling, and the number of scale steps is small, but noticeable. For example, a typical difference hovers between .2 to .3 of a point difference (on a 5-point scale).

You won’t start seeing these differences until your sample size exceeds 100 responses or so. As with most effects on response scales, the bias appears to occur more when the item being rated is phrased positively.  Research suggests the bias is more extreme when the subject of the question is highly emotional or even in some cases, politically loaded.

Why the Bias?

It is hypothesized that the bias comes into play with participant motivation, reading habits, and education level in conjunction with a primacy effect. Despite this left-side bias, our recommendation continues to be negative-to-positive scaling as a best practice method to "deflate" any possible positive bias with the opposite rating scaling.

Key Take-Aways:

• Putting the favorable response on the left-side will inflate the response a bit.
• If you are comparing the responses to past or future responses, don’t worry—whatever bias exists in the responses it will occur in both surveys. Comparisons are always more meaningful than standalone results.
• You’ll get the most accurate comparison to MemberXP’s industry scores if you use our standard left to right scale.
• One is not necessarily right or wrong—if you have an existing scale stick with it.