This blog post is part of a series called Asked and Answered, about writing great survey questions and visualizing the results with high impact graphs. Dr. Sheila B. Robinson is authoring the Asked series, on writing great questions. Dr. Stephanie Evergreen is authoring the Answered series, on data visualization. View the Answered counterpart to this post on Dr. Evergreen’s website.
Ranking survey questions are a subset of multiple choice questions that allow multiple responses, but with the twist of requiring respondents to place their choices in a defined order. There are many varieties of ranking questions, some of which include:
- Asking respondents to rank ALL given response options
- Asking respondents to rank SOME given response options (e.g., rank your top 3 choices from these 7 options)
- Asking respondents to rank SOME options at BOTH ends of a given dimension (e.g., rank your 2 most important and 2 least important areas)
Ranking Survey Questions Can Be Useful
Ranking questions can be useful especially when you think respondents might feel favorably about most or all choices. Remember this example from the previous blog on Check All That Apply (CATA) survey questions?
This question probably won’t be helpful in this check all that apply format especially if you get a respondent like me who enjoys ALL of these flavors. (You really gotta mess up ice cream for me not to like it!)
But what if the question was asked as a ranking question?
Let’s imagine what might happen if I’m your respondent and answer this ranking survey question. You would find out in a hurry that vanilla, chocolate, and rocky road are among my favorites because I ranked them 2, 1, and 3 respectively. Good to know, right?
And if you asked this ice cream flavor ranking question to bunches of customers each year you would know how each flavor fared year over year and whether it went up or down in the rankings. Very useful for buyers and marketers, among others.
Ranking Survey Questions Can Be Problematic
Problems can lurk in the design of ranking survey questions, just as they can in other types. Let’s continue with our example of the ice cream flavors question as a ranking survey question. While it may be an improvement over the check all that apply survey question, as a ranking question, it isn’t perfect either.
Problem #1: We know some, but not much.
From my response, you would know that pistachio is my least favorite of these flavors, but there’s no way to know (without asking me another question) whether I actually enjoy it or not or to what degree.
We can imagine a number of different scenarios with this particular response.
A. I love pistachio ice cream, even though I ranked it 6, but just prefer all the others given the choice.
B. I enjoy most of the flavors, don’t like butter pecan all that much (rank = 5) but despise pistachio (rank = 6).
C. I don’t really enjoy any ice cream flavors except for vanilla (rank = 1), but just assigned a rank to the rest in the order of how distasteful they are to me.
And the list of possibilities could go on.
Although ranking questions allow respondents to help us understand how they see response options in relation to each other, these questions do not allow us to understand the strength or intensity of respondents’ feelings.
-Designing Quality Survey Questions, 2018, p. 103
Problem #2: Long lists of options to rank make for tired respondents.
Ranking items in order of desirability or importance (or some other dimension) is hard work and presents a relatively heavy cognitive burden for respondents. If your list of response options is long or if you expect respondents to rank more than 5 or 6 choices, survey fatigue and satisficing (i.e., respondents tiring and just giving any answer that completes the item without thinking deeply or expending a lot of mental energy on it) can threaten your results.
If I selected 15 books off your bookshelf and asked you to rank them according to preference, you might have a hard time. You might even have two or three that compete for that top spot, or it might get difficult after your favorite five or six books. Or you might just tire of the whole thing and just start assigning any old numbers just to complete the task and get rid of me!
Problem #3: Being first counts.
Another problem is that people tend to choose items from the top or beginning of a list, especially when that list is on the longer side (see Krosnick &Alwin, 1987). If you must use a longer list and can randomize the order of choices and randomize which respondents get different versions of the survey, you can potentially mitigate this primacy effect.
Do your homework!
The key to designing quality survey questions (of any type) is like mise en place for a chef – have everything ready and in its place before you begin. For your survey to yield the best data you will need to have these “ingredients” ready:
1.) One or more research or evaluation questions. I call these the “big” questions to distinguish them from the questions we ask on surveys or in interviews.
2.) A clearly articulated purpose for using a survey. You need a clear picture of how a survey will help answer the big questions.
3.) An understanding of what you need to measure to answer your big questions.
4.) Knowledge about your respondents. Have an idea of who they are, what they know, and what kinds of questions they are capable and willing to answer.
Problem #4: Analysis paralysis.
There’s nothing inherently bad about ranking survey questions, and they can serve a purpose, but be forewarned. They’re not so straightforward to analyze and interpret. You can use a weighted points system, perhaps assigning the most points to the item respondents ranked as their first or top choice. What’s problematic is that ranking questions operate on the principle that there exists the same interval between choices. If I really love all ice cream flavors except pistachio, there might be similar intervals of “liking” between the other choices, but when it gets down to butter pecan (which for the purpose of our example, I may like a little but not love) and pistachio (which I may despise and won’t go near), there is a HUGE interval. There might even be no practical interval between vanilla and chocolate, which I like equally as much, but forced into a choice, I may let vanilla edge out by a tiny margin.
If You DO Use Ranking Survey Questions
As usual, one survey question in isolation can never be expected to do a whole lot for us in terms of answering our research or evaluation questions, but it should serve a distinct purpose to the entire survey instrument and help move us toward this goal. If you decide to use ranking survey questions, of course you’ll want to design other purposeful questions that will help you flesh out what you’re learning from respondents.
The great news is, if you do have a ranking question in your survey and need to visualize the data, you have some really easy ways to do it.
Up Next in the Series…
If you want to know more about the people behind the responses, you’re in luck. Next up we tackle one of the trickiest topics in survey design – demographic questions.
Pro Tips:
- Consider alternative question format
- Mutually exclusive responses (necessary for ANY multiple choice style survey question!)
- Limit number of response options (or people will fatigue and skim)
- Randomize response option order if possible (this mitigates the possibility of the primacy effect- people choosing what they see first)
In the other posts in our Asked and Answered series, we provide options for Rating Scale questions, Check All That Apply questions, and Demographics.
See you soon.
We go into way more detail on these topics in our books. Dr. Sheila B. Robinson is co-author of Designing Quality Survey Questions. Dr. Stephanie Evergreen wrote Effective Data Visualization.