This blog post is part of a series called Asked and Answered, about writing great survey questions and visualizing the results with high impact graphs. Dr. Sheila B. Robinson is authoring the Asked series, on writing great questions. Dr. Stephanie Evergreen is authoring the Answered series, on data visualization. View the Answered counterpart to this post on Dr. Evergreen’s website.
Let’s talk about check all that apply survey questions. I’ll admit up front, they’re not my favorite.
I love ice cream. As a respondent, the survey question below would be really easy for me to answer. I’d simply check them all. (It’s not the same with vegetables. I still can’t stand asparagus.)
Check all that apply (CATA) survey questions are easy to compose. Just ask a question, list a bunch of options, and let people know they can choose as many as they like, right? The thing is, problems can easily hide in these types of survey questions. But first…
Do your homework!
The key to designing quality survey questions (of any type) is like mise en place for a chef – have everything ready and in its place before you begin. For your survey to yield the best data you will need to have these “ingredients” ready:
1.) One or more research or evaluation questions. I call these the “big” questions to distinguish them from the questions we ask on surveys or in interviews.
2.) A clearly articulated purpose for using a survey. You need a clear picture of how a survey will help answer the big questions.
3.) An understanding of what you need to measure to answer your big questions.
4.) Knowledge about your respondents. Have an idea of who they are, what they know, and what kinds of questions they are capable and willing to answer.
Problems with CATA
Let’s start with the example above. If I check all of the response options, you won’t learn very much about me except that I really like ice cream or that I’m not that fussy about flavors or perhaps even both – and you won’t know which it really is. CATA questions don’t always yield actionable insights.
Secondly, CATA questions can be tricky to analyze. One way to think about analysis is to treat each response option as if it was a dichotomous – YES or NO – question. How many respondents checked chocolate? How many checked vanilla? And so on…
Also problematic is that research* has confirmed (and we’ve all probably experienced!) that survey respondents don’t always read all the choices carefully, especially if the list is long. This means that if a respondent leaves a box unchecked, you won’t know if the person:
- simply skimmed past and really didn’t consider that option,
- was unsure if the option applied to them, or
- if that option truly didn’t apply to that respondent.
Alternatives to CATA
While there’s no way to ensure a respondent reads and considers each option on the list, there are alternatives to check all that apply survey questions. One is to take each response option and make it a dichotomous question – that is, a question with two response options, usually YES or NO. This essentially asks respondents to read and mark a response one way or the other to each choice.
Dichotomous isn’t the only alternative, however. Returning to your research/evaluation question, your purpose for the survey, and what you need to measure, you may consider asking people to what extent they like or dislike each flavor. Then, offer a set of response options such as “like very much”, “like a little”, “dislike a little”, and “dislike a lot.” Or simply “like it” “don’t like it”, and “no opinion.” The specific response options used will be a result of your own deep thinking about what exactly you want to measure and what you hope to learn from the data. It’s important to mention that research* on survey question formats has shown that respondents typically endorse more options in these formats than in a CATA format, but it’s not necessarily clear as to whether accuracy is improved with these formats.
Up Next in the Series…
Perhaps you need to know which ice cream flavors people like best and least – something that calls for a ranking question. You’re in luck… look for a blog on ranking questions coming soon!
In general, I encourage avoiding CATA questions unless you are confident they will yield useful data, AND you have a solid plan for analysis. If this is the case, go for it!
Pro Tips:
- Consider alternative question format (such as dichotomous or rating scale)
- Mutually exclusive responses (necessary for ANY multiple choice style survey question!)
- Limit number of response options (or people will fatigue and skim)
- Randomize response option orderif possible (this mitigates the possibility of the primacy effect- people choosing what they see first)
* See for example: When Online Survey Respondents Only Select Some That Apply; Comparing Check-All and Forced-Choice Question Formats in Web Surveys and Yes–no answers versus check-all in self-administered modes: A systematic review and analyses
In the other posts in our Asked and Answered series, we provide options for Rating Scale questions, Ranking questions, and Demographics.
See you soon.
We go into way more detail on these topics in our books. Dr. Sheila B. Robinson is co-author of Designing Quality Survey Questions. Dr. Stephanie Evergreen wrote Effective Data Visualization.