Now, THAT’S a very good question!

Sheila here, writing with the fabulous Kim Firth Leonard of the actionable data blog.

We have been tweeting and emailing about challenges related to survey design and as a result, this is the first in a series of posts we have co-authored about our discussions. 

I (Kim) have recently been flummoxed by the lack of good simple, current, published advice (preferably evidence-based) about phrasing survey questions. I feel like my go-to sources for survey design don’t include enough detailed information about what works when crafting questions.

We’ve shared a few resources with one another, and continue to digest those and seek others. There are certainly great resources covering threats to validity, reliability concerns, and broad advice like “use simple language”, “avoid double negatives”, etc. But nothing really tackles what we’ve been discussing — what makes a good question good – particularly in terms of phrasing? Is it hard to generalize advice about wording questions because context is too great a driver of that success? Are there known best practices regarding this aspect of survey design that we just haven’t discovered yet?

Despite a (at least perceived) lack of solid published information about survey question crafting, anecdotal advice abounds about how questions should be phrased. Some advice is hard to verify, or confirm. For example, take advice Kim has heard about not using emotive language (e.g. “do you feel included…”) because it’s too subjective. Yet, wording questions in other ways (e.g. “I am included”) seems in some ways more direct (which could be a good thing) and in other ways may become more difficult to answer. It’s certainly possible these two phrasings could result in different answers; two questions that seem to ask the same thing, may in fact, be about two different things.

Photo credit: Eleaf – Ethan Lofton (Flickr)

Ultimately, it is MUCH easier to be critical of poorly crafted questions than than to write successful ones. As surveys have become ubiquitous, we’ve both encountered numerous examples of “bad” questions.

Consider this example: I (Sheila) recently analyzed data from a brief survey of open-ended questions asked of a small number of practitioners across a few departments. The first question was “In what areas do you feel you have expertise?” The purpose of the question was unclear, and it could have been asked for any number of reasons. Was there intent to build a database of department members with expertise in different areas? Was there a need to get a general sense of departmental knowledge or experience? The word “expertise” is problematic as well, and open to interpretation. In fact, many respondents crossed out the word and substituted “experience” or “strength.” But what we really don’t know from a question worded in this way is whether people have simply had training or exposure in a certain areas, whether they have a certain level of confidence in a certain area, whether they feel they have enough knowledge or skill to practice in the area, or whether they have the ability to train others in that area. These are all vastly different questions, or at the very least one question stem – the “areas” with several response options (i.e. I have had training… I consider myself an expert, etc.). So, after analyzing data from this question with current wording, what do we really know now? The answer is… not very much.

As evaluators, we often think we have a good sense of how to effectively craft survey protocols and feel confident in our ability to ask good questions, but wouldn’t it be wonderful to have solid evidence for what works regarding survey phrasing? What resources would you would recommend that tackle this topic?

And even more interestingly, perhaps — What do YOU think it is that makes a good question good with regards to its phrasing?

Teaser! Look for future collaborative posts on: survey design resources, dos and don’ts of survey question design, and related topics.