This blog post is part of a series called Asked and Answered, about writing great survey questions and visualizing the results with high impact graphs. Dr. Sheila B. Robinson is authoring the Asked series, on writing great questions. Dr. Stephanie Evergreen is authoring the Answered series, on data visualization. View the Answered counterpart to this post on Dr. Evergreen’s website.
When you think of demographic survey questions, does your mind automatically turn to age, gender, or race? Of course! These are some of the most common demographics we collect with survey questions. But there are many, many more demographic variables.
In Designing Quality Survey Questions, my co-author Kimberly F. Leonard and I share a pretty extensive list of 23 demographic variables we found in another text:
Alreck and Settle (2004) offer a list of what can potentially be measured with demographic questions.
Commonly Used Demographic Variables
A. Sex of respondent
B. Sex of family members
C. Age of respondent
D. Age of each head of household
E. Age of family members
F. Age of youngest child in the home
G. Education of respondent
H. Education of each head of household
I. Employment of respondent
J. Employment of each head of household
K. Occupation of respondent
L. Occupation of each head of household
M. Annual income of respondent
N. Annual income of each head of household
O. Annual family income
P. Racial or ethnic identity of respondent
Q. Race or ethnicity of each head of household
R. Religious preference of respondent
S. Religion of each head of household
T. Type of family dwelling
U. ZIP code or location of residence
V. Time of residence at present location
W. Self-designated social class membership
(Designing Quality Survey Questions p. 171)
Although this list is fairly comprehensive, a few notes are important: First, Alreck and Settle include “education of respondent” and “education of each head of household” (Items G and H). To be more precise, education is typically measured as highest level of education. Second, they include race and ethnicity (Item P) on the same line; however, these variables are distinctly different… “Annual family income” (Item O) is a variable more appropriately termed “annual household income” because family status and household members are not necessarily the same. Third, regarding “religious preference of respondent” and “religion of each head of household” (Items R and S), it is important to note that religious preference and religion may in fact be quite different…
(Designing Quality Survey Questions, p. 141)
Demographic Survey Questions are Personal Questions
Demographic survey questions can be a challenge to compose because they are so deeply personal to each respondent. We all feel differently about these questions, even if we’re in the same demographic groups. I may be perfectly comfortable answering questions about my age but not my income, and it may be just the opposite for you. Demographic variables can be especially sensitive topics for many people.
As survey designers, it easy to get excited about adding demographic survey questions to the instrument. Wouldn’t it be great to know all of this information about our respondents? The thing is, we have to think of our respondents. We don’t want to burden them by asking for information we don’t need, and we don’t want to risk having them tire of answering questions and either start satisficing – giving “minimally acceptable answers, rather than optimal ones that more accurately depict their thoughts or behaviors” (DQSQ, p. 64) – or becoming victims of survey fatigue, quitting the survey altogether.
If the survey is anonymous, the more demographic survey questions we ask, the more that anonymity is threatened. As a teacher, I responded to a survey from our union that asked which school I worked in, how many years I had been teaching, and in what subject. As you may imagine, I was the only special education teacher at the high school who had exactly 15 years’ experience.
Do your homework!
The key to designing quality survey questions (of any type) is like mise en place for a chef – have everything ready and in its place before you begin. For your survey to yield the best data you will need to have these “ingredients” ready:
1.) One or more research or evaluation questions. I call these the “big” questions to distinguish them from the questions we ask on surveys or in interviews.
2.) A clearly articulated purpose for using a survey. You need a clear picture of how a survey will help answer the big questions.
3.) An understanding of what you need to measure to answer your big questions.
4.) Knowledge about your respondents. Have an idea of who they are, what they know, and what kinds of questions they are capable and willing to answer.
Equity, Respect and Inclusion
One key to getting good data from survey respondents is to demonstrate respect and inclusion, especially when it comes to questions around race/ethnicity, gender, sex, sexual orientation and religion. It’s too easy to fall into the trap of asking only whether respondents are male or female without allowing them to answer differently. Even having “male” as the first or top response option is problematic. The same goes for asking about race/ethnicity and having “white” as the first or topmost option.
If our respondents don’t see themselves in any of the options presented, or if they see themselves further down the list than they should be, or worse, they only find themselves in the option labeled “other,” then how can we expect them to want to give us good data?
One blog article is hardly enough to do justice to the question of exactly how to ask these sensitive questions, and the truth is that there is no one right accepted way to ask them but I can offer some general advice:
- If at all possible, ask the question as an open-ended question and allow respondents to self-identify.
- If you must offer response options, consider alphabetizing them as opposed to ordering them according to who you think make up the largest subgroups.
- Along with a set of response options, offer the opportunity to opt out. I use “prefer not to answer” as a response option.
- Along with a set of response options, offer the opportunity to write-in their own response. I use “prefer to describe” as a response option.
- Avoid using the term “other.”
What About the Census Categories?
I get asked this a lot. What categories should we use for race/ethnicity, sex, or gender and aren’t the census categories best? The answer is… it’s complicated. If you need to compare to national datasets, then yes, those categories might make sense for you. But, if you know your respondent population and there are reasons you need to describe them in ways the census doesn’t capture, then use a different set of categories, and vet them with members of your respondent population. Pretest your demographic survey questions just like you would with the rest of your survey questions.
Do you hesitate to make changes in a regularly administered survey because then you won’t have trend data for awhile? You have 3 options:
A. Keep the demographic survey questions the same if you’re getting the data you need;
B. Keep the old and add some new demographic questions if it’s important to capture different information AND still be able to look back at trend data or compare to a national sample; or
C. Toss the old and start anew! Revisit your research or evaluation questions and the purpose for the survey and design demographic survey questions that will yield the rich, detailed, nuanced data that you need about your respondents.
We Have a Lot to Learn!
There is a great deal of meaning and nuance in the terms we use when asking people about their attributes. Terms not only change over time but are different among people from different generations, different cultural traditions, and with different levels of historical knowledge. It’s worth looking into a range of current literature and thinking especially when writing questions about race and ethnicity, gender, sex, and sexual orientation (or sexual preference). For example, do you know:
- The difference between asking a person’s gender and asking their sex?
- The difference between gender identity and gender expression?
- What cisgender, transgender, and non-binary mean?
- When and under what circumstances to use the terms queer or genderqueer?
- Whether or when to use terms such as Hispanic, Latino/a, Latinx, African American, Black, Caucasian, or White?
As someone who actively studies how best to design survey questions, I’ll admit that I’m still learning in the area of demographic survey questions and expect that learning will likely never stop. Look for a future article specifically addressing questions on gender and sex – I’m in conversation with some thinking partners about that topic now.
Lastly, if you look around the Internet for resources on asking specific demographic survey questions, just be sure you’re looking at recent information.
Pro Tips:
- Ask whether you need to know (because we don’t want to burden respondents!)
- Consider inclusive or qualitative options (tends to be the most respectful for all)
- Mutually exclusive responses(necessary for ANY multiple choice style survey question!)
- Limit number of questions (for anonymity)
In the other posts in our Asked and Answered series, we provide options for Rating Scale questions, Check All That Apply questions, and Ranking questions.
See you soon.
We go into way more detail on these topics in our books. Dr. Sheila B. Robinson is co-author of Designing Quality Survey Questions. Dr. Stephanie Evergreen wrote Effective Data Visualization.
Reference:
Alreck, P. L., & Settle, R. B. (2004). The survey research handbook (3rd ed.). Chicago: Irwin.