You never know when a simple root cause exercise like “the 5 whys” will come in handy. When a program evaluation grad student met with me to discuss her evaluation project, she started with “Well, I’m going to survey teachers.”
“Why?” I asked.
And she went on to share something of interest she wanted to know.
Again, I asked, “Why?”
And again, she explained.
And we continued this pattern until she was able to clearly articulate:
- The purpose for conducting the evaluation project.
- What broad evaluation questions surveying those teachers would help to answer.
- What additional data she might need to collect to fully answer those broad questions.
And finally,
4. How the answers to those broad questions would inform important program-related decisions.
Like the linchpin to a wheel, evaluation questions are vital to an evaluation project. If we don’t pose evaluation questions that connect the project pieces, the pin comes out and the wheel falls off.
What are evaluation questions?
Let’s begin with what they are not. Evaluation questions are NOT survey questions, interview questions, or assessment questions. Evaluation questions are not individual questions we would ask individual humans. Evaluation questions are asked of the program. They’re analogous to research questions a researcher would pose for a study.
Much like research, evaluation begins with a small set of broad questions that drive the project. They provide foundation and direction for the work, and point the way toward which data to collect and how to collect it.
What happens in the absence of evaluation questions?
An evaluation project that doesn’t start with a set of evaluation questions is like going to the grocery store without a list. You get off course, lured in by new products, interesting endcap displays, and sales. And you come home with twelve pounds of cheese your kids won’t eat, and a lifetime supply of pretty paper napkins that don’t match anything.
An evaluation without a set of evaluation questions is one that goes off course, gathering interesting “shiny” data that doesn’t connect to information needs and won’t yield meaningful, actionable information. This is why my student’s project might have failed, had she not started with evaluation questions, rather than starting with a data collection strategy. And it’s not easy! I LOVE developing surveys and watching the new data come in. That to me is the fun part.
How should evaluation questions be developed?
In short? Begin with purpose.
As with any project you embark on, developing evaluation questions start with good questions!
And the first one is: What is the purpose of the evaluation?
Some evaluations set out to examine outcomes of the program or the degree to which goals have been met. Other evaluations aim to explore processes and look at what may contribute to outcomes being achieved (or not achieved). Still other evaluations look at cost-benefit or cost-analysis, or assessing community needs.
Once you are able to identify and articulate a purpose for conducting evaluation, consider these questions:
- What does the evaluation hope to uncover?
- Who is requesting the evaluation?
- Whose interests are being served by conducting an evaluation?
- What are the information needs?
- What do you need to know about the program for the purpose of the evaluation?
- What do you need to know in order to make necessary decisions about the program?
- Will evaluation questions be determined by the evaluator, program personnel, other people with an interest in the program or its outcomes?
- Will evaluation questions be developed and determined collaboratively?
Evaluation questions are a critical component of evaluation planning. As you plan your evaluation project, ask yourself the 5 (or even more than five!) whys!
A gift for you! Here’s a set of generic evaluation questions that can give you a starting point.
Read my other articles on evaluation.
Interested in a talk or workshop on any of the topics I offer? I’d love to chat with you.
Image credit: Thamizhpparithi Maari on Wikimedia