I‘m pleased to publish this guest post by my evaluator colleagues in the Idaho State Legislature’s Office of Performance Evaluations. Working in a governmental context gives them a unique perspective on education evaluation, as their work encompasses numerous fields and they experience evaluation in many different contexts. I’ve also long been a fan of their evaluation reports which exemplify a modern approach to reporting and dissemination: the report formats are visually appealing, and therefore easy to navigate, read, and comprehend, and they use well-designed visualizations to communicate key data points.
Rakesh Mohan is currently a candidate for the president of the American Evaluation Association. Since 2002, he has been the director of the Office of Performance Evaluations, an independent agency of the Idaho State Legislature. Under Rakesh’s leadership, the office received the 2011 Alva and Gunnar Myrdal Government Evaluation Award.
Lance McCleve is a principal evaluator with the Office of Performance Evaluations. He was the lead evaluator on the three evaluation examples mentioned in this post.
At least in the USA, K-12 public education evaluations seem to be curried with a bit of extra politics, so much so that the politics is too spicy for many evaluators to handle.[i] Like a curry, the politics of education evaluations is regional/local, people centered, and contains one or two secret political ingredients that evaluators are last to know, if ever.
All evaluations are inherently political, but evaluators and the evaluation process are expected to be non-political.[ii] Usually the politics of evaluation is seen at two stages: (1) when the decision is made to ask for an evaluation and (2) how the results of an evaluation are used or not used.
Here we use three examples of recently conducted education evaluations that came with especially spicy political curry:
Workforce Issues Affecting Public School Teachers
The K-12 Longitudinal Data System (ISEE)
Idaho’s Instructional Management System (Schoolnet) Offers Lessons for Future IT Projects
These three reports give the reader a clear sense of the politics that permeated the environment in which these three evaluations were assigned, conducted, and reported. For additional details reflecting the politics, see the related media stories:
Idaho teacher workforce OPE report: ‘Strong undercurrent of despair’
Goedde: Pay cuts beat teacher layoffs, focus on successes to build teacher morale
Lawmakers receive stinging report on student data system
After the $61 million Schoolnet fiasco, a scramble begins
The purpose of using these stories in this post is to illustrate the political nature of education evaluations. We do not intend to take a position on any of the political issues discussed in these stories.
Reasons for the Extra Politics
We think at least five factors contribute to the extra politics in education evaluations:
- Nearly everyone went to some school and has assumptions, beliefs, or reactions based on individual experiences. Unlike nuclear physics, nanotechnology, and neurosurgery, people from all walks of life act as though they are experts in education.
- Education policies affect most people either because they have children in school or they are taxpayers, or both.
- Providing adequate and equitable educational opportunities to all children is the responsibility of each individual state. Generally, one of the biggest parts of a state’s annual budget is for public education. For example, Idaho spends 48 percent of its general fund on public education.
- Among various stakeholders, teachers’ unions play a significant role in influencing education policies.
- Education policies have consequences for the future of our children and the nation.
So what should we evaluators do about the extra politics?
Suggestions for Managing the Extra Politics
The following four suggestions will apply to any type of evaluation, but they should be treated like edicts when it comes to conducting education evaluations.
Listen to all stakeholders with respect, regardless of how much you disagree with their views. This will help you fully understand the context of the evaluation, that is, who the key stakeholders are and what their competing, conflicting, and complementing interests are. Furthermore, this is the first step in establishing trust with all stakeholders. If you start on the right foot with stakeholders, your life as an evaluator will be pleasant and productive. The absence of trust with stakeholders could at best make the evaluation report useless from the get go and at worst, make conducting the evaluation a hellish experience. The methodology sections of our evaluation reports list the large number of stakeholders we needed to work with.
Don’t take sides, or you will get swallowed up in the politics. In all three evaluations, stakeholders were strongly polarized in their views about the condition of the programs we were evaluating and on who was to blame. We were able to avoid drawing conclusions based on stakeholders’ opinions, as persuasive or forceful as they might have been, by sticking to defensible methods. However, the bigger challenge was carefully conducting the evaluation and developing findings in a way that avoided the appearance of us having taken sides.
Stand up and own your evaluation approach and findings or you will get trampled. We were happy to see our report about workforce issues affecting teachers make headlines, but it came at a cost. Because of the intensely political nature of K-12 education issues, we immediately saw political backlash. Those who were not happy with our evaluation headed straight for our evaluation approach and methods to start criticizing. We easily could have stepped aside and let people do as they will with our evaluation but instead we took a stand and defended our approach and methods.
The disagreement between our office and those who were unhappy with the evaluation was significant enough to become the subject of several additional news stories. Although the disagreement was uncomfortable to say the least, it only served to strengthen our credibility.
Go out of your way to show respect for those who you’ve had disagreements with. You never know when they might champion your evaluation message in the future. Over the years, every policymaker mentioned in the news stories has agreed more than disagreed with our findings and recommendations. Remember—those who you had disagreements with are also doing their job, likely with the same passion as you do, and addressing the needs and demands of their constituents and stakeholders. Most of the time, disagreements are not personal – and even if they may seem personal, you should treat the people involved with total respect. This goes a long way toward building relationships and establishing credibility as independent evaluators who do not have a dog in the fight.
Acting on political knowledge is complex and potentially dangerous to evaluators’ credibility. However, by acknowledging the fact that all evaluations are inherently political and adhering to the four suggestions mentioned above, you will be able to manage the extra politics that comes with almost every education evaluation.
[i] K-12 public education evaluations are henceforth referred to as education evaluations.
[ii] Evaluator Advocacy: It Is All in a Day’s Work. American Journal of Evaluation, September 2014, 397-403.
I’d suggest a sixth driver of politics in education evaluations, which I increasingly think is at the real center of this dynamic – a fundamental disagreement about the nature of public education; whether it serves a sorting purpose or a human capital building purpose. Call it “competition” versus “cooperation” if it’s useful to focus on observable processes and cultural attributes rather than the bigger concept.
Some people (in whatever stakeholder group) believe that, broadly speaking, education is about sorting kids with more talent (or intelligence, desire, etc.) from kids with less, by giving all of them the same opportunity to succeed. Others believe that education should be aim to improve all kids’ situations, regardless of where they start and where they might end up.
These two groups are going to have completely different definitions of success for any education endeavor, will value different kinds of programs differently, and will have corresponding opinions about funding and evaluating them.
The hints above apply here but I find that keeping this dichotomy in mind helps me attend better to context (e.g., knowing how “sides” might be defined as I’m trying not to take one).
Kirk Knestis PhD
Thank you, Kirk. You raise an excellent point. I hope readers see your comment. Having lived in this country for 39 years, I totally understand the importance of this dichotomy. I wish this dichotomy existed in India when I went to school and college there in the sixties and the seventies. Unfortunately, the attitudes of the public, policymakers, and educators in India were heavily tilted toward the “competition” model.
We certainly have seen this dichotomy (although not explicitly) play out as we conduct our work in Idaho.