I‘m pleased to publish this guest post by my evaluator colleagues in the Idaho State Legislature’s Office of Performance Evaluations. Working in a governmental context gives them a unique perspective on education evaluation, as their work encompasses numerous fields and they experience evaluation in many different contexts. I’ve also long been a fan of their evaluation reports which exemplify a modern approach to reporting and dissemination: the report formats are visually appealing, and therefore easy to navigate, read, and comprehend, and they use well-designed visualizations to communicate key data points. 

Rakesh Mohan

Rakesh Mohan is currently a candidate for the president of the American Evaluation Association. Since 2002, he has been the director of the Office of Performance Evaluations, an independent agency of the Idaho State Legislature. Under Rakesh’s leadership, the office received the 2011 Alva and Gunnar Myrdal Government Evaluation Award.

Lance McCLeve

Lance McCleve is a principal evaluator with the Office of Performance Evaluations. He was the lead evaluator on the three evaluation examples mentioned in this post.

At least in the USA, K-12 public education evaluations seem to be curried with a bit of extra politics, so much so that the politics is too spicy for many evaluators to handle.[i] Like a curry, the politics of education evaluations is regional/local, people centered, and contains one or two secret political ingredients that evaluators are last to know, if ever.

All evaluations are inherently political, but evaluators and the evaluation process are expected to be non-political.[ii] Usually the politics of evaluation is seen at two stages: (1) when the decision is made to ask for an evaluation and (2) how the results of an evaluation are used or not used.

Here we use three examples of recently conducted education evaluations that came with especially spicy political curry:

Workforce Issues Affecting Public School Teachers

The K-12 Longitudinal Data System (ISEE)

Idaho’s Instructional Management System (Schoolnet) Offers Lessons for Future IT Projects

These three reports give the reader a clear sense of the politics that permeated the environment in which these three evaluations were assigned, conducted, and reported. For additional details reflecting the politics, see the related media stories:

Idaho teacher workforce OPE report: ‘Strong undercurrent of despair’

Goedde: Pay cuts beat teacher layoffs, focus on successes to build teacher morale

Lawmakers receive stinging report on student data system

After the $61 million Schoolnet fiasco, a scramble begins

The purpose of using these stories in this post is to illustrate the political nature of education evaluations. We do not intend to take a position on any of the political issues discussed in these stories.

Reasons for the Extra Politics

We think at least five factors contribute to the extra politics in education evaluations:

1. Nearly everyone went to some school and has assumptions, beliefs, or reactions based on individual experiences. Unlike nuclear physics, nanotechnology, and neurosurgery, people from all walks of life act as though they are experts in education.
2. Education policies affect most people either because they have children in school or they are taxpayers, or both.
3. Providing adequate and equitable educational opportunities to all children is the responsibility of each individual state. Generally, one of the biggest parts of a state’s annual budget is for public education. For example, Idaho spends 48 percent of its general fund on public education.
4. Among various stakeholders, teachers’ unions play a significant role in influencing education policies.
5. Education policies have consequences for the future of our children and the nation.

So what should we evaluators do about the extra politics?

Suggestions for Managing the Extra Politics

The following four suggestions will apply to any type of evaluation, but they should be treated like edicts when it comes to conducting education evaluations.

Listen to all stakeholders with respect, regardless of how much you disagree with their views. This will help you fully understand the context of the evaluation, that is, who the key stakeholders are and what their competing, conflicting, and complementing interests are. Furthermore, this is the first step in establishing trust with all stakeholders. If you start on the right foot with stakeholders, your life as an evaluator will be pleasant and productive. The absence of trust with stakeholders could at best make the evaluation report useless from the get go and at worst, make conducting the evaluation a hellish experience. The methodology sections of our evaluation reports list the large number of stakeholders we needed to work with.

Don’t take sides, or you will get swallowed up in the politics. In all three evaluations, stakeholders were strongly polarized in their views about the condition of the programs we were evaluating and on who was to blame. We were able to avoid drawing conclusions based on stakeholders’ opinions, as persuasive or forceful as they might have been, by sticking to defensible methods. However, the bigger challenge was carefully conducting the evaluation and developing findings in a way that avoided the appearance of us having taken sides.

Stand up and own your evaluation approach and findings or you will get trampled. We were happy to see our report about workforce issues affecting teachers make headlines, but it came at a cost. Because of the intensely political nature of K-12 education issues, we immediately saw political backlash. Those who were not happy with our evaluation headed straight for our evaluation approach and methods to start criticizing. We easily could have stepped aside and let people do as they will with our evaluation but instead we took a stand and defended our approach and methods.

The disagreement between our office and those who were unhappy with the evaluation was significant enough to become the subject of several additional news stories. Although the disagreement was uncomfortable to say the least, it only served to strengthen our credibility. 

Go out of your way to show respect for those who you’ve had disagreements with. You never know when they might champion your evaluation message in the future. Over the years, every policymaker mentioned in the news stories has agreed more than disagreed with our findings and recommendations. Remember—those who you had disagreements with are also doing their job, likely with the same passion as you do, and addressing the needs and demands of their constituents and stakeholders. Most of the time, disagreements are not personal – and even if they may seem personal, you should treat the people involved with total respect. This goes a long way toward building relationships and establishing credibility as independent evaluators who do not have a dog in the fight.

Final Thoughts

Acting on political knowledge is complex and potentially dangerous to evaluators’ credibility. However, by acknowledging the fact that all evaluations are inherently political and adhering to the four suggestions mentioned above, you will be able to manage the extra politics that comes with almost every education evaluation.

[i] K-12 public education evaluations are henceforth referred to as education evaluations.

[ii] Evaluator Advocacy: It Is All in a Day’s Work. American Journal of Evaluation, September 2014, 397-403.