Evaluating Transformations


Michael Quinn Patton, Director, Utilization-Focused Evaluation, USA; Steward – Transforming A&E Working Group, SDG Transformations Forum

The greatest danger in times of turbulence is not the turbulence – it is to act with yesterday’s logic.Peter Drucker (1909-2005), management consultant

Evaluating transformation is different from evaluating programmes. What we call in evaluation jargon the ‘evaluand’, the thing evaluated, determines the focus and methods of an evaluation. The evaluation profession’s origins were in evaluating projects and programmes, and, from my perspective, we remain in the grip of a self-limiting project mentality. Such tools as logic models and SMART goals work well for project and programme evaluation. They do not work well for evaluating transformational systems change.

Projects and programmes are closed systems. Boundaries are established and control is exercised by focusing on identifiable inputs, planned and implemented activities, expected outputs, and clear, specific, and measurable outcomes for targeted programme participants. The evaluation profession has developed methods, designs, and measures that answer these basic effectiveness questions validly, credibly, and usefully.

In contrast, interventions introduced into complex dynamic systems unfold in open systems characterized by volatility, uncertainty, and unpredictability, all of which make control problematic. For those designing and implementing systems change interventions, they must be innovative, adaptive, responsive, nimble, and agile. Evaluations under such conditions must be emergent, developmental, adaptable, dynamic, responsive, and networked. Using an inappropriate evaluation approach, one not well-matched to the nature and complexity of the situation, and intervention not only fails to generate meaningful findings but can do real and lasting harm.

If evaluators force complex systems change interventions into traditional project boxes aimed at standardization, predictability, and simple, linear attribution, they inhibit innovation, adaptation, and responsiveness, and thereby doom transformation-focused interventions to failure.

This isn’t just an evaluation problem. Forcing complex systems change interventions into traditional project boxes occurs among the full range of people and institutions attempting to bring about change. Repeatedly, as a participant in international conferences focused on Transformation, I have witnessed the challenges being framed as complex, multi-dimensional, multi-layered, cross-silos, and dynamic with full recognition of the necessity of being innovative and adaptive. Then, what follows are presentations on projects and programmes that are anything but transformational by their very conceptualization and accompanying evaluations.

The project mentality has thrived for a half-century. The project mindset is dominant in every sphere of change. The project approach is deeply embedded in institutional strategies. But if we have learned anything in 50 years of international development and corresponding project and programme evaluation, projects and programmes do not lead to transformation.

Let me pause to acknowledge that effective programmes help a great many people. Successful health programmes make people healthier and prevent disease. Effective school programmes increase student learning. Exemplary employment training programmes help the unemployed get jobs. Well-conceived and well-implemented programmes for the homeless get people in need off the streets and into safe housing. I am not disputing that effective programmes of all kinds achieve important and desired outcomes for intended beneficiaries. What they don’t do is transform systems. Indeed, my conclusion after observing effective and exemplary programmes over five decades of evaluation practice is that, when programmes are successful, it is often because they have succeeded in insulating themselves from the status-quo-serving systems that surround them. They create islands of protected and isolated effectiveness in a sea of need and suffering. They do good, meritorious good, significant good, worthy good, but they don’t do transformative good.

On the other hand, projects and programmes that are ineffective often fail because they’re not able to insulate themselves from the status-quo-serving systems that surround them, and of which they are a part. They are crushed in their attempts to innovate by the dominant forces in those systems that push back against and undermine their efforts at change.

Hyperbole? Overgeneralization? Perhaps. I invite you to look around and make your own assessment. How much transformation do you see going on? You will have no difficulty locating programmes and projects of all kinds, many effective, many ineffective, a mixed bag, to be sure. But what you won’t find much of, if any at all, is projects and programmes transforming systems. Which is why there is now a worldwide cry for transformative change. Evaluation must also be transformed to evaluate transformative change.

This blog was produced in partnership with the Royal Scottish Geographical Society.

Please share this blog generously and encourage people to sign up to receive Forum blogs weekly.

Share Post

Leave a Reply

Your email address will not be published. Required fields are marked *


A New Economic Orthodoxy Economics for Life


Systems Change Funding