Moving from “evaluation” to “impact management”
by Richa Verma, Resident Entrepreneur at Social Cops. This post originally appeared on the Social Cops blog on August 28, 2017.
When I say that Impact Evaluation is history, I mean it. Some people will question this. After all, Impact Evaluation just became mainstream in the last decade, driven by great improvements in experimental design methods like randomized control trials (RCTs). So how can I say that it’s already a thing of the past? It’s not Impact Evaluation’s fault. The world changed.
Methodologies like RCTs came from medical science, where you can give patients a pill and assess its impact with randomized trials. However, development is not a space where one pill will work for everyone. In development, the patients change faster, the illness evolves faster, and the pill needs to keep pace with both the patients and the illness. That’s where Impact Management comes in.
What Is Impact Management?
New Philanthropy Capital‘s 2017 Global Innovation in Measurement and Evaluation Report counts Impact Management as one of the top 7 innovations of 2017.
So what is Impact Management? Let me first explain what it is not. It’s not a one-time evaluation. It’s not collecting data for answering a limited set of questions. It’s not a separate activity from your program. It’s not just monitoring and evaluation.
It’s a way of making data-driven decisions at every step of your program. It’s about keeping a pulse on your program every day and finding new questions to answer, rather than just focusing on specific questions predetermined by your monitoring and evaluation team or funders.
“The question that’s being asked more and more is, ‘How does evaluation feed into better management decisions?’ That’s a shift from measurement of impact, to measurement for impact.”
– Megan Campbell (Feedback Labs)
How Does Impact Management Work?
Impact Management uses the basic components of monitoring and evaluation, but with an outlook shift. It involves frequent data collection, regular reporting and monitoring of your data, and iteratively updating your program indicators and metrics as data comes in and the program changes.
Impact Management differs from Impact Assessment in that it promotes course correction on a daily basis. Organizations collect data on their programs as they conduct activities, analyze that information on a regular basis, and make changes to the program.
With an outlook that encourages frequent changes, as if you were trading in stocks, organizations will have the ability to A/B test their programs with real-time data to make decisions immediately; rather than wait to compare and contrast two different surveys. They can test out new things and make changes as they receive data in servers, even at the end of the day rather than waiting for the official year-end review. It becomes a way of deciding how they should execute a program daily rather than only seeing strategic changes through.
“[Data collection] should be ongoing — it’s a value driver not a compliance requirement.”
– Tom Adams (Acumen)
In many ways, this is how decisions are made on Wall Street or Dalal Street in India. Analysts don’t wait until the end of the year to make investments by reviewing annual reports. They watch daily as the market fluctuates and strike as soon as they see new potential.
Impact Management works exactly the same. You should strive to increase your impact as soon as opportunity arrives, rather than waiting for a year-end external evaluation or approval.
How Can You Implement Impact Management?
To make Impact Management possible, switch from static data files to a flexible data system.
Today, most of your program officers and even your beneficiaries are armed with mini-computers in their pockets (read: smartphones). Leverage these to create a network of data ingestion devices, continuously tracking and measuring the impact of your programs. Use mobile data collection apps to add forms, deploy them to the field, and reach out not just to your field force but also your beneficiaries — not just at the end of the month or quarter, but as frequently as possible.
Then don’t let this data sit in Excel files. Use today’s technologies to create your own data management system, one that will link your beneficiaries, connect your programs, and answer queries. Have someone with an analytical bent look at this data regularly, or draw on machine power to analyze this data and generate meaningful insights or reports in real time.
“We’re moving away from a static data world, where you work on datasets, and you write reports, to a dynamic data world where data is always being generated and created and it helps you do your job better.”
– Andrew Means (beyond.uptake)
Lastly, it’s crucial to tie this flexible data system back to your decisions. Make real-time data — rather than guesses or last year’s data — the basis of every program decision and the foundation of even weekly catch-ups. And don’t hesitate to test out new things. Data will tell you whether something worked or not.
Many of our partners are using our platform to make Impact Management possible and track their programs in real time. The platform lets them create and tweak data collection forms, and monitor incoming data in real time on their computer, in regular reports, or even on map-based dashboards. They are asking new questions about how their programs are doing and answering them with data.
If we really want to create the best development programs, we’ll have to think differently and use evidence not just once every month or year, but as we make crucial decisions every day. All backed by the tenets of Impact Management: test, fail, improve, repeat.
Join us at MERL Tech London on March 19-20 – where we’ll be debating this topic!