Indaba is an isiZulu and isiXhosa word for an important meeting held by leaders in South Africa to discuss critical matters. This past week, I’ve been listening in at the Africa Evaluation Indaba, organized by The University of the Witwatersrand and CLEAR Anglophone Africa. Critical matters have indeed been discussed!
At the upcoming session, we will discuss ways that M&E practitioners can improve data management and how they can play a role in improving data governance practices at the institutional and national levels. We will open the floor for discussion and consultation on priority areas and gaps in the practical aspects of responsible data management as well as in data governance processes that improve accountability.
Following the Indaba, will draft up a plan that lays out how we can best offer training, guidance and support for the M&E community with relation to responsible data management and data governance. We also plan to develop a set of guidance documents on responsible data governance and M&E together with the RDiME Working Group, which is made up of a group of experts who have data governance, data protection, and evaluation-related expertise and experience. We hope the RDiME Alliance’s work will support government evaluation efforts as well as civil society organizations and evaluation firms.
What makes the Africa Indaba Evaluation conversations so exciting (for me, at least!) is that they are framed within a lens of decolonization and transformation. This past week, topics included:
“Transforming Evaluation: The Race, Power, Gender and Class Struggle,” with speakers covering questions like: how do we locate evaluation within the historical context of asymmetrical global power relations and aid dependency? What needs to be done to dismantle systems and structures so that evaluation does not become complicit in entrenching existing inequalities? (Monday 16 November)
The Made in Africa (MAE) Evaluation approachwhich arose out of the quest for contextually relevant approaches, methods that emphasize the centrality of contextual relevance and place importance on indigenous knowledge systems. (Tuesday 17 November)
The launch of the Global Evaluation Initiative, (GEI) which aims to offer better coordination of evaluation resources and to support local and international organizations working in the area of evaluation. (Wednesday 18 November)
(Recordings of these sessions will be available soon).
Guest post from Janna Rous. Original published here.
So, all of a sudden you’re stuck at home because of the new coronavirus. You’re looking at your M&E commitments and your program commitments. Do you put them all on hold and postpone them until the coronavirus threat has passed and everything goes back to normal? Or is there a way to still get things done”!? This article reviews 8 ways you can adapt your M&E during the pandemic.
Here are a few ideas that you and your team might consider doing to make sure you can stay on track (and maybe even IMPROVE your MEAL practices) even if you might currently be in the middle of a lockdown, or if you think you might be going into a lockdown soon:
1. Phone Call Interviews instead of In-Person Interviews
Do you have any household assessments or baseline surveys or post-distribution monitoring that you had planned in the next 1 to 3 months? Is there a way that you can carry out these interviews by phone or WhatsApp calls? This is the easiest and most direct way to carry on with your current M&E plan. Instead of doing these interviews face-to-face, just get them on a call. I’ve created a checklist to help you prepare for doing phone call interviews – click here to get the “Humanitarian’s Phone Call Interview Checklist”. Here are a few things you need to think through to transition to a phone-call methodology:
You need phone numbers and names of people that need to be surveyed. Do you have these? Or is there a community leader who might be able to help you get these?
You also need to expect that a LOT of people may not answer their phone. So instead of “sampling” people for a survey, you might want to just plan on calling almost everyone on that list.
Just like for a face-to-face interview, you need to know what you’re going to say. So you need to have a script ready for how you introduce yourself and ask for consent to do a phone questionnaire. It’s best to have a structured interview questionnaire that you follow for every phone call, just like you would in a face-to-face assessment.
You also need to have a way to enter data as you ask the questions. This usually depends on what you’re most comfortable with – but I recommend preparing an ODK or KoboToolbox questionnaire, just like you would for an in-person survey, and filling it out as you do the interview over the phone. I find it easiest to enter the data into KoboToolbox “Webform” instead of the mobile app, because I can type information faster into my laptop rather than thumb-type it into a mobile device. But use what you have!
If you’re not comfortable in KoboToolbox, you could also prepare an Excel sheet for directly entering answers – but this will probably require a lot more data cleaning later on.
When you’re interviewing, it’s usually faster to type down the answers in the language you’re interviewing in. If you need your final data collection to be in English, go back and do the translation after you’ve hung up the phone.
If you want a record of the interview, ask if you can record the phone call. When the person says yes, then just record it so you can go back and double check an answer if you need to.
Very practically – if you’re doing lots of phone calls in a day, it is easier on your arm and your neck if you use a headset instead of holding your phone to your ear all day!
2. Collect Videos & Photos Directly from Households and Communities
When you’re doing any in-person MEAL activities, you’re always able to observe evidence. You can look around and SEE impact, you don’t just hear it through an interview or group discussion. But when you’re doing M&E remotely, you can’t double-check to see what impact really looks like. So I recommend:
Connect with as many beneficiaries and team members as possible through WhatsApp or another communication app and collect photos and videos of evidence directly from them.
Video – Maybe someone has a story of impact they can share with you through video. Or if you’re overseeing a Primary Health Care clinic, perhaps you can have a staff member walk you through the clinic with a video so you can do a remote assessment.
Pictures – Maybe you can ask everyone to send you a picture of (for example) their “hand washing station with soap and water” (if you’re monitoring a WASH program). Or perhaps you want evidence that the local water point is functioning.
3. Programme Final Evaluation
It’s a good practice to do a final evaluation review when you reach the end of a program. If you have a program finishing in the next 1-3 months, and you want to do a final review to assess lessons learned overall, then you can also do this remotely!
Make a list of all the stakeholders that would be great to talk to: staff members, a few beneficiaries, government authorities (local and/or national), other NGOs, coordination groups, partner organizations, local community leaders.
Then go in search of either their phone numbers, their email addresses, their Skype accounts, or their WhatsApp numbers and get in touch.
It’s best if you can get on a video chat with as many of them as possible – because it’s much more personal and easy to communicate if you can see one another’s faces! But if you can just talk with audio – that’s okay too.
Prepare a semi-structured interview, a list of questions you want to talk through about the impact, what went well, what could have gone better. And if there’s anything interesting that comes up, don’t worry about coming up with some new questions on the spot or skipping questions that don’t make sense in the context.
You can also gather together any monitoring reports/analysis that was done on the project throughout its implementation period, plus pictures of the interventions.
Use all this information to create a final “lessons learned” evaluation document. This is a fantastic way to continually improve the way you do humanitarian programming.
4. Adapt Your Focus Group Discussion Plan
If everyone is at home because your country has imposed a lockdown, it will be very difficult to do a focus group discussion because….you can’t be in groups! So, with your team decide if it might be better to switch your monitoring activity from collecting qualitative data in group discussions to actually just having one-on-one interviews on the phone with several people to collect the same information.
There are some dynamics that you will miss in one-to-one interviews, information that may only come out during group discussions. (Especially where you’re collecting sensitive or “taboo” data.) Identify what that type of information might be – and either skip those types of questions for now, or brainstorm how else you could collect the information through phone-calls.
5. Adapt Your Key Informant Interviews
If you normally carry out Key Informant Interviews, it would be a great idea to think what “extra” questions you need to ask this month in the midst of the coronavirus pandemic.
If you normally ask questions around your program sector areas, think about just collecting a few extra data points about feelings, needs, fears, and challenges that are a reality in light of Covid-19. Are people facing any additional pressures due to the epidemic? Or are there any new humanitarian needs right now? Are there any upcoming needs that people are anticipating?
It goes without saying that if your Key Informant Interviews are normally in person, you’ll want to carry these out by phone for the foreseeable future!
6. What To Do About Third Party Monitoring
Some programs and donors use Third Party Monitors to assess their program results independently. If you normally hire third party monitors, and you’ve got some third party monitoring planned for the next 1-3 months, you need to get on the phone with this team and make a new plan. Here are a few things you might want to think through with your third party monitors:
But also think through – is it worth it to get a third party monitor to assess results remotely? Is it better to postpone their monitoring? Or is it worth it to carry on regardless?
What is the budget implication? If cars won’t be used, is there any cost-savings? Is there any additional budget they’ll need for air-time costs for their phones?
Make sure there is a plan to gather as much photo and video evidence as possible (see point 2 above!)
If they’re carrying out phone call interviews it would also be a good recommendation to record phone calls if possible and with consent, so you have the records if needed.
7. Manage Expectations – The Coronavirus Pandemic May Impact Your Program Results.
You probably didn’t predict that a global pandemic would occur in the middle of your project cycle and throw your entire plan off. Go easy on yourself and your team! It is most likely that the results you’d planned for might not end up being achieved this year. Your donors know this (because they’re probably also on lockdown). You can’t control the pandemic, but you can control your response. So proactively manage your own expectations, your manager’s expectations and your donor’s expectations.
Get on a Skype or Zoom call with the project managers and review each indicator of your M&E plan. In light of the pandemic, what indicator targets will most likely change?
Look through the baseline numbers in your M&E plan – is it possible that the results at the END of your project might be worse than even your baseline numbers? For example, if you have a livelihoods project, it is possible that income and livelihoods will be drastically reduced by a country-wide lockdown. Or are you running an education program? If schools have been closed, then will a comparison to the baseline be possible?
Once you’ve done a review of your M&E plan, create a very simple revised plan that can be talked through with your program donor.
8. Talk To Your Donors About What You Can Do Remotely
When you’re on the phone with your donors, don’t only talk about revised program indicators.
Also talk about a revised timeframe – is there any flexibility on the program timeframe, or deadlines for interim reporting on indicators? What are their expectations?
Also talk about what you CAN do remotely. Discuss with them the plan you have for carrying on everything possible that can be done remotely.
And don’t forget to discuss financial implications of changes to timeframe.
This is a summary of a Lightning Talk presented by Salla Mankinen, Good Return, at MERL Tech London in 2017.
When collecting data from the most vulnerable target groups, organizations often use methods such as guesstimating, interviewing done by enumerators, SMS, or IVR. The organization Good Return created a smart phone and tablet app that allowed vulnerable groups to interact directly with the data collection tool, without training or previous exposure to any technology.
At MERL Tech London in February 2017, Salla Mankinen shared Good Return’s experiences with using tablets for self-service check in at village training centers in Cambodia.
“Our challenge was whether we could have app-based, self-service data collection for the most vulnerable and in the most remote locations,” she said. “And could there be a journey from technology illiteracy to technology confidence” in the process?
The team created a voice and image based application that worked even for those who had little technology knowledge. It collected data from village participants such as “Why did you miss the last training session?” or “Do you have any money left this week?”
By the end of the exercise, 72% of participants felt confident with the app and 83% said they felt a lot more confident with technology in general.
How can we assess progress on a second-generation way of working in the transparency, accountability and participation (TAP) field? Monitoring, evaluation, research, and learning (MERL) maturity models can provide some inspiration. The 2017 MERL Tech conference in Washington, DC was a two-day bonanza of lightening talks, plenary sessions, and hands-on workshops among participants who use technology for MERL.
Here are key conference takeaways from two MEL practitioners in the TAP field.
1. Making open data useful
Several MERL Tech sessions resonated deeply with the TAP field’s efforts to transition from fighting for transparent and open data towards linking open data to accountability and governance outcomes. Development Gateway and InterAction drew on findings from “Avoiding Data Graveyards” as we explored progress and challenges for International Aid Transparency Initiative (IATI) data use cases. While transparency is an important value, what is gained (or lost) in data use for collaboration when there are many different potential data consumers?
And finally, as TAP practitioners are keenly aware, power and politics can overshadow evidence in decision making. Another Development Gateway presentation reminded us that it is important to work with data producers and users to identify decisions that are (or can be) data-driven, and to recognize when other factors are driving decisions. (The incentives to supply open data is whole other can of worms!)
Drawing on our second-generation TAP approach, more work is needed for the TAP and MERL fields to move from “open data everywhere, all of the time” to planning for, and encouraging more effective data use.
2. Tech for MERL for improved policy, practice, and outcomes
Among our favorite moments at MERL Tech was when Dobility Founder and CEO Christopher Robert remarked that “the most interesting innovations at MERL Tech aren’t the new, cutting-edge technology developments, but generic technology applied in innovative ways.” Unsurprising for a tech company focused on using affordable technology to enable quality data collection for social impact, but a refreshing reminder amidst the talk of ‘AI’, ‘chatbots’, and ‘blockchains’ for development coursing through the conference.
We are undoubtedly on the precipice of revolutionary technological advancements that can be readily (and maybe even affordably) deployed to solve complex global challenges, but they will still be tools and not solutions.
3. Balancing inclusion and participation with efficiency and accuracy
We explored a constant conundrum for MERL: how to balance inclusion and participation with efficiency and accuracy. Girl Effect and Praekelt Foundation took “mixed methods” research to another level, combining online and offline efforts to understand user needs of adolescent girls and to support user-developed digital media content. Their iterative process showcased an effective way to integrate tech into the balancing act of inclusive – and holistic – design, paired with real-time data use.
This session on technology in citizen generated data brought to light two case studies of how tech can both help and hinder this balancing act. The World Café discussions underscored the importance of planning for – and recognizing the constraints on – feedback loops. And provided us a helpful reminder that MERL and tech professionals are often considering different “end users” in their design work!
So, which is it – balancing act or zero-sum game between inclusion and efficiency? The MERL community has long applied participatory methods. And tech solutions abound that can help with efficiency, accuracy, and inclusion. Indeed, the second-generation TAP focus on learning and collaboration is grounded in effective data use – but there are many potential “end users” to consider. These principles and practices can force uncomfortable compromises – particularly in the face of finite resources and limited data availability – but they are not at odds with each other. Perhaps the MERL and TAP communities can draw lessons from each other in striking the right balance.
4. Tech sees no development sector silos
One of the things that makes MERL Tech such an exciting conference, is the deliberate mixing of tech nerds with MERL nerds. It’s pretty unique in its dual targeting of both types of professionals who share a common purpose of social impact (where as conferences like ICT4D cast a wider net looking at application of technology to broader development issues). And, though we MERL professionals like to think of design and iteration as squarely within our wheelhouse, being in a room full of tech experts can quickly remind you that our adaptation game has a lot of room to grow. We talk about user-centered design in TAP, but when the tech crowd was asked in plenary “would you even think of designing software or an app without knowing who was going to use it?” they responded with a loud and exuberant laugh.
Tech has long employed systematic approaches to user-centered design, prototyping, iteration, and adaptation, all of which can offer compelling lessons to guide MERL practices and methods. Though we know Context is King, it is exhilarating to know that the tech specialists assembled at the conference work across traditional silos of development work (from health to corruption, and everything in between). End users are, of course, crucial to the final product but the life cycle process and steps follow a regular pattern, regardless of the topic area or users.
The second-generation wave in TAP similarly moved away from project-specific, fragmented, or siloed planning and learning towards a focus on collective approaches and long-term, more organic engagement.
American Evaluation Association President, Kathy Newcomer, quipped that maybe an ‘Academy Awards for Adaptation’ could inspire better informed and more adept evolutions to strategy as circumstances and context shift around us. Adding to this, and borrowing from the tech community, we wonder where we can build more room to beta test, follow real demand, and fail fast. Are we looking towards other sectors and industries enough or continuing to reinvent the wheel?
Alison left thinking:
Concepts and practices are colliding across the overlapping MERL, tech, and TAP worlds! In leading the Transparency and Accountability Initiative’s learning strategy, and supporting our work on data use for accountability, I often find myself toggling between different meanings of ‘data’, ‘data users’, and tech applications that can enable both of these themes in our strategy. These worlds don’t have to be compatible all the time, and concepts don’t have to compute immediately (I am personally still working out hypothetical blockchain applications for my MERL work!). But this collision of worlds is a helpful reminder that there are many perspectives to draw from in tackling accountable governance outcomes.
Maturity models come in all shapes and sizes, as we saw in the creative depictions created at MERL Tech that included, steps, arrows, paths, circles, cycles, and carrots! And the transparency and accountability field is collectively pursuing a next generation of more effective practice that will take unique turns for different accountability actors and outcomes. Regardless of what our organizational or programmatic models look like, MERL Tech reminded me that champions of continuous improvement are needed at all stages of the model – in MERL, in tech for development, and in the TAP field.
Megan left thinking:
That I am beginning to feel like I’m a Dr. Seuss book. We talked ‘big data’, ‘small data’, ‘lean data’, and ‘thick data’. Such jargon-filled conversations can be useful for communicating complex concepts simply with others. Ah, but this is also the problem. This shorthand glosses over the nuances that explain what we actually mean. Jargon is also exclusive—it clearly defines the limits of your community and makes it difficult for newcomers. In TAP, I can’t help but see missed opportunities for connecting our work to other development sectors. How can health outcomes improve without holding governments and service providers accountable for delivering quality healthcare? How can smallholder farmers expect better prices without governments budgeting for and building better roads? Jargon is helpful until it divides us up. We have collective, global problems and we need to figure out how to talk to each other if we’re going to solve them.
In general, I’m observing a trend towards organic, participatory, and inclusive processes—in MERL, in TAP, and across the board in development and governance work. This is, almost universally speaking, a good thing. In MERL, a lot of this movement is a backlash to randomistas and imposing The RCT Gold Standard to social impact work. And, while I confess to being overjoyed that the “RCT-or-bust” mindset is fading out, I can’t help but think we’re on a slippery slope. We need scientific rigor, validation, and objective evidence. There has to be a line between ‘asking some good questions’ and ‘conducting an evaluation’. Collectively, we are working to eradicate unjust systems and eliminate poverty, and these issues require not just our best efforts and intentions, but workable solutions. Listen to Freakonomic’s recent podcast When Helping Hurts and commit with me to find ways to keep participatory and inclusive evaluation techniques rigorous and scientific, too.
By Claire Benard, formerly of Crisis UK and now with National Council for Voluntary Organizations (NCVO).
Most people who work with data in MERL will have heard of R. Some people will have been properly introduced to it, but only a few will invest the necessary time in learning how to use it. Being a relatively late convert, I wanted to share my experience of moving from a traditional data analysis software package to a language based one, so I did a Lightning Talk at MERL Tech London. (You can watch the video below.)
First things first, what is R?
Aside from being the 18th letter of the alphabet, R is also a language and environment for statistical computing and graphics.
But wait, you say… why should I use it?
This is what the five-minute video below is about, but in short, here are a few reasons:
There is nothing your current software package does that R doesn’t do.
R is free.
Using a programming language makes the analysis easy to reproduce, whether it’s because you need to produce similar analysis year on year or because you have a team of analysts who need to collaborate and understand each other’s work.
R is an open source technology. People from all backgrounds contribute to it and make new tools available for free regularly. This is you’re insurance to stay at the cutting edge of what is being developed.
Well, then, how do I get started? you wonder…
If you’re more MERL than Tech, learning a new programming language can be daunting. There is a time and money cost to it and it’s hard to know where to start if you’re on your own.
In the video, I give a few tips. It’s also worth checking out free/cheap training online (for example here or here) ; looking out for a user group near you and getting advice from blogs, forums and newsletters.
Post by Julia Connors of Voltaicsystems. Email Julia with questions: firstname.lastname@example.org
What is solar for M&E?
Solar technology can be extremely useful for M&E projects in areas with minimal or inconsistent access to power. Portable solar chargers can eliminate power constraints and keep phones and tablets reliably charged up in the field.
In this post we’ll discuss:
How to decide if solar is right for your project
How to properly size a solar charging system to meet your needs
Do you really need solar?
In many cases solar is not necessary and will simply add complexity and costs to your project. If your team can return every day to a central location with access to power, then the battery power of the tablet is sufficient in most scenarios. If not, we recommend implementing standard power saving tips to reduce power consumption during time out collecting data.
POWER SAVING TIPS
If you do have daily access to the grid but find that users need to recharge at least once while out or need to spend more than one day without power, then add an external battery pack. This cost-effective option allows your team to have extra power without carrying a full solar charging system. To size a battery for your needs, skip down to ‘Step 3’ below.
If you don’t have reliable access to grid power, the next section will help you determine which size solar charging system is best for you.
Sizing your solar charger system
The key to making solar successful in your project is finding the best system for your needs. If a system is underpowered then your team can still run out of power when they’re collecting data. On the other hand, if your system is too powerful it will be heavier and more expensive than needed. We recommend the following three steps for sizing your solar needs:
Estimate your daily power consumption
Determine your minimum solar panel size
Determine your minimum battery size
Step 1: Estimate your daily power consumption
Once you have chosen the device you will be using in the field, it’s easy to determine your daily power consumption. First you’ll need to figure out the size of your device’s battery (in Watt hours). This can often be found by looking on the back of the battery itself or doing a quick Google search to find your device’s technical specifications.
Next, you’ll need to determine your battery usage per day. For example, if you use half of your device’s battery on a typical day of data collection, then your usage is 50%. If you need to recharge twice in one day, then your usage is 200%.
Once you have those numbers, use the formula below to find your daily power consumption:
Size of Device’s Battery (Wh) x Battery Usage (per day) =
Daily Power Consumption (Wh/day)
Step 2: Determine your minimum solar panel size
The larger your device, the bigger the solar panel (measured in Watts) you’ll need. This is because larger solar panels can generate more power from the sun than smaller panels. To determine the best solar panel size for your needs, use our formula below:
Daily Power Consumption (from Step 1) / Expected Hours of Good Sun*
x 2 (Standard Power Loss Variable) =
Solar Panel Minimum (Watts)
*We typically use 5 hours as a baseline for good sun and then adjust up or down depending on the conditions. High temperatures, clouds, or shading will reduce the power produced by the panel.
Since solar conditions change frequently throughout the day, we recommend choosing a panel that is 2-4 times the minimum size required.
Step 3: Determine minimum battery size
External batteries offer extra power storage so that your device will be charged when you need it. The battery acts as a perfect backup on cloudy and rainy days so it’s important to choose the right size for your device.
It can vary, but typically about 30% of power is lost in the transfer from the external battery to your device. Therefore, to determine the battery capacity needed for one day of use, we’ll use our power consumption data from Step 1 and divide by 0.7 (100% – 30% power loss).
Watt hours per day / 0.7 hours =
Watt battery capacity needed for 1 day of use
Picking the right system for your project
Now that you’ve done the math, you’re one step closer to choosing a solar charging system for your project. Since solar chargers come in many different forms, the last step to determining your perfect system is to think about how your team will be using the solar chargers in their work. It’s important to factor in storage for device/cables and how the user will be carrying the system.
Most users aren’t that technical, so having a pack that stores the battery and the device can simplify their experience (rather than handing over a battery and a panel that they need to figure how to organize during their day). By simply finding the right style and size, you’ll experience higher usage rates and make your team’s solar-powered data collection go more smoothly.