All posts by Guest Post

M&E Squared: Evaluating M&E Technologies

by Roger Nathanial Ashby, Co-Founder & Principal Consultant, OpenWise

The universe of MERL Tech solutions has grown exponentially. In 2008 monitoring and evaluating tech within global development could mostly be confined to mobile data collection tools like Open Data Kit (ODK), and Excel spreadsheets to analyze and visualize survey data. In the intervening decade a myriad of tools, companies and NGOs have been created to advance the efficiency and effectiveness of monitoring, evaluation, research and learning (MERL) through the use of technology. Whether it’s M&E platforms or suites, satellite imagery, remote sensors, or chatbots, new innovations are being deployed every day in the field.

However, how do we evaluate the impact when MERL Tech is the intervention itself? That was the question and task put to participants of the “M&E Squared” workshop at MERL Tech 2017.

Workshop participants were separated into three groups that were each given a case study to discuss and analyze. One group was given a case about improving the learning efficiency of health workers in Liberia through the mHero Health Information System (HIS). The system was deployed as a possible remedy to some of the information communication challenges identified during the 2014 West African Ebola outbreak. A second group was given a case about the use of RapidPro to remind women to attend antenatal care (ANC) for preventive malaria medicine in Guinea. The USAID StopPalu project goal was to improve the health of infants by increasing the percent of women attending ANC visits. The final group was given a case about using remote images to assist East African pastoralists. The Satellite Assisted Pastoral Resource Management System (SAPARM) informs pastoralists of vegetation through remote sensing imagery so they can make better decisions about migrating their livestock.

After familiarizing ourselves with the particulars of the case studies, each group was tasked to present their findings to all participants after pondering a series of questions. Some of the issues under discussion included

(1) “How would you assess your MERL Tech’s relevance?”

(2) “How would you evaluate the effectiveness of your MERL Tech?”

(3) “How would you measure efficiency?” and

(4) “How will you access sustainability?”.

Each group came up with some innovative answers to the questions posed and our facilitators and session leads (Alexandra Robinson & Sutyajeet Soneja from USAID and Molly Chen from RTI) will soon synthesize the workshop findings and notes into a concise written brief for the MERL Tech community.

Before the workshop closed we were all introduced to the great work done by SIMLab (Social Impact Lab) in this area through their SIMLab Monitoring and Evaluation Framework. The framework identifies key criteria for evaluating M&E including:

  1. Relevance – The extent to which the technology choice is appropriately suited to the priorities and capacities of the context of the target group or organization.
  2. Effectiveness – A measure of the extent to which an information and communication channel, technology tool, technology platform, or a combination of these attains its objectives.
  3. Efficiency – Measure of the outputs (qualitative and quantitative) in relation to the inputs.
  4. Impact – The positive and negative changed produced by technology introduction, change in a technology tool, or platform on the overall development intervention (directly or indirectly; intended or unintended).
  5. Sustainability – Measure of whether the benefits of a technology tool or platform are likely to continue after donor funding has been withdrawn.
  6. Coherence – How related is the technology to the broader policy context (development, market, communication networks, data standards & interoperability mandates, and national & international law) within which the technology was developed and implemented.

While it’s unfortunate that SIMLab stopped most operations in early September 2017, their exceptional work in this and other areas lives on and you can access the full framework here.

I learned a great deal in this session from the facilitators and my colleagues attending the workshop. I would encourage everyone in the MERL Tech community to take the ideas generated during this workshop and the great work done by SIMLab into their development practice. We certainly intend to integrate much of these insights into our work at OpenWise. Read more about “The Evidence Agenda” here on SIMLab’s blog. 

 

 

 

Making (some) sense of data storage and presentation in Excel

By Anna Vasylytsya. Anna is in the process of completing her Master’s in Public Policy with an emphasis on research methods. She is excited about the role that data can play in improving people’s lives!

At the MERL Tech Conference, I attended a session called “The 20 skills that solve 80% of M&E problems” presented by Dr. Leslie Sage of DevResults. I was struck by the practical recommendations Leslie shared that can benefit anyone that uses Excel to store and/or present data.

I boiled down the 20 skills presented in the session into three key takeaways, below.

1. Discerning between data storage and data presentation

Data storage and data presentation serve two different functions and never the two shall meet. In other words, data storage is never data presentation.

Proper data storage should not contain merged cells, subheadings, color used to denote information, different data types within cells (numbers and letters), more than one piece of data in a cell (such as disaggregations). Additionally, in proper data storage, columns should be the variables and rows as the observations or vice versa. Poor data storage practices need to be avoided because they mean that you cannot use Excel’s features to present the data.

A common example of poor data storage:

Excel 1

 

One of the reasons that this is not good data storage is because you are not able to manipulate this data using Excel’s features. If you needed this data in a different format or you wanted to visualize it, you would have to do this manually, which would be time consuming.

Here is the same data presented in a “good” storage format:

2Good_Data_Storage

 

Data stored this way may not look as pretty, but it is not meant to be presented or read in within the sheet. This is an example of good data storage because each unique observation gets a new row in the spreadsheet. When you properly store data, it is easy for Excel to aggregate the data and summarize it in a pivot table, for example.

2. Use Excel’s features to organize and clean data

You do not have to use precious time to organize or clean data manually. Here are a few recommendations on Excel’s data organization and cleaning features:

  • To join to cells that have text into one cell, use the concatenate function.
  • To split text from one cell into different cells, use the text to columns
  • To clean text data, use Excel’s functions: trim, lower, upper, proper, right, left, and len.
  • To move data from rows into columns or columns into rows, use Excel’s transpose feature.
  • There is a feature to remove duplicates from the data.
  • Create a macro to automate simple repetitive steps in Excel.
  • Insert data validation in an excel spreadsheet if you are sending a data spreadsheet to implementers or partners to fill out.
    • This restricts the type of data or values that can be entered in certain parts of the spreadsheet.
    • It also saves you time from having to clean the data after you receive it.
  • Use the vlookup function in Excel in your offline version to look up a Unique ID
    • Funders or donors normally require that data is anonymized if it is made public. While not the best option for anonymizing data, you can use Excel if you haven’t been provided with specific tools or processes.
    • You can create an “online” anonymized version that contains a Unique ID and an “offline version” (not public) containing the ID and Personally Identifiable Information (PII). Then, if you needed to answer a question about a Unique ID, for example, your survey was missing data and you needed to go back and collect it, you can use vlookup to find a particular record.

3. Use Excel’s features to visualize data

One of the reasons to organize data properly so that you can use Excel’s Pivot Table feature.

Here is an example of a pivot table made from the data in the good data storage example above (which took about a minute to make):

3Pivot_Table

Using the pivot table, you can then use Excel’s Create a Chart Feature to quickly make a bar graph:

4BarGraph

In the Future

I have fallen prey to poor data storage practices in the past. Now that I have learned these best practices and features of Excel, I know I will improve my data storage and presentation practices. Also, now that I have shared them with you; I hope that you will too!

Please note that in this post I did not discuss how Excel’s functions or features work or how to use them. There are plenty of resources online to help you discover and explore them. Some helpful links have been included as a start. Additionally, the data presented here are fictional and created purely for demonstration purposes.

Qualitative Coding: From Low Tech to High Tech Options

by Daniel Ramirez-Raftree, MERL Tech volunteer

In their MERL Tech DC session on qualitative coding, Charles Guedenet and Anne Laesecke from IREX together with Danielle de Garcia of Social Impact offered an introduction to the qualitative coding process followed by a hands-on demonstration on using Excel and Dedoose for coding and analyzing text.

They began by defining content analysis as any effort to make sense of qualitative data that takes a volume of qualitative material and attempts to identify core consistencies and meanings. More concretely, it is a research method that uses a set of procedures to make valid inferences from text. They also shared their thoughts on what makes for a good qualitative coding method.

Their belief is that: it should

  • consider what is already known about the topic being explored
  • be logically grounded in this existing knowledge
  • use existing knowledge as a basis for looking for evidence in the text being analyzed

With this definition laid out, they moved to a discussion about the coding process where they elaborated on four general steps:

  1. develop codes and a codebook
  2. decide on a sampling plan
  3. code your data
  4. go back and do it again!
  5. test for reliability

Developing codes and a codebook is important for establishing consistency in the coding process, especially if there will be multiple coders working on the data. A good way to start developing these codes is to consider what is already known. For example, you can think about literature that exists on the subject you’re studying. Alternatively, you can simply turn to the research questions the project seeks to answer and use them as a guide for creating your codes. Beyond this, it is also useful to go through the content and think about what you notice as you read. Once a codebook is created, it will lend stability and some measure of objectivity to the project.

The next important issue is the question of sampling. When determining sample size, though a larger sample will yield more robust results, one must of course consider the practical constraints of time, cost and effort. Does the benefit of higher quality results justify the additional investment? Fortunately, the type of data will often inform sampling. For example, if there is a huge volume of data, it may be impossible to analyze it all, but it would be prudent to sample at least 30% of it. On the other hand, usually interview and focus group data will all be analyzed, because otherwise the effort of obtaining the data would have gone to waste.

Regarding sampling method, session leads highlighted two strategies that produce sound results. One is systematic random sampling and the other is quota sampling–a method employed to ensure that the proportions of demographic group data are fairly represented.

Once these key decisions have been made, the actual coding can begin. Here, all coders should work from the same codebook and apply the codes to the same unit of analysis. Typical units of analysis are: single words, themes, sentences, paragraphs, and items (such as articles, images, books, or programs). Consistency is essential. A way to test the level of consistency is to have a 10% overlap in the content each coder analyzes and aim for 80% agreement between their coding of that content. If the coders are not applying the same codes to the same units this could either mean that they are not trained properly or that the code book needs to be altered.

Along a similar vein, the fourth step in the coding process is to test for reliability. Challenges in producing stable and consistent results in coding could include: using a unit of analysis that is too large for a simple code to be reliably applied, coding themes or concepts that are ambiguous, and coding nonverbal items. For each of these, the central problem is that the units of analysis leave too much room for subjective interpretation that can introduce bias. Having a detailed codebook can help to mitigate against this.

After giving an overview of the coding process, the session leads suggested a few possible strategies for data visualization. One is to use a word tree, which helps one look at the context in which a word appears. Another is a bubble chart, which is useful if one has descriptive data and demographic information. Thirdly, correlation maps are good for showing what sorts of relationships exist among the data. The leads suggested visiting the website stephanieevergreen.com/blog for more ideas about data visualization.

Finally, the leads covered low-tech and high-tech options for coding. On the low-tech end of the spectrum, paper and pen get the job done. They are useful when there are few data sources to analyze, when the coding is simple, and when there is limited tech literacy among the coders. Next up the scale is Excel, which works when there are few data sources and when the coders are familiar with Excel. Then the session leads closed their presentation with a demonstration of Dedoose, which is a qualitative coding tool with advanced capabilities like the capacity to code audio and video files and specialized visualization tools. In addition to Dedoose, the presenters mentioned Nvivo and Atlas as other available qualitative coding software.

Despite the range of qualitative content available for analysis, there are a few core principles that can help ensure that it is analyzed well, these include consistency and disciplined methodology. And if qualitative coding will be an ongoing part of your organization’s operations, there are several options for specialized software that are available for you to explore. [Click here for links and additional resources from the session.]

Being data driven… can it be more than a utopia?

by Emily Tomkys Valteri, the ICT in Program Accountability Project Manager at Oxfam GB. In her role, Emily drives Oxfam’s thinking on the use of Information and Communications Technologies (ICT) for accountability and supports staff with applications of ICTs within their work. 

Every day the human race generates enough data to fill 10 million blu-ray discs and if you stacked them up it would be four times the height of the Eiffel tower. Although the data we process at Oxfam is tiny in comparison, sometimes the journey towards being “data driven” feels like following the yellow brick road to The Emerald City. It seems like a grand ideal, but for anyone who knows the film, inflated expectations are set to be dashed. Does data actually help organisations like Oxfam better understand the needs of communities affected by disaster or poverty? Or do we need to pull back the curtain and manage our expectations about getting the basics right? When there are no ruby slippers, we need to understand what it is we can do with data and improve the way data is managed and analysed across countries and projects.

The problem

Oxfam works in over 90 countries using a variety of different data management and analysis tools that are developed or purchased in country. In the past, we have experimented with software licenses and database expertise, but we have started aiming for a more joined up approach. It’s our belief that good systems which build in privacy by design can help us stay true to values in our rights based Responsible Program Data Policy and Information Systems Data Security guidelines – which are about treating those people whom data is about with dignity and respect.

One of our most intractable challenges is that Oxfam’s data is analysed in system silos. Data is usually collected and viewed through a project level lens. Different formats and data standards make it difficult to compare across countries, regions or even globally. When data remains in source systems, trying to analyse between different systems is long and manual, meaning that any meta analysis is rarely done. One of the key tenants of Responsible Data is to only collect data you can use and to make the most of that information to effectively meet people’s needs. Oxfam collects a lot of valuable data and we think we need to do more with it: analyse more efficiently, effectively, at national and beyond level to drive our decision making in our programmes.

The solution

In response, Oxfam has begun creating the DataHub: a system which integrates programme data into a standard set of databases and presents it to a reporting layer for analysis. It bakes in principles of privacy and compliance with new data protection laws by design. Working with our in-house agile software development team we conducted four tech sprints, each lasting two weeks. Now we have the foundations. One of our standard data collection tools, SurveyCTO, is being pushed via a webhook into our unstructured database, Azure Cosmos DB. Within this database, the data is organised into collections, currently set up by country. From here, the data can be queried using Power BI and presented to programme teams for analysis. Although we only have one source system into quantitative analysis for now, the bigger picture will have lots of source systems and a variety of analysis options available.

To get to where we are today, Oxfam’s ICT in Programme team worked closely with the Information Systems teams to develop a solution that was in line with strategy and future trends. Despite the technology being new to Oxfam, the solution is relatively simple and we ensured good process, interoperability and that tools available to us were fit for purpose. This collaborative approach gave us the organisational support to prioritise these activities as well as the resources required to carry them out.

This journey wasn’t without its challenges, some of which are still being worked on. The EU General Data Protection Regulation (GDPR) coming into force in May 2018, and Oxfam has had to design the DataHub with this in mind. At this stage, data is anonymised during integration and so no Personally Identifiable Information (PII) enters the DataHub due to a series of configurations and processes we have put in place. Training and capacity is another challenge, we need to encourage a culture of valuing the data. This will only be of benefit to teams and the organisation if they make use of the system, investing time and resources to learning it.

We are excited about the potential of the DataHub and the success we have already had in setting up the infrastructure to enable more efficient data analysis and more responsive programming as well as save resources. We are keen to work with and share ideas with others. We know there is a lot of work ahead to foster a data driven organisation but we’re starting to feel, with the right balance of technology, process and culture it’s more realistic than we might have first hoped.

 

 

You can’t have Aid…without AI: How artificial intelligence may reshape M&E

by Jacob Korenblum, CEO of Souktel Digital Solutions

Photo: wikipedia.org/

Potential—And Risk

The rapid growth of Artificial Intelligence—computers behaving like humans, and performing tasks which people usually carry out—promises to transform everything from car travel to personal finance. But how will it affect the equally vital field of M&E? As evaluators, most of us hate paper-based data collection—and we know that automation can help us process data more efficiently. At the same time, we’re afraid to remove the human element from monitoring and evaluation: What if the machines screw up?

Over the past year, Souktel has worked on three areas of AI-related M&E, to determine where new technology can best support project appraisals. Here are our key takeaways on what works, what doesn’t, and what might be possible down the road.

Natural Language Processing

For anyone who’s sifted through thousands of Excel entries, natural language processing sounds like a silver bullet: This application of AI interprets text responses rapidly, often matching them against existing data sets to find trends. No need for humans to review each entry by hand! But currently, it has two main limitations: First, natural language processing works best for sentences with simple syntax. Throw in more complex phrases, or longer text strings, and the power of AI to grasp open-ended responses goes downhill. Second, natural language processing only works for a limited number of (mostly European) languages—at least for now. English and Spanish AI applications? Yes. Chichewa or Pashto M&E bots? Not yet. Given these constraints, we’ve found that AI apps are strongest at interpreting basic misspelled answer text during mobile data collection campaigns (in languages like English or French). They’re less good at categorizing open-ended responses by qualitative category (positive, negative, neutral). Yet despite these limitations, AI can still help evaluators save time.

Object Differentiation

AI does a decent job of telling objects apart; we’ve leveraged this to build mobile applications which track supply delivery more quickly & cheaply. If a field staff member submits a photo of syringes and a photo of bandages from their mobile, we don’t need a human to check “syringes” and “bandages” off a list of delivered items. The AI-based app will do that automatically—saving huge amounts of time and expense, especially during crisis events. Still, there are limitations here too: While AI apps can distinguish between a needle and a BandAid, they can’t yet tell us whether the needle is broken, or whether the BandAid is the exact same one we shipped. These constraints need to be considered carefully when using AI for inventory monitoring.

Comparative Facial Recognition

This may be the most exciting—and controversial—application of AI. The potential is huge: “Qualitative evaluation” takes on a whole new meaning when facial expressions can be captured by cameras on mobile devices. On a more basic level, we’ve been focusing on solutions for better attendance tracking: AI is fairly good at determining whether the people in a photo at Time A are the same people in a photo at Time B. Snap a group pic at the end of each community meeting or training, and you can track longitudinal participation automatically. Take a photo of a larger crowd, and you can rapidly estimate the number of attendees at an event.

However, AI applications in this field have been notoriously bad at recognizing diversity—possibly because they draw on databases of existing images, and most of those images contain…white men. New MIT research has suggested that “since a majority of the photos used to train [AI applications] contain few minorities, [they] often have trouble picking out those minority faces”. For the communities where many of us work (and come from), that’s a major problem.

Do’s and Don’ts

So, how should M&E experts navigate this imperfect world? Our work has yielded a few “quick wins”—areas where Artificial Intelligence can definitely make our lives easier: Tagging and sorting quantitative data (or basic open-ended text), simple differentiation between images and objects, and broad-based identification of people and groups. These applications, by themselves, can be game-changers for our work as evaluators—despite their drawbacks. And as AI keeps evolving, its relevance to M&E will likely grow as well. We may never reach the era of robot focus group facilitators—but if robo-assistants help us process our focus group data more quickly, we won’t be complaining.

Visualizing what connects us: Social Network Analysis (SNA) in M&E

by Anne Laesecke (IREX) and Danielle de García (Social Impact). This post also appears on the Social Impact blog and  the IREX blog.

SNA, or Social Network Analysis, continues to gain momentum in the M&E space. This year at MERL Tech, we held an SNA 101 session, giving a quick-and-dirty overview of what it is, how it can contribute to M&E, and useful tips and tools for conducting an SNA. If you missed it, here’s what you need to know:

What is SNA?

SNA is a way to analyze social systems through relationships. Analyzing and visualizing networks can reveal critical insights for understanding relationships between organizations, supply chains; social movements; and/or between individuals. It’s a very versatile tool which can be used throughout the program cycle to measure things like trust and social capital, information flows, resources, collaboration, and disease spread, among other things.

SNA uses a different vocabulary than other types of analyses. For example, the relationships we measure are called ties or links, and the entities that make up a network are called nodes or actors. These can be organizations, people, or even whole networks themselves. We can study nodes more closely by looking at their attributes – things that characterize them (like demographic information), and we can learn more about how nodes interact and cluster by studying communities or modalities within networks. Various measures of the roles nodes play in a network, as well as measures that characterize the networks themselves, can reveal a lot about the systems and hidden relationships at play. For example, we can determine who has the most ties with other actors; who is relatively cut off from the network, or who is connected to the most well-connected actors.

Why would you use SNA for M&E?

The term “social network analysis” often triggers associations with social media, but SNA uses data from a variety of platforms (including but not limited to social media!). For instance, SNA can identify key influencers in systems – important for programs that rely on thinking politically. SNA can also be a useful tool in processing big data with applications for cybersecurity as well as creating biological and epidemiological projections. Beyond looking at networks of individuals, SNA can explore relationships with concepts through analysis of qualitative data and concept mapping. It can also look at organizational risks and processes (think about comparing an organizational chart with who people actually go to within an organization for information).

How do you do SNA?

Conducting SNA mostly follows the same procedure as other analysis.

  1. Determine your purpose and questions. What decisions do you need to make based on the data you get? Who is your audience and what do they need to know? Answering these questions can help you decided what you are trying to measure and how.
  2. Collect your data. SNA can incorporate lots of different data forms, including customized surveys and interviews asking actors in your network about the links they have, external data such as census information or other public records to further inform attributes or triangulate your custom data; and mapping of key locations or concepts. One thing to consider while conducting an SNA – data cleaning is usually a heavier lift than for other types of analysis.
  3. Crunch those numbers. SNA uses matrices to calculate various measures – from types of centrality to network density and beyond. Lucky for us, there are plenty of tools that take on both the analysis and visualization portions of SNA. However, another consideration as you analyze your data is that network data is often not generalizable in the same way as some statistical analysis. If you miss a key node in the network, you may miss an entire portion that is only linked through that node.
  4. Visualize the network. Network visualizations are one of the most distinctive features of SNA and can be incredibly useful as tools to engage partners about your findings. There is a wealth of analysis and visualization tools that can help you do this. We created a worksheet that outlines several, but a few of the most popular are UCINet, Gephi, and NodeXL.
  5. Interpret your results. You now have a beautiful graph that shows what nodes are important in your network. So what? How does it relate to your program? Your interpretation should answer the questions around the purpose of your analysis, but beyond interpretation can serve to improve your programming. Often, SNA results can help make projections for program sustainability based on who key players are and who can continue championing work, or projecting where trends seem to be going and anticipating activities around those areas.

Conclusions and resources

We barely scratched the surface of what SNA can do and there are so many more applications! Some great resources to learn more are the SNA TIG of the American Evaluation Association, Stephen Borgatti’s course website on SNA, and a site of his dedicated completely to designing surveys for SNA.

Data Security and Privacy – MERL Tech presentation spurs action

By Stacey Berlow of Project Balance. The original was posted on Project Balance’s blog.

I had the opportunity to attend MERL Tech (September 7-8, 2017 Washington, DC). I was struck by the number of very thoughtful and content driven sessions. Coming from an IT/technology perspective, it was so refreshing to hear about the intersection of technology and humanitarian programs and how technology can provide the tools and data to positively impact decision making.
.
One of the sessions, “Big data, big problems, big solutions: Incorporating responsible data principles in institutional data management” was particularly poignant. The session was presented by Paul Perrin from University of Notre Dame, Alvaro Cobo & Jeff Lundberg from Catholic Relief Services and Gillian Kerr from LogicalOutcomes. The overall theme of the presentation was that in the field of evaluation and ICT4D, we must be thoughtful, diligent and take responsibility for protecting people’s personal and patient data; the potential risk for having a data breach is very high.

PaulPerrinDataRisk

Paul started the session by highlighting the fact that data breaches which expose our personal data, credit card information and health information have become a common occurrence. He brought the conversation back to monitoring and evaluation and research and the gray area between the two, leading to confusion about data privacy. Paul’s argument is that evaluation data is used for research later in a project without proper approval of those receiving services. The risk for misuse and incorrect data handling increases significantly.

Alvaro and Jeff talked about a CRS data warehousing project and how they have made data security and data privacy a key focus. The team looked at the data lifecycle – repository design, data collection, storage, utilization, sharing and retention/destruction – and they are applying best data security practices throughout. And finally, Gillian described the very concerning situation that at NGOs, M&E practitioners may not be aware of data security and privacy best practices or don’t have the funds to correctly meet minimum security standards and leave this critical data aspect behind as “too complicated to deal with.”

The presentation team advocates for the following:

  • Deeper commitment to informed consent
  • Reasoned use of identifiers
  • Need to know vs. nice to know
  • Data security and privacy protocols
  • Data use agreements and protocols for outside parties
  • Revisit NGO primary and secondary data IRB requirements

This message resonated with me in a surprising way. Project Balance specializes in developing data collection applications, data warehousing and data visualization. When we embark on a project we are careful to make sure that sensitive data is handled securely and that client/patient data is de-identified appropriately. We make sure that client data can only be viewed by those that should have access; that tables or fields within tables that hold identifying information are encrypted. Encryption is used for internet data transmission and depending on the application the entire database may be encrypted. And in some cases the data capture form that holds a client’s personal and identifying information may require that the user of the system re-log in.

After hearing the presentation I realized Project Balance could do better. As part of our regular software requirements management process, we will now create a separate and specialized data security and privacy plan document, which will enhance our current process. By making this a defined requirements gathering step, the importance of data security and privacy will be highlighted and will help our customers address any gaps that are identified before the system is built.

Many thanks to the session presenters for bringing this topic to the fore and for inspiring me to improve our engagement process!

Big data, big problems, big solutions

by Alvaro Cobo-Santillan, Catholic Relief Services (CRS); Jeff Lundberg, CRS; Paul Perrin, University of Notre Dame; and Gillian Kerr, LogicalOutcomes Canada. 

In the year 2017, with all of us holding a mini-computer at all hours of the day and night, it’s probably not too hard to imagine that “A teenager in Africa today has access to more information than the President of United States had 15 years ago”. So it also stands to reason that the ability to appropriately and ethically grapple with the use of that immense amount information has grown proportionately.

At the September MERL Tech event in Washington D.C. a panel that included folks from University of Notre Dame, Catholic Relief Services, and LogicalOutcomes spoke at length about three angles of this opportunity involving big data.

The Murky Waters of Development Data

What do we mean when we say that the world of development—particularly evaluation—data is murky? A major factor in this sentiment is the ambiguous polarity between research and evaluation data.

  • “Research seeks to prove; evaluation seeks to improve.” – CDC
  • “Research studies involving human subjects require IRB review. Evaluative studies and activities do not.”
Source: Patricia Rogers (2014), Ways of Framing the difference between research and evaluation, Better Evaluation Network.

This has led to debates as to the actual relationship between research and evaluation. Some see them as related, but separate activities, others see evaluation as a subset of research, and still others might posit that research is a specific case of evaluation.

But regardless, though motivations of the two may differ, research and evaluation look the same due to their stakeholders, participants, and methods.

If that statement is true, then we must hold both to similar protections!

What are some ways to make the waters less murky?

  • Deeper commitment to informed consent
  • Reasoned use of identifiers
  • Need to know vs. nice to know
  • Data security and privacy protocols
  • Data use agreements and protocols for outside parties
  • Revisit NGO primary and secondary data IRB requirements

Alright then, what can we practically do within our individual agencies to move the needle on data protection?

  • In short, governance. Responsible data is absolutely a crosscutting responsibility, but can be primarily championed through close partnerships between the M&E and IT Departments
  • Think about ways to increase usage of digital M&E – this can ease the implementation of R&D
  • Can existing agency processes and resources be leveraged?
  • Plan and expect to implement gradual behavior change and capacity building as a pre-requisite for a sustainable implementation of responsible data protections
  • Think in an iterative approach. Gradually introduce guidelines, tools and training materials
  • Plan for business and technical support structures to support protections

Is anyone doing any of the practical things you’ve mentioned?

Yes! Gillian Kerr from LogicalOutcomes spoke about highlights from an M&E system her company is launching to provide examples of the type of privacy and security protections they are doing in practice.

As a basis for the mindset behind their work, she notably presented a pretty fascinating and simple comparison of high risk vs. low risk personal information – year of birth, gender, and 3 digit zip code is unique for .04% of US residents, but if we instead include a 5 digit zip code over 50% of US residents could be uniquely identified. Yikes.

In that vein, they are not collecting names or identification and only year of birth (not month or day) and seek for minimal sensitive data defining data elements by level of risk to the client (i.e. city of residence – low, glucose level – medium, and HIV status – high).

In addition, asking for permission not only in the original agency permission form, but also in each survey. Their technical system maintains two instances – one containing individual level personal information with tight permission even for administrators and another with aggregated data with small cell sizes. Other security measures such as multi-factor authentication, encryption, and critical governance; such as regular audits are also in place.

It goes without saying that we collectively have ethical responsibilities to protect personal information about vulnerable people – here are final takeaways:

  • If you can’t protect sensitive information, don’t collect it.
  • If you can’t keep up with current security practices, outsource your M&E systems to someone who can.
  • Your technology roadmap should aspire to give control of personal information to the people who provide it (a substantial undertaking).
  • In the meantime, be more transparent about how data is being stored and shared
  • Continue the conversation by visiting https://responsibledata.io/blog
Register for MERL Tech London, March 19-20th 2018! Session ideas due November 10th.

Data quality in the age of lean data

by Daniel Ramirez-Raftree, MERL Tech support team.

Evolving data collection methods call for evolving quality assurance methods. In their session titled Data Quality in the Age of Lean Data, Sam Schueth of Intermedia, Woubedle Alemayehu of Oxford Policy Management, Julie Peachey of the Progress out of Poverty Index, and Christina Villella of MEASURE Evaluation discussed problems, solutions, and ethics related to digital data collection methods. [Bios and background materials here]

Sam opened the conversation by comparing the quality assurance and control challenges in paper assisted personal interviewing (PAPI) to those in digital assisted personal interviewing (DAPI). Across both methods, the fundamental problem is that the data that is delivered is a black box. It comes in, it’s turned into numbers and it’s disseminated, but in this process alone there is no easily apparent information about what actually happened on the ground.

During the age of PAPI, this was dealt with by sending independent quality control teams to the field to review the paper questionnaire that was administered and perform spot checks by visiting random homes to validate data accuracy. Under DAPI, the quality control process becomes remote. Survey administrators can now schedule survey sessions to be recorded automatically and without the interviewer’s knowledge, thus effectively gathering a random sample of interviews that can give them a sense of how well the sessions were conducted. Additionally, it is now possible to use GPS to track the interviewers’ movements and verify the range of households visited. The key point here is that with some creativity, new technological capacities can be used to ensure higher data quality.

Woubedle presented next and elaborated on the theme of quality control for DAPI. She brought up the point that data quality checks can be automated, but that this requires pre-survey-implementation decisions about what indicators to monitor and how to manage the data. The amount of work that is put into programming this upfront design has a direct relationship on the ultimate data quality.

One useful tool is a progress indicator. Here, one collects information on trends such as the number of surveys attempted compared to those completed. Processing this data could lead to further questions about whether there is a pattern in the populations that did or did not complete the survey, thus alerting researchers to potential bias. Additionally, one can calculate the average time taken to complete a survey and use it to identify outliers that took too little or too long to finish. Another good practice is to embed consistency checks in the survey itself; for example, making certain questions required or including two questions that, if answered in a particular way, would be logically contradictory, thus signaling a problem in either the question design or the survey responses. One more practice could be to apply constraints to the survey, depending on the households one is working with.

After this discussion, Julie spoke about research that was done to assess the quality of different methods for measuring the Progress out of Poverty Index (PPI). She began by explaining that the PPI is a household level poverty measurement tool unique to each country. To create it, the answers to 10 questions about a household’s characteristics and asset ownership are scored to compute the likelihood that the household is living below the poverty line. It is a simple, yet effective method to evaluate household level poverty. The research project Julie described set out to determine if the process of collecting data to create the PPI could be made less expensive by using SMS, IVR or phone calls.

Grameen Foundation conducted the study and tested four survey methods for gathering data: 1) in-person and at home, 2) in-person and away from home, 3) in-person and over the phone, and 4) automated and over the phone. Further, it randomized key aspects of the study, including the interview method and the enumerator.

Ultimately, Grameen Foundation determined that the interview method does affect completion rates, responses to questions, and the resulting estimated poverty rates. However, the differences in estimated poverty rates was likely not due to the method itself, but rather to completion rates (which were affected by the method). Thus, as long as completion rates don’t differ significantly, neither will the results. Given that the in-person at home and in-person away from home surveys had similar completion rates (84% and 91% respectively), either could be feasibly used with little deviation in output. On the other hand, in-person over the phone surveys had a 60% completion rate and automated over the phone surveys had a 12% completion rate, making both methods fairly problematic. And with this understanding, developers of the PPI have an evidence-based sense of the quality of their data.

This case study illustrates the the possibility of testing data quality before any changes are made to collection methods, which is a powerful strategy for minimizing the use of low quality data.

Christina closed the session with a presentation on ethics in data collection. She spoke about digital health data ethics in particular, which is the intersection of public health ethics, clinical ethics, and information systems security. She grounded her discussion in MEASURE Evaluation’s experience thinking through ethical problems, which include: the vulnerability of devices where data is collected and stored, the privacy and confidentiality of the data on these devices, the effect of interoperability on privacy, data loss if the device is damaged, and the possibility of wastefully collecting unnecessary data.

To explore these issues, MEASURE conducted a landscape assessment in Kenya and Tanzania and analyzed peer reviewed research to identify key themes for ethics. Five themes emerged: 1) legal frameworks and the need for laws, 2) institutional structures to oversee implementation and enforcement, 3) information systems security knowledge (especially for countries that may not have the expertise), 4) knowledge of the context and users (are clients comfortable with their data being used?), and 5) incorporating tools and standard operating procedures.

Based in this framework, MEASURE has made progress towards rolling out tools that can help institute a stronger ethics infrastructure. They’ve been developing guidelines that countries can use to develop policies, building health informatic capacity through a university course, and working with countries to strengthen their health information systems governance structures.

Finally, Christina explained her take on how ethics are related to data quality. In her view, it comes down to trust. If a device is lost, this may lead to incomplete data. If the clients are mistrustful, this could lead to inaccurate data. If a health worker is unable to check or clean data, this could create a lack of confidence. Each of these risks can lead to the erosion of data integrity.

Register for MERL Tech London, March 19-20th 2018! Session ideas due November 10th.

MERL Tech and the World of ICT Social Entrepreneurs (WISE)

by Dale Hill, an economist/evaluator with over 35 years experience in development and humanitarian work. Dale led the session on “The growing world of ICT Social Entrepreneurs (WISE): Is social Impact significant?” at MERL Tech DC 2018.

Roger Nathanial Ashby of OpenWise and Christopher Robert of Dobility share experiences at MERL Tech.
Roger Nathanial Ashby of OpenWise and Christopher Robert of Dobility share experiences at MERL Tech.

What happens when evaluators trying to build bridges with new private sector actors meet real social entrepreneurs? A new appreciation for the dynamic “World of ICT Social Entrepreneurs (WISE)” and the challenges they face in marketing, pricing, and financing (not to mention measurement of social impact.)

During this MERL Tech session on WISE, Dale Hill, evaluation consultant, presented grant funded research on measurement of social impact of social entrepreneurship ventures (SEVs) from three perspectives. She then invited five ICT company CEOs to comment.

The three perspectives are:

  • the public: How to hold companies accountable, particularly if they have chosen to be legal or certified “benefit corporations”?
  • the social entrepreneurs, who are plenty occupied trying to reach financial sustainability or profit goals, while also serving the public good; and
  • evaluators, who see the important influence of these new actors, but know their professional tools need adaptation to capture their impact.

Dale’s introduction covered overlapping definitions of various categories of SEVs, including legally defined “benefit corporations”, and “B Corps”, which are intertwined with the options of certification available to social entrepreneurs. The “new middle” of SEVs are on a spectrum between for-profit companies on one end and not-for profit organizations on the other. Various types of funders, including social impact investors, new certification agencies, and monitoring and evaluation (M&E) professionals, are now interested in measuring the growing social impact of these enterprises. A show of hands revealed that representatives of most of these types of actors were present at the session.

The five social entrepreneur panelists all had ICT businesses with global reach, but they varied in legal and certification status and the number of years operating (1 to 11). All aimed to deploy new technologies to non-profit organizations or social sector agencies on high value, low price terms. Some had worked in non-profits in the past and hoped that venture capital rather than grant funding would prove easier to obtain. Others had worked for Government and observed the need for customized solutions, which required market incentives to fully develop.

The evaluator and CEO panelists’ identification of challenges converged in some cases:

  • maintaining affordability and quality when using market pricing
  • obtaining venture capital or other financing
  • worry over “mission drift” – if financial sustainability imperatives or shareholder profit maximization preferences prevail over founders’ social impact goals; and
  • the still present digital divide, when serving global customers (insufficient bandwidth, affordability issues, limited small business capital in some client countries.

New issues raised by the CEOs (and some social entrepreneurs in the audience) included:

  • the need to provide incentives to customers to use quality assurance or security features of software, to avoid falling short of achieving the SEV’s “public good” goals;
  • the possibility of hostile takeover, given high value of technological innovations;
  • the fact that mention of a “social impact goal” was a red flag to some funders who then went elsewhere to seek profit maximization.

There was also a rich discussion on the benefits and costs of obtaining certification: it was a useful “branding and market signal” to some consumers, but a negative one to some funders; also, it posed an added burden on managers to document and report social impact, sometimes according to guidelines not in line with their preferences.

Surprises?

a) Despite the “hype”, social impact investment funding proved elusive to the panelists. Options for them included: sliding scale pricing; establishment of a complementary for-profit arm; or debt financing;

b) Many firms were not yet implementing planned monitoring and evaluation (M&E) programs, despite M&E being one of their service offerings; and

c) The legislation on reporting social impact of benefit corporations among the 31 states varies considerably, and the degree of enforcement is not clear.

A conclusion for evaluators: Social entrepreneurs’ use of market solutions indeed provides an evolving, dynamic environment which poses more complex challenges for measuring social impact, and requires new criteria and tools, ideally timed with an understanding of market ups and downs, and developed with full participation of the business managers.