MERL Tech News

You can’t have Aid…without AI: How artificial intelligence may reshape M&E

by Jacob Korenblum, CEO of Souktel Digital Solutions

Photo: wikipedia.org/

Potential—And Risk

The rapid growth of Artificial Intelligence—computers behaving like humans, and performing tasks which people usually carry out—promises to transform everything from car travel to personal finance. But how will it affect the equally vital field of M&E? As evaluators, most of us hate paper-based data collection—and we know that automation can help us process data more efficiently. At the same time, we’re afraid to remove the human element from monitoring and evaluation: What if the machines screw up?

Over the past year, Souktel has worked on three areas of AI-related M&E, to determine where new technology can best support project appraisals. Here are our key takeaways on what works, what doesn’t, and what might be possible down the road.

Natural Language Processing

For anyone who’s sifted through thousands of Excel entries, natural language processing sounds like a silver bullet: This application of AI interprets text responses rapidly, often matching them against existing data sets to find trends. No need for humans to review each entry by hand! But currently, it has two main limitations: First, natural language processing works best for sentences with simple syntax. Throw in more complex phrases, or longer text strings, and the power of AI to grasp open-ended responses goes downhill. Second, natural language processing only works for a limited number of (mostly European) languages—at least for now. English and Spanish AI applications? Yes. Chichewa or Pashto M&E bots? Not yet. Given these constraints, we’ve found that AI apps are strongest at interpreting basic misspelled answer text during mobile data collection campaigns (in languages like English or French). They’re less good at categorizing open-ended responses by qualitative category (positive, negative, neutral). Yet despite these limitations, AI can still help evaluators save time.

Object Differentiation

AI does a decent job of telling objects apart; we’ve leveraged this to build mobile applications which track supply delivery more quickly & cheaply. If a field staff member submits a photo of syringes and a photo of bandages from their mobile, we don’t need a human to check “syringes” and “bandages” off a list of delivered items. The AI-based app will do that automatically—saving huge amounts of time and expense, especially during crisis events. Still, there are limitations here too: While AI apps can distinguish between a needle and a BandAid, they can’t yet tell us whether the needle is broken, or whether the BandAid is the exact same one we shipped. These constraints need to be considered carefully when using AI for inventory monitoring.

Comparative Facial Recognition

This may be the most exciting—and controversial—application of AI. The potential is huge: “Qualitative evaluation” takes on a whole new meaning when facial expressions can be captured by cameras on mobile devices. On a more basic level, we’ve been focusing on solutions for better attendance tracking: AI is fairly good at determining whether the people in a photo at Time A are the same people in a photo at Time B. Snap a group pic at the end of each community meeting or training, and you can track longitudinal participation automatically. Take a photo of a larger crowd, and you can rapidly estimate the number of attendees at an event.

However, AI applications in this field have been notoriously bad at recognizing diversity—possibly because they draw on databases of existing images, and most of those images contain…white men. New MIT research has suggested that “since a majority of the photos used to train [AI applications] contain few minorities, [they] often have trouble picking out those minority faces”. For the communities where many of us work (and come from), that’s a major problem.

Do’s and Don’ts

So, how should M&E experts navigate this imperfect world? Our work has yielded a few “quick wins”—areas where Artificial Intelligence can definitely make our lives easier: Tagging and sorting quantitative data (or basic open-ended text), simple differentiation between images and objects, and broad-based identification of people and groups. These applications, by themselves, can be game-changers for our work as evaluators—despite their drawbacks. And as AI keeps evolving, its relevance to M&E will likely grow as well. We may never reach the era of robot focus group facilitators—but if robo-assistants help us process our focus group data more quickly, we won’t be complaining.

MERL Tech London session ideas due this Friday, Nov 10th!

MERL Tech London is coming up on March 19-20, 2018. Session ideas are due by Friday, November 10th, so be sure to get yours in this week!!

Submission Deadline: Friday, November 10, 2017.

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early December and, if selected, you will be asked to submit an updated and final session title, summary and outline by January 19th, 2018.

Topics we’re looking for:

  • Case studies: Sharing end-to-end experiences/learning from a MERL Tech process
  • MERL Tech 101: How-to use a MERL Tech tool or approach
  • Methods & Frameworks: Sharing/developing/discussing methods and frameworks for MERL Tech
  • Data: Big, large, small, quant, qual, real-time, online-offline, approaches, quality, etc.
  • Innovations: Brand new, untested technologies or approaches and their application to MERL(Tech)
  • Debates: Lively discussions, big picture conundrums, thorny questions, contentious topics related to MERL Tech
  • Management: People, organizations, partners, capacity strengthening, adaptive management, change processes related to MERL Tech
  • Evaluating MERL Tech: comparisons or learnings about MERL Tech tools/approaches and technology in development processes
  • Failures: What hasn’t worked and why, and what can be learned from this?
  • Demo Tables: to share MERL Tech approaches, tools, and technologies
  • Other topics we may have missed!

To get you thinking — take a look at past agendas from MERL Tech LondonMERL Tech DC and MERL Tech News.

Submit your session idea now!

We’re actively seeking a diverse (in every way) set of MERL Tech practitioners to facilitate every session. We encourage organizations to open this opportunity to colleagues and partners working outside of headquarters and to support their participation. (And please, no all-male panels!)

MERL Tech is dedicated to creating a safe, inclusive, welcoming and harassment-free experience for everyone. Please review our Code of Conduct. Session submissions are reviewed by our steering committee.

Submit your session ideas by November 10th!

If you have any questions about your submission idea, please contact Linda Raftree.

(Registration is also open!)

Visualizing what connects us: Social Network Analysis (SNA) in M&E

by Anne Laesecke (IREX) and Danielle de García (Social Impact). This post also appears on the Social Impact blog and  the IREX blog.

SNA, or Social Network Analysis, continues to gain momentum in the M&E space. This year at MERL Tech, we held an SNA 101 session, giving a quick-and-dirty overview of what it is, how it can contribute to M&E, and useful tips and tools for conducting an SNA. If you missed it, here’s what you need to know:

What is SNA?

SNA is a way to analyze social systems through relationships. Analyzing and visualizing networks can reveal critical insights for understanding relationships between organizations, supply chains; social movements; and/or between individuals. It’s a very versatile tool which can be used throughout the program cycle to measure things like trust and social capital, information flows, resources, collaboration, and disease spread, among other things.

SNA uses a different vocabulary than other types of analyses. For example, the relationships we measure are called ties or links, and the entities that make up a network are called nodes or actors. These can be organizations, people, or even whole networks themselves. We can study nodes more closely by looking at their attributes – things that characterize them (like demographic information), and we can learn more about how nodes interact and cluster by studying communities or modalities within networks. Various measures of the roles nodes play in a network, as well as measures that characterize the networks themselves, can reveal a lot about the systems and hidden relationships at play. For example, we can determine who has the most ties with other actors; who is relatively cut off from the network, or who is connected to the most well-connected actors.

Why would you use SNA for M&E?

The term “social network analysis” often triggers associations with social media, but SNA uses data from a variety of platforms (including but not limited to social media!). For instance, SNA can identify key influencers in systems – important for programs that rely on thinking politically. SNA can also be a useful tool in processing big data with applications for cybersecurity as well as creating biological and epidemiological projections. Beyond looking at networks of individuals, SNA can explore relationships with concepts through analysis of qualitative data and concept mapping. It can also look at organizational risks and processes (think about comparing an organizational chart with who people actually go to within an organization for information).

How do you do SNA?

Conducting SNA mostly follows the same procedure as other analysis.

  1. Determine your purpose and questions. What decisions do you need to make based on the data you get? Who is your audience and what do they need to know? Answering these questions can help you decided what you are trying to measure and how.
  2. Collect your data. SNA can incorporate lots of different data forms, including customized surveys and interviews asking actors in your network about the links they have, external data such as census information or other public records to further inform attributes or triangulate your custom data; and mapping of key locations or concepts. One thing to consider while conducting an SNA – data cleaning is usually a heavier lift than for other types of analysis.
  3. Crunch those numbers. SNA uses matrices to calculate various measures – from types of centrality to network density and beyond. Lucky for us, there are plenty of tools that take on both the analysis and visualization portions of SNA. However, another consideration as you analyze your data is that network data is often not generalizable in the same way as some statistical analysis. If you miss a key node in the network, you may miss an entire portion that is only linked through that node.
  4. Visualize the network. Network visualizations are one of the most distinctive features of SNA and can be incredibly useful as tools to engage partners about your findings. There is a wealth of analysis and visualization tools that can help you do this. We created a worksheet that outlines several, but a few of the most popular are UCINet, Gephi, and NodeXL.
  5. Interpret your results. You now have a beautiful graph that shows what nodes are important in your network. So what? How does it relate to your program? Your interpretation should answer the questions around the purpose of your analysis, but beyond interpretation can serve to improve your programming. Often, SNA results can help make projections for program sustainability based on who key players are and who can continue championing work, or projecting where trends seem to be going and anticipating activities around those areas.

Conclusions and resources

We barely scratched the surface of what SNA can do and there are so many more applications! Some great resources to learn more are the SNA TIG of the American Evaluation Association, Stephen Borgatti’s course website on SNA, and a site of his dedicated completely to designing surveys for SNA.

MERL Tech Round Up | November 1, 2017

It’s time for our second MERL Tech Round Up, a monthly compilation of MERL Tech News!

On the MERL Tech Blog:

We’ve been posting session summaries from MERL Tech DC. Here are some posts you may have missed in October:

Stuff we’re reading/watching/bookmarking:

There’s quite a bit to learn both in our “MERL / Tech” sector and in related sectors whose experiences are relatable to MERL Tech. Some thought-provoking pieces here:

Events:

Jobs

Head over to ICT4DJobs for a ton of tech related jobs. Here are some interesting ones for folks in the MERL Tech space:

If you’re not already signed up to the Pelican Initiative: Platform for Evidence-based Learning & Communication for Social Change, we recommend doing that. You will find all kinds of MERL and MERLTech related jobs and MERL-related advice. (Note: the Platform is an extremely active forum, so you may want to adjust your settings to receive weekly compilations).

Tag us on Twitter using #MERLTech if you have resources, events, or other news you’d like us to include here!

Don’t forget to submit your session ideas for MERL Tech London by November 10th!

Data Security and Privacy – MERL Tech presentation spurs action

By Stacey Berlow of Project Balance. The original was posted on Project Balance’s blog.

I had the opportunity to attend MERL Tech (September 7-8, 2017 Washington, DC). I was struck by the number of very thoughtful and content driven sessions. Coming from an IT/technology perspective, it was so refreshing to hear about the intersection of technology and humanitarian programs and how technology can provide the tools and data to positively impact decision making.
.
One of the sessions, “Big data, big problems, big solutions: Incorporating responsible data principles in institutional data management” was particularly poignant. The session was presented by Paul Perrin from University of Notre Dame, Alvaro Cobo & Jeff Lundberg from Catholic Relief Services and Gillian Kerr from LogicalOutcomes. The overall theme of the presentation was that in the field of evaluation and ICT4D, we must be thoughtful, diligent and take responsibility for protecting people’s personal and patient data; the potential risk for having a data breach is very high.

PaulPerrinDataRisk

Paul started the session by highlighting the fact that data breaches which expose our personal data, credit card information and health information have become a common occurrence. He brought the conversation back to monitoring and evaluation and research and the gray area between the two, leading to confusion about data privacy. Paul’s argument is that evaluation data is used for research later in a project without proper approval of those receiving services. The risk for misuse and incorrect data handling increases significantly.

Alvaro and Jeff talked about a CRS data warehousing project and how they have made data security and data privacy a key focus. The team looked at the data lifecycle – repository design, data collection, storage, utilization, sharing and retention/destruction – and they are applying best data security practices throughout. And finally, Gillian described the very concerning situation that at NGOs, M&E practitioners may not be aware of data security and privacy best practices or don’t have the funds to correctly meet minimum security standards and leave this critical data aspect behind as “too complicated to deal with.”

The presentation team advocates for the following:

  • Deeper commitment to informed consent
  • Reasoned use of identifiers
  • Need to know vs. nice to know
  • Data security and privacy protocols
  • Data use agreements and protocols for outside parties
  • Revisit NGO primary and secondary data IRB requirements

This message resonated with me in a surprising way. Project Balance specializes in developing data collection applications, data warehousing and data visualization. When we embark on a project we are careful to make sure that sensitive data is handled securely and that client/patient data is de-identified appropriately. We make sure that client data can only be viewed by those that should have access; that tables or fields within tables that hold identifying information are encrypted. Encryption is used for internet data transmission and depending on the application the entire database may be encrypted. And in some cases the data capture form that holds a client’s personal and identifying information may require that the user of the system re-log in.

After hearing the presentation I realized Project Balance could do better. As part of our regular software requirements management process, we will now create a separate and specialized data security and privacy plan document, which will enhance our current process. By making this a defined requirements gathering step, the importance of data security and privacy will be highlighted and will help our customers address any gaps that are identified before the system is built.

Many thanks to the session presenters for bringing this topic to the fore and for inspiring me to improve our engagement process!

Big data, big problems, big solutions

by Alvaro Cobo-Santillan, Catholic Relief Services (CRS); Jeff Lundberg, CRS; Paul Perrin, University of Notre Dame; and Gillian Kerr, LogicalOutcomes Canada. 

In the year 2017, with all of us holding a mini-computer at all hours of the day and night, it’s probably not too hard to imagine that “A teenager in Africa today has access to more information than the President of United States had 15 years ago”. So it also stands to reason that the ability to appropriately and ethically grapple with the use of that immense amount information has grown proportionately.

At the September MERL Tech event in Washington D.C. a panel that included folks from University of Notre Dame, Catholic Relief Services, and LogicalOutcomes spoke at length about three angles of this opportunity involving big data.

The Murky Waters of Development Data

What do we mean when we say that the world of development—particularly evaluation—data is murky? A major factor in this sentiment is the ambiguous polarity between research and evaluation data.

  • “Research seeks to prove; evaluation seeks to improve.” – CDC
  • “Research studies involving human subjects require IRB review. Evaluative studies and activities do not.”
Source: Patricia Rogers (2014), Ways of Framing the difference between research and evaluation, Better Evaluation Network.

This has led to debates as to the actual relationship between research and evaluation. Some see them as related, but separate activities, others see evaluation as a subset of research, and still others might posit that research is a specific case of evaluation.

But regardless, though motivations of the two may differ, research and evaluation look the same due to their stakeholders, participants, and methods.

If that statement is true, then we must hold both to similar protections!

What are some ways to make the waters less murky?

  • Deeper commitment to informed consent
  • Reasoned use of identifiers
  • Need to know vs. nice to know
  • Data security and privacy protocols
  • Data use agreements and protocols for outside parties
  • Revisit NGO primary and secondary data IRB requirements

Alright then, what can we practically do within our individual agencies to move the needle on data protection?

  • In short, governance. Responsible data is absolutely a crosscutting responsibility, but can be primarily championed through close partnerships between the M&E and IT Departments
  • Think about ways to increase usage of digital M&E – this can ease the implementation of R&D
  • Can existing agency processes and resources be leveraged?
  • Plan and expect to implement gradual behavior change and capacity building as a pre-requisite for a sustainable implementation of responsible data protections
  • Think in an iterative approach. Gradually introduce guidelines, tools and training materials
  • Plan for business and technical support structures to support protections

Is anyone doing any of the practical things you’ve mentioned?

Yes! Gillian Kerr from LogicalOutcomes spoke about highlights from an M&E system her company is launching to provide examples of the type of privacy and security protections they are doing in practice.

As a basis for the mindset behind their work, she notably presented a pretty fascinating and simple comparison of high risk vs. low risk personal information – year of birth, gender, and 3 digit zip code is unique for .04% of US residents, but if we instead include a 5 digit zip code over 50% of US residents could be uniquely identified. Yikes.

In that vein, they are not collecting names or identification and only year of birth (not month or day) and seek for minimal sensitive data defining data elements by level of risk to the client (i.e. city of residence – low, glucose level – medium, and HIV status – high).

In addition, asking for permission not only in the original agency permission form, but also in each survey. Their technical system maintains two instances – one containing individual level personal information with tight permission even for administrators and another with aggregated data with small cell sizes. Other security measures such as multi-factor authentication, encryption, and critical governance; such as regular audits are also in place.

It goes without saying that we collectively have ethical responsibilities to protect personal information about vulnerable people – here are final takeaways:

  • If you can’t protect sensitive information, don’t collect it.
  • If you can’t keep up with current security practices, outsource your M&E systems to someone who can.
  • Your technology roadmap should aspire to give control of personal information to the people who provide it (a substantial undertaking).
  • In the meantime, be more transparent about how data is being stored and shared
  • Continue the conversation by visiting https://responsibledata.io/blog
Register for MERL Tech London, March 19-20th 2018! Session ideas due November 10th.

Data quality in the age of lean data

by Daniel Ramirez-Raftree, MERL Tech support team.

Evolving data collection methods call for evolving quality assurance methods. In their session titled Data Quality in the Age of Lean Data, Sam Schueth of Intermedia, Woubedle Alemayehu of Oxford Policy Management, Julie Peachey of the Progress out of Poverty Index, and Christina Villella of MEASURE Evaluation discussed problems, solutions, and ethics related to digital data collection methods. [Bios and background materials here]

Sam opened the conversation by comparing the quality assurance and control challenges in paper assisted personal interviewing (PAPI) to those in digital assisted personal interviewing (DAPI). Across both methods, the fundamental problem is that the data that is delivered is a black box. It comes in, it’s turned into numbers and it’s disseminated, but in this process alone there is no easily apparent information about what actually happened on the ground.

During the age of PAPI, this was dealt with by sending independent quality control teams to the field to review the paper questionnaire that was administered and perform spot checks by visiting random homes to validate data accuracy. Under DAPI, the quality control process becomes remote. Survey administrators can now schedule survey sessions to be recorded automatically and without the interviewer’s knowledge, thus effectively gathering a random sample of interviews that can give them a sense of how well the sessions were conducted. Additionally, it is now possible to use GPS to track the interviewers’ movements and verify the range of households visited. The key point here is that with some creativity, new technological capacities can be used to ensure higher data quality.

Woubedle presented next and elaborated on the theme of quality control for DAPI. She brought up the point that data quality checks can be automated, but that this requires pre-survey-implementation decisions about what indicators to monitor and how to manage the data. The amount of work that is put into programming this upfront design has a direct relationship on the ultimate data quality.

One useful tool is a progress indicator. Here, one collects information on trends such as the number of surveys attempted compared to those completed. Processing this data could lead to further questions about whether there is a pattern in the populations that did or did not complete the survey, thus alerting researchers to potential bias. Additionally, one can calculate the average time taken to complete a survey and use it to identify outliers that took too little or too long to finish. Another good practice is to embed consistency checks in the survey itself; for example, making certain questions required or including two questions that, if answered in a particular way, would be logically contradictory, thus signaling a problem in either the question design or the survey responses. One more practice could be to apply constraints to the survey, depending on the households one is working with.

After this discussion, Julie spoke about research that was done to assess the quality of different methods for measuring the Progress out of Poverty Index (PPI). She began by explaining that the PPI is a household level poverty measurement tool unique to each country. To create it, the answers to 10 questions about a household’s characteristics and asset ownership are scored to compute the likelihood that the household is living below the poverty line. It is a simple, yet effective method to evaluate household level poverty. The research project Julie described set out to determine if the process of collecting data to create the PPI could be made less expensive by using SMS, IVR or phone calls.

Grameen Foundation conducted the study and tested four survey methods for gathering data: 1) in-person and at home, 2) in-person and away from home, 3) in-person and over the phone, and 4) automated and over the phone. Further, it randomized key aspects of the study, including the interview method and the enumerator.

Ultimately, Grameen Foundation determined that the interview method does affect completion rates, responses to questions, and the resulting estimated poverty rates. However, the differences in estimated poverty rates was likely not due to the method itself, but rather to completion rates (which were affected by the method). Thus, as long as completion rates don’t differ significantly, neither will the results. Given that the in-person at home and in-person away from home surveys had similar completion rates (84% and 91% respectively), either could be feasibly used with little deviation in output. On the other hand, in-person over the phone surveys had a 60% completion rate and automated over the phone surveys had a 12% completion rate, making both methods fairly problematic. And with this understanding, developers of the PPI have an evidence-based sense of the quality of their data.

This case study illustrates the the possibility of testing data quality before any changes are made to collection methods, which is a powerful strategy for minimizing the use of low quality data.

Christina closed the session with a presentation on ethics in data collection. She spoke about digital health data ethics in particular, which is the intersection of public health ethics, clinical ethics, and information systems security. She grounded her discussion in MEASURE Evaluation’s experience thinking through ethical problems, which include: the vulnerability of devices where data is collected and stored, the privacy and confidentiality of the data on these devices, the effect of interoperability on privacy, data loss if the device is damaged, and the possibility of wastefully collecting unnecessary data.

To explore these issues, MEASURE conducted a landscape assessment in Kenya and Tanzania and analyzed peer reviewed research to identify key themes for ethics. Five themes emerged: 1) legal frameworks and the need for laws, 2) institutional structures to oversee implementation and enforcement, 3) information systems security knowledge (especially for countries that may not have the expertise), 4) knowledge of the context and users (are clients comfortable with their data being used?), and 5) incorporating tools and standard operating procedures.

Based in this framework, MEASURE has made progress towards rolling out tools that can help institute a stronger ethics infrastructure. They’ve been developing guidelines that countries can use to develop policies, building health informatic capacity through a university course, and working with countries to strengthen their health information systems governance structures.

Finally, Christina explained her take on how ethics are related to data quality. In her view, it comes down to trust. If a device is lost, this may lead to incomplete data. If the clients are mistrustful, this could lead to inaccurate data. If a health worker is unable to check or clean data, this could create a lack of confidence. Each of these risks can lead to the erosion of data integrity.

Register for MERL Tech London, March 19-20th 2018! Session ideas due November 10th.

MERL Tech and the World of ICT Social Entrepreneurs (WISE)

by Dale Hill, an economist/evaluator with over 35 years experience in development and humanitarian work. Dale led the session on “The growing world of ICT Social Entrepreneurs (WISE): Is social Impact significant?” at MERL Tech DC 2018.

Roger Nathanial Ashby of OpenWise and Christopher Robert of Dobility share experiences at MERL Tech.
Roger Nathanial Ashby of OpenWise and Christopher Robert of Dobility share experiences at MERL Tech.

What happens when evaluators trying to build bridges with new private sector actors meet real social entrepreneurs? A new appreciation for the dynamic “World of ICT Social Entrepreneurs (WISE)” and the challenges they face in marketing, pricing, and financing (not to mention measurement of social impact.)

During this MERL Tech session on WISE, Dale Hill, evaluation consultant, presented grant funded research on measurement of social impact of social entrepreneurship ventures (SEVs) from three perspectives. She then invited five ICT company CEOs to comment.

The three perspectives are:

  • the public: How to hold companies accountable, particularly if they have chosen to be legal or certified “benefit corporations”?
  • the social entrepreneurs, who are plenty occupied trying to reach financial sustainability or profit goals, while also serving the public good; and
  • evaluators, who see the important influence of these new actors, but know their professional tools need adaptation to capture their impact.

Dale’s introduction covered overlapping definitions of various categories of SEVs, including legally defined “benefit corporations”, and “B Corps”, which are intertwined with the options of certification available to social entrepreneurs. The “new middle” of SEVs are on a spectrum between for-profit companies on one end and not-for profit organizations on the other. Various types of funders, including social impact investors, new certification agencies, and monitoring and evaluation (M&E) professionals, are now interested in measuring the growing social impact of these enterprises. A show of hands revealed that representatives of most of these types of actors were present at the session.

The five social entrepreneur panelists all had ICT businesses with global reach, but they varied in legal and certification status and the number of years operating (1 to 11). All aimed to deploy new technologies to non-profit organizations or social sector agencies on high value, low price terms. Some had worked in non-profits in the past and hoped that venture capital rather than grant funding would prove easier to obtain. Others had worked for Government and observed the need for customized solutions, which required market incentives to fully develop.

The evaluator and CEO panelists’ identification of challenges converged in some cases:

  • maintaining affordability and quality when using market pricing
  • obtaining venture capital or other financing
  • worry over “mission drift” – if financial sustainability imperatives or shareholder profit maximization preferences prevail over founders’ social impact goals; and
  • the still present digital divide, when serving global customers (insufficient bandwidth, affordability issues, limited small business capital in some client countries.

New issues raised by the CEOs (and some social entrepreneurs in the audience) included:

  • the need to provide incentives to customers to use quality assurance or security features of software, to avoid falling short of achieving the SEV’s “public good” goals;
  • the possibility of hostile takeover, given high value of technological innovations;
  • the fact that mention of a “social impact goal” was a red flag to some funders who then went elsewhere to seek profit maximization.

There was also a rich discussion on the benefits and costs of obtaining certification: it was a useful “branding and market signal” to some consumers, but a negative one to some funders; also, it posed an added burden on managers to document and report social impact, sometimes according to guidelines not in line with their preferences.

Surprises?

a) Despite the “hype”, social impact investment funding proved elusive to the panelists. Options for them included: sliding scale pricing; establishment of a complementary for-profit arm; or debt financing;

b) Many firms were not yet implementing planned monitoring and evaluation (M&E) programs, despite M&E being one of their service offerings; and

c) The legislation on reporting social impact of benefit corporations among the 31 states varies considerably, and the degree of enforcement is not clear.

A conclusion for evaluators: Social entrepreneurs’ use of market solutions indeed provides an evolving, dynamic environment which poses more complex challenges for measuring social impact, and requires new criteria and tools, ideally timed with an understanding of market ups and downs, and developed with full participation of the business managers.

Tools, tips and templates for making Responsible Data a reality

by David Leege, CRS; Emily Tomkys, Oxfam GB; Nina Getachew, mSTAR/FHI 360; and Linda Raftree, Independent Consultant/MERL Tech; who led the session “Tools, tips and templates for making responsible data a reality.

The data lifecycle.
The data lifecycle.

For this year’s MERL Tech DC, we teamed up to do a session on Responsible Data. Based on feedback from last year, we knew that people wanted less discussion on why ethics, privacy and security are important, and more concrete tools, tips and templates. Though it’s difficult to offer specific do’s and don’ts, since each situation and context needs individualized analysis, we were able to share a lot of the resources that we know are out there.

To kick off the session, we quickly explained what we meant by Responsible Data. Then we handed out some cards from Oxfam’s Responsible Data game and asked people to discuss their thoughts in pairs. Some of the statements that came up for discussion included:

  • Being responsible means we can’t openly share data – we have to protect it
  • We shouldn’t tell people they can withdraw consent for us to use their data when in reality we have no way of doing what they ask
  • Biometrics are a good way of verifying who people are and reducing fraud

Following the card game we asked people to gather around 4 tables with a die and a print out of the data lifecycle where each phase corresponded to a number (Planning = 1, collecting = 2, storage = 3, and so on…). Each rolled the die and, based on their number, told a “data story” of an experience, concern or data failure related to that phase of the lifecycle. Then the group discussed the stories.

For our last activity, each of us took a specific pack of tools, templates and tips and rotated around the 4 tables to share experiences and discuss practical ways to move towards stronger responsible data practices.

Responsible data values and principles

David shared Catholic Relief Services’ process of developing a responsible data policy, which they started in 2017 by identifying core values and principles and how they relate to responsible data. This was based on national and international standards such as the Humanitarian Charter including the Humanitarian Protection Principles and the Core and Minimum Standards as outlined in Sphere Handbook Protection Principle 1; the Protection of Human Subjects, known as the “Common Rule” as laid out in the Department of Health and Human Services Policy for Protection of Human Research Subjects; and the Digital Principles, particularly Principle 8 which mandates that organizations address privacy and security.

As a Catholic organization, CRS follows the principles of Catholic social teaching, which directly relate to responsible data in the following ways:

  • Sacredness and dignity of the human person – we will respect and protect an individual’s personal data as an extension of their human dignity;
  • Rights and responsibilities – we will balance the right to be counted and heard with the right to privacy and security;
  • Social nature of humanity – we will weigh the benefits and risks of using digital tools, platforms and data;
  • Common good – we will open data for the common good only after minimizing the risks;
  • Subsidiarity – we will prioritize local ownership and control of data for planning and decision-making;
  • Solidarity – we will work to educate inform and engage our constituents in responsible data approaches;
  • Option for the poor – we will take a preferential option for protecting and securing the data of the poor; and
  • Stewardship – we will responsibly steward the data that is provided to us by our constituents.

David shared a draft version of CRS’ responsible data values and principles.

Responsible data policy, practices and evaluation of their roll-out

Oxfam released its Responsible Program Data Policy in 2015. Since then, they have carried out six pilots to explore how to implement the policy in a variety of countries and contexts. Emily shared information on these these pilots and the results of research carried out by the Engine Room called Responsible Data at Oxfam: Translating Oxfam’s Responsible Data Policy into practice, two years on. The report concluded that the staff that have engaged with Oxfam’s Responsible Data Policy find it both practically relevant and important. One of the recommendations of this research showed that Oxfam needed to increase uptake amongst staff and provide an introductory guide to the area of responsible data.  

In response, Oxfam created the Responsible Data Management pack, (available in English, Spanish, French and Arabic), which included the game that was played in today’s session along with other tools and templates. The card game introduces some of the key themes and tensions inherent in making responsible data decisions. The examples on the cards are derived from real experiences at Oxfam and elsewhere, and they aim to generate discussion and debate. Oxfam’s training pack also includes other tools, such as advice on taking photos, a data planning template, a poster of the data lifecycle and general information on how to use the training pack. Emily’s session also encouraged discussion with participants about governance and accountability issues like who in the organisation manages responsible data and how to make responsible data decisions when each context may require a different action.

Emily shared the following resources:

A packed house for the responsible data session.
A packed house for the responsible data session.

Responsible data case studies

Nina shared early results of four case studies mSTAR is conducting together with Sonjara for USAID. The case studies are testing a draft set of responsible data guidelines, determining whether they are adequate for ‘on the ground’ situations and if projects find them relevant, useful and usable. The guidelines were designed collaboratively, based on a thorough review and synthesis of responsible data practices and policies of USAID and other international development and humanitarian organizations. To conduct the case studies, Sonjara, Nina and other researchers visited four programs which are collecting large amounts of potentially sensitive data in Nigeria, Kenya and Uganda. The researchers interviewed a broad range of stakeholders and looked at how the programs use, store, and manage personally identifiable data (PII). Based on the research findings, adjustments are being made to the guidelines. It is anticipated that they will be published in October.

Nina also talked about CALP/ELAN’s data sharing tipsheets, which include a draft data-sharing agreement that organizations can adapt to their own contracting contracting documents. She circulated a handout which identifies the core elements of the Fair Information Practice Principles (FIPPs) that are important to consider when using PII data.  

Responsible data literature review and guidelines

Linda mentioned that a literature review of responsible data policy and practice has been done as part of the above mentioned mSTAR project (which she also worked on). The literature review will provide additional resources and analysis, including an overview of the core elements that should be included in organizational data guidelines, an overview of USAID policy and regulations, emerging legal frameworks such as the EU’s General Data Protection Regulation (GDPR), and good practice on how to develop guidelines in ways that enhance uptake and use. The hope is that both the Responsible Data Literature Review and the of Responsible Data Guidelines will be suitable for adopting and adapting by other organizations. The guidelines will offer a set of critical questions and orientation, but that ethical and responsible data practices will always be context specific and cannot be a “check-box” exercise given the complexity of all the elements that combine in each situation. 

Linda also shared some tools, guidelines and templates that have been developed in the past few years, such as Girl Effect’s Digital Safeguarding Guidelines, the Future of Privacy Forum’s Risk-Benefits-Harms framework, and the World Food Program’s guidance on Conducting Mobile Surveys Responsibly.

More tools, tips and templates

Check out this responsible data resource list, which includes additional tools, tips and templates. It was developed for MERL Tech London in February 2017 and we continue to add to it as new documents and resources come out. After a few years of advocating for ‘responsible data’ at MERL Tech to less-than-crowded sessions, we were really excited to have a packed room and high levels of interest this year!   

How to buy M&E software and not get bamboozled

by Josh Mandell, a Director at DevResults where he leads strategy and business development. Josh can be reached at josh@devresults.com.

While there is no way to guarantee that M&E software will solve all of your problems or make all of your colleagues happy, there absolutely are things you can do during the discovery, procurement, and contracts stages to mitigate against the risk of getting bamboozled.

#1 – Trust no one. Test everything.

Most development practitioners I speak with are balancing a heavy load of client work, internal programmatic and BD support, and other organization initiatives. I can appreciate that time is scarce and testing software you may not buy could feel like a giant waste of time.

However, when it comes to reducing uncertainty and building confidence in your decision, the single most productive use of your time is spent testing. When you don’t test, what evidence do you have to base your decision on? The vendor’s marketing and proposal materials. Don’t take the BD guy’s word for it and whatever you do, don’t trust screenshots, brochures, or proposals. Like a well-curated social media profile, marketing collateral gives you a sense for what’s possible, but probably isn’t the most accurate reflection of reality. If you really want to understand usability, performance, and culture fit, you simply need to see for yourself.

We have found that the organizations that take the time to identify and test what they’ll actually be doing in DevResults are much better off than those who buy based on what they see in documentation and presentations, or based on someone else’s recommendation.

And it makes our lives easier too! We may have to spend a little more time upfront in the discovery and procurement phases, but by properly setting expectations early on, we have to provide far less support over the long-term. This makes for smoother, lower-cost implementations and happier customers.

#2 – Document what success looks like in plain language.

We obviously need contracts for defining the scope of work, payment terms, SLA, and other legalese, but the reality is that the people leading procurement and contracts are often not the people leading the day to day data operations.

Contracts are also typically dense and hard to use as a point of reference for frequent, human communication. So, it’s incredibly important that the implementation leads themselves define what success looks like in their own words and that is what drives the implementation.

It took us years to figure this out, but we’ve taken the lesson to heart. What we do now with each of our engagements is create an Implementation Charter that documents, in the words of the implementation leads, things like a summary baseline, roles and responsibilities, and a list of desired outcomes, i.e. ‘what success looks like.’ We then use the charter as the primary point of reference for determining whether or not we’re doing a good job and we evaluate ourselves against the charter quarterly.

Similar to the point about testing above, we have found this practice to dramatically increase transparency, properly set expectations, and establish more effective channels for communication, all of which are crucial in enterprise software implementations.

#3 – Plan for the long-haul and create the right financial incentives. Spread out the payments.

Whether at the project or organizational levels, M&E software implementations are long-term efforts. Unlike custom, external-facing websites where the bulk of work is done up front and the rest is mostly maintenance, enterprise software is constantly evolving. Rapidly changing technology and industry trends, shifting user requirements, and quality user experience all require persistent attention and ongoing development.

Your contract and payment structure should reflect that reality.

The easiest way to achieve this alignment is to spread the payments out over time. I’m not going to get into the merits of a software as a service (SaaS) business model here (we’ll be putting another post out on that in the coming weeks), but suffice to say that you get better service when your technology partner needs to continuously earn your money month after month and year after year.

This not only shifts the focus from checking boxes in a contract to delivering actual utility for users over the long-term, but it also hedges against the prospect of paying for unused software (or even paying for vaporware, as in the case of the BMGF case against Saama).

We know from experience that shifting to a new way of doing things can be difficult. We used to be a custom-web development shop and we did pretty well in that old model. The transition to a SaaS offering was painful because we had to work harder to earn our money and expectations went up dramatically. Nonetheless, we know the pain has been worth it because our customers are holding us to a different standard and it’s forcing us to deliver the best product we’re capable of. As a result, we’ll not only have happier customers, but a stronger, more sustainable business doing what we love.

Stop the bamboozling.

If you have any tips or recommendations for buying software, please share those in the comments below, or feel free to reach out to me directly. We’re always looking to share what we know and learn from others. Good luck!

MERL Tech London is coming up on March 20-21, 2018 — Submit your session ideas or register to attend!