Tag Archives: learning

MERL Tech 101: Google forms

by Daniel Ramirez-Raftree, MERL Tech volunteer

In his MERL Tech DC session on Google Forms, Samhir Vesdev from IREX led a hands-on workshop on Google Forms and laid out some of the software’s capabilities and limitations. Much of the session focused on Google Forms’ central concepts and the practicality of building a form.

At its most fundamental level, a form is made up of several sections, and each section is designed to contain a question or prompt. The centerpiece of a section is the question cell, which is, as one would imagine, the cell dedicated to the question. Next to the question cell there is a drop down menu that allows one to select the format of the question, which ranges from multiple-choice to short answer.


At the bottom right hand corner of the section you will find three dots arranged vertically. When you click this toggle, a drop-down menu will appear. The options in this menu vary depending on the format of the question. One common option is to include a few lines of description, which is useful in case the question needs further elaboration or instruction. Another is the data validation option, which restricts the kinds of text that a respondent can input. This is useful in the case that, for example, the question is in a short answer format but the form administrators need the responses to be limited numerals for the sake of analysis.

The session also covered functions available in the “response” tab, which sits at the top of the page. Here one can find a toggle labeled “accepting responses” that can be turned off or on depending on the needs for the form.

Additionally, in the top right corner this tab, there are three dots arranged vertically, and this is the options menu for this tab. Here you will find options such as enabling email notifications for each new response, which can be used in case you want to be alerted when someone responds to the form. Also in this drop down, you can click “select response destination” to link the Google Form with Google Sheets, which simplifies later analysis. The green sheets icon next to the options drop-down will take you to the sheet that contains the collected data.

Other capabilities in Google Forms include the option for changing the color scheme, which you can access by clicking the palette icon at the top of the screen. Also, by clicking the settings button at the top of the screen you can limit the response amount to restrict people’s ability to skew the data by submitting multiple responses, or you can enable response editing after submission to allow respondents to go in and correct their response after submitting it.

Branching is another important tool in Google Forms. It can be used in the case that you want a particular response to a question (say, a multiple choice question) to lead the respondent to another related question only if they respond in a certain way.

For example, if in one section you ask “did you like the workshop?” with the answer options being “yes” and “no,” and if you want to know what they didn’t like about the workshop only if they answer “no,” you can design the sheet to take the respondent to a section with the question “what didn’t you like about the workshop?” only in the case that they answer “no,” and then you can design the sheet to bring the respondent back to the main workflow after they’ve answered this additional question.

To do this, create at least two new sections (by clicking “add section” in the small menu to the right of the sections), one for each path that a person’s response will lead them down. Then, in the options menu on the lower right hand side select “go to section based on answer” and using the menu that appears, set the path that you desire.

These are just some of the tools that Google Forms offers, but with just these it is possible to build an effective form to collect the data you need. Samhir ended with a word of caution that Google has been known to shut down popular apps, so you should be wary about building an organization strategy around Google Forms.

M&E Squared: Evaluating M&E Technologies

by Roger Nathanial Ashby, Co-Founder & Principal Consultant, OpenWise

The universe of MERL Tech solutions has grown exponentially. In 2008 monitoring and evaluating tech within global development could mostly be confined to mobile data collection tools like Open Data Kit (ODK), and Excel spreadsheets to analyze and visualize survey data. In the intervening decade a myriad of tools, companies and NGOs have been created to advance the efficiency and effectiveness of monitoring, evaluation, research and learning (MERL) through the use of technology. Whether it’s M&E platforms or suites, satellite imagery, remote sensors, or chatbots, new innovations are being deployed every day in the field.

However, how do we evaluate the impact when MERL Tech is the intervention itself? That was the question and task put to participants of the “M&E Squared” workshop at MERL Tech 2017.

Workshop participants were separated into three groups that were each given a case study to discuss and analyze. One group was given a case about improving the learning efficiency of health workers in Liberia through the mHero Health Information System (HIS). The system was deployed as a possible remedy to some of the information communication challenges identified during the 2014 West African Ebola outbreak. A second group was given a case about the use of RapidPro to remind women to attend antenatal care (ANC) for preventive malaria medicine in Guinea. The USAID StopPalu project goal was to improve the health of infants by increasing the percent of women attending ANC visits. The final group was given a case about using remote images to assist East African pastoralists. The Satellite Assisted Pastoral Resource Management System (SAPARM) informs pastoralists of vegetation through remote sensing imagery so they can make better decisions about migrating their livestock.

After familiarizing ourselves with the particulars of the case studies, each group was tasked to present their findings to all participants after pondering a series of questions. Some of the issues under discussion included

(1) “How would you assess your MERL Tech’s relevance?”

(2) “How would you evaluate the effectiveness of your MERL Tech?”

(3) “How would you measure efficiency?” and

(4) “How will you access sustainability?”.

Each group came up with some innovative answers to the questions posed and our facilitators and session leads (Alexandra Robinson & Sutyajeet Soneja from USAID and Molly Chen from RTI) will soon synthesize the workshop findings and notes into a concise written brief for the MERL Tech community.

Before the workshop closed we were all introduced to the great work done by SIMLab (Social Impact Lab) in this area through their SIMLab Monitoring and Evaluation Framework. The framework identifies key criteria for evaluating M&E including:

  1. Relevance – The extent to which the technology choice is appropriately suited to the priorities and capacities of the context of the target group or organization.
  2. Effectiveness – A measure of the extent to which an information and communication channel, technology tool, technology platform, or a combination of these attains its objectives.
  3. Efficiency – Measure of the outputs (qualitative and quantitative) in relation to the inputs.
  4. Impact – The positive and negative changed produced by technology introduction, change in a technology tool, or platform on the overall development intervention (directly or indirectly; intended or unintended).
  5. Sustainability – Measure of whether the benefits of a technology tool or platform are likely to continue after donor funding has been withdrawn.
  6. Coherence – How related is the technology to the broader policy context (development, market, communication networks, data standards & interoperability mandates, and national & international law) within which the technology was developed and implemented.

While it’s unfortunate that SIMLab stopped most operations in early September 2017, their exceptional work in this and other areas lives on and you can access the full framework here.

I learned a great deal in this session from the facilitators and my colleagues attending the workshop. I would encourage everyone in the MERL Tech community to take the ideas generated during this workshop and the great work done by SIMLab into their development practice. We certainly intend to integrate much of these insights into our work at OpenWise. Read more about “The Evidence Agenda” here on SIMLab’s blog. 

 

 

 

You can’t have Aid…without AI: How artificial intelligence may reshape M&E

by Jacob Korenblum, CEO of Souktel Digital Solutions

Photo: wikipedia.org/

Potential—And Risk

The rapid growth of Artificial Intelligence—computers behaving like humans, and performing tasks which people usually carry out—promises to transform everything from car travel to personal finance. But how will it affect the equally vital field of M&E? As evaluators, most of us hate paper-based data collection—and we know that automation can help us process data more efficiently. At the same time, we’re afraid to remove the human element from monitoring and evaluation: What if the machines screw up?

Over the past year, Souktel has worked on three areas of AI-related M&E, to determine where new technology can best support project appraisals. Here are our key takeaways on what works, what doesn’t, and what might be possible down the road.

Natural Language Processing

For anyone who’s sifted through thousands of Excel entries, natural language processing sounds like a silver bullet: This application of AI interprets text responses rapidly, often matching them against existing data sets to find trends. No need for humans to review each entry by hand! But currently, it has two main limitations: First, natural language processing works best for sentences with simple syntax. Throw in more complex phrases, or longer text strings, and the power of AI to grasp open-ended responses goes downhill. Second, natural language processing only works for a limited number of (mostly European) languages—at least for now. English and Spanish AI applications? Yes. Chichewa or Pashto M&E bots? Not yet. Given these constraints, we’ve found that AI apps are strongest at interpreting basic misspelled answer text during mobile data collection campaigns (in languages like English or French). They’re less good at categorizing open-ended responses by qualitative category (positive, negative, neutral). Yet despite these limitations, AI can still help evaluators save time.

Object Differentiation

AI does a decent job of telling objects apart; we’ve leveraged this to build mobile applications which track supply delivery more quickly & cheaply. If a field staff member submits a photo of syringes and a photo of bandages from their mobile, we don’t need a human to check “syringes” and “bandages” off a list of delivered items. The AI-based app will do that automatically—saving huge amounts of time and expense, especially during crisis events. Still, there are limitations here too: While AI apps can distinguish between a needle and a BandAid, they can’t yet tell us whether the needle is broken, or whether the BandAid is the exact same one we shipped. These constraints need to be considered carefully when using AI for inventory monitoring.

Comparative Facial Recognition

This may be the most exciting—and controversial—application of AI. The potential is huge: “Qualitative evaluation” takes on a whole new meaning when facial expressions can be captured by cameras on mobile devices. On a more basic level, we’ve been focusing on solutions for better attendance tracking: AI is fairly good at determining whether the people in a photo at Time A are the same people in a photo at Time B. Snap a group pic at the end of each community meeting or training, and you can track longitudinal participation automatically. Take a photo of a larger crowd, and you can rapidly estimate the number of attendees at an event.

However, AI applications in this field have been notoriously bad at recognizing diversity—possibly because they draw on databases of existing images, and most of those images contain…white men. New MIT research has suggested that “since a majority of the photos used to train [AI applications] contain few minorities, [they] often have trouble picking out those minority faces”. For the communities where many of us work (and come from), that’s a major problem.

Do’s and Don’ts

So, how should M&E experts navigate this imperfect world? Our work has yielded a few “quick wins”—areas where Artificial Intelligence can definitely make our lives easier: Tagging and sorting quantitative data (or basic open-ended text), simple differentiation between images and objects, and broad-based identification of people and groups. These applications, by themselves, can be game-changers for our work as evaluators—despite their drawbacks. And as AI keeps evolving, its relevance to M&E will likely grow as well. We may never reach the era of robot focus group facilitators—but if robo-assistants help us process our focus group data more quickly, we won’t be complaining.

MERL Tech Round Up | November 1, 2017

It’s time for our second MERL Tech Round Up, a monthly compilation of MERL Tech News!

On the MERL Tech Blog:

We’ve been posting session summaries from MERL Tech DC. Here are some posts you may have missed in October:

Stuff we’re reading/watching/bookmarking:

There’s quite a bit to learn both in our “MERL / Tech” sector and in related sectors whose experiences are relatable to MERL Tech. Some thought-provoking pieces here:

Events:

Jobs

Head over to ICT4DJobs for a ton of tech related jobs. Here are some interesting ones for folks in the MERL Tech space:

If you’re not already signed up to the Pelican Initiative: Platform for Evidence-based Learning & Communication for Social Change, we recommend doing that. You will find all kinds of MERL and MERLTech related jobs and MERL-related advice. (Note: the Platform is an extremely active forum, so you may want to adjust your settings to receive weekly compilations).

Tag us on Twitter using #MERLTech if you have resources, events, or other news you’d like us to include here!

Don’t forget to submit your session ideas for MERL Tech London by November 10th!

Big data, big problems, big solutions

by Alvaro Cobo-Santillan, Catholic Relief Services (CRS); Jeff Lundberg, CRS; Paul Perrin, University of Notre Dame; and Gillian Kerr, LogicalOutcomes Canada. 

In the year 2017, with all of us holding a mini-computer at all hours of the day and night, it’s probably not too hard to imagine that “A teenager in Africa today has access to more information than the President of United States had 15 years ago”. So it also stands to reason that the ability to appropriately and ethically grapple with the use of that immense amount information has grown proportionately.

At the September MERL Tech event in Washington D.C. a panel that included folks from University of Notre Dame, Catholic Relief Services, and LogicalOutcomes spoke at length about three angles of this opportunity involving big data.

The Murky Waters of Development Data

What do we mean when we say that the world of development—particularly evaluation—data is murky? A major factor in this sentiment is the ambiguous polarity between research and evaluation data.

  • “Research seeks to prove; evaluation seeks to improve.” – CDC
  • “Research studies involving human subjects require IRB review. Evaluative studies and activities do not.”
Source: Patricia Rogers (2014), Ways of Framing the difference between research and evaluation, Better Evaluation Network.

This has led to debates as to the actual relationship between research and evaluation. Some see them as related, but separate activities, others see evaluation as a subset of research, and still others might posit that research is a specific case of evaluation.

But regardless, though motivations of the two may differ, research and evaluation look the same due to their stakeholders, participants, and methods.

If that statement is true, then we must hold both to similar protections!

What are some ways to make the waters less murky?

  • Deeper commitment to informed consent
  • Reasoned use of identifiers
  • Need to know vs. nice to know
  • Data security and privacy protocols
  • Data use agreements and protocols for outside parties
  • Revisit NGO primary and secondary data IRB requirements

Alright then, what can we practically do within our individual agencies to move the needle on data protection?

  • In short, governance. Responsible data is absolutely a crosscutting responsibility, but can be primarily championed through close partnerships between the M&E and IT Departments
  • Think about ways to increase usage of digital M&E – this can ease the implementation of R&D
  • Can existing agency processes and resources be leveraged?
  • Plan and expect to implement gradual behavior change and capacity building as a pre-requisite for a sustainable implementation of responsible data protections
  • Think in an iterative approach. Gradually introduce guidelines, tools and training materials
  • Plan for business and technical support structures to support protections

Is anyone doing any of the practical things you’ve mentioned?

Yes! Gillian Kerr from LogicalOutcomes spoke about highlights from an M&E system her company is launching to provide examples of the type of privacy and security protections they are doing in practice.

As a basis for the mindset behind their work, she notably presented a pretty fascinating and simple comparison of high risk vs. low risk personal information – year of birth, gender, and 3 digit zip code is unique for .04% of US residents, but if we instead include a 5 digit zip code over 50% of US residents could be uniquely identified. Yikes.

In that vein, they are not collecting names or identification and only year of birth (not month or day) and seek for minimal sensitive data defining data elements by level of risk to the client (i.e. city of residence – low, glucose level – medium, and HIV status – high).

In addition, asking for permission not only in the original agency permission form, but also in each survey. Their technical system maintains two instances – one containing individual level personal information with tight permission even for administrators and another with aggregated data with small cell sizes. Other security measures such as multi-factor authentication, encryption, and critical governance; such as regular audits are also in place.

It goes without saying that we collectively have ethical responsibilities to protect personal information about vulnerable people – here are final takeaways:

  • If you can’t protect sensitive information, don’t collect it.
  • If you can’t keep up with current security practices, outsource your M&E systems to someone who can.
  • Your technology roadmap should aspire to give control of personal information to the people who provide it (a substantial undertaking).
  • In the meantime, be more transparent about how data is being stored and shared
  • Continue the conversation by visiting https://responsibledata.io/blog
Register for MERL Tech London, March 19-20th 2018! Session ideas due November 10th.

Submit your session ideas for MERL Tech London by Nov 10th!

MERL Tech London

Please submit a session idea, register to attend, or reserve a demo table for MERL Tech London, on March 19-20, 2018, for in-depth sharing and exploration of what’s happening across the multidisciplinary monitoring, evaluation, research and learning field.

Building on MERL Tech London 2017, we will engage 200 practitioners from across the development and technology ecosystems for a two-day conference seeking to turn the theories of MERL technology into effective practice that delivers real insight and learning in our sector.

MERL Tech London 2018

Digital data and new media and information technologies are changing MERL practices. The past five years have seen technology-enabled MERL growing by leaps and bounds, including:

  • Adaptive management and ‘developmental evaluation’
  • Faster, higher quality data collection.
  • Remote data gathering through sensors and self-reporting by mobile.
  • Big Data and social media analytics
  • Story-triggered methodologies

Alongside these new initiatives, we are seeing increasing documentation and assessment of technology-enabled MERL initiatives. Good practice guidelines and new frameworks are emerging and agency-level efforts are making new initiatives easier to start, build on and improve.

The swarm of ethical questions related to these new methods and approaches has spurred greater attention to areas such as responsible data practice and the development of policies, guidelines and minimum ethical frameworks and standards for digital data.

Please submit a session idea, register to attend, or reserve a demo table for MERL Tech London to discuss all this and more! You’ll have the chance to meet, learn from, debate with 150-200 of your MERL Tech peers and to see live demos of new tools and approaches to MERL.

Submit Your Session Ideas Now!

Like previous conferences, MERL Tech London will be a highly participatory, community-driven event and we’re actively seeking practitioners in monitoring, evaluation, research, learning, data science and technology to facilitate every session.

Please submit your session ideas now. We are particularly interested in:

  • Case studies: Sharing end-to-end experiences/learning from a MERL Tech process
  • MERL Tech 101: How-to use a MERL Tech tool or approach
  • Methods & Frameworks: Sharing/developing/discussing methods and frameworks for MERL Tech
  • Data: Big, large, small, quant, qual, real-time, online-offline, approaches, quality, etc.
  • Innovations: Brand new, untested technologies or approaches and their application to MERL(Tech)
  • Debates: Lively discussions, big picture conundrums, thorny questions, contentious topics related to MERL Tech
  • Management: People, organizations, partners, capacity strengthening, adaptive management, change processes related to MERL Tech
  • Evaluating MERL Tech: comparisons or learnings about MERL Tech tools/approaches and technology in development processes
  • Failures: What hasn’t worked and why, and what can be learned from this?
  • Demo Tables: to share MERL Tech approaches, tools, and technologies
  • Other topics we may have missed!

Session Submission Deadline: Friday, November 10, 2017.

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early December and, if selected, you will be asked to submit an updated and final session title, summary and outline by Friday, January 19th, 2018.

Register Now!

Please register to attend, or reserve a demo table for MERL Tech London 2018 to examine these trends with an exciting mix of educational keynotes, lightning talks, and group breakouts, including an evening Fail Festival reception to foster needed networking across sectors.

We are charging a modest fee to better allocate seats and we expect to sell out quickly again this year, so buy your tickets or demo tables now. Event proceeds will be used to cover event costs and to offer travel stipends for select participants implementing MERL Tech activities in developing countries.

MERL Tech Maturity Models

by Maliha Khan, a development practitioner in the fields of design, measurement, evaluation and learning. Maliha led the Maturity Model sessions at MERL Tech DC and Linda Raftree, independent consultant and lead organizer of MERL Tech.

MERL Tech is a platform for discussion, learning and collaboration around the intersection of digital technology and Monitoring, Evaluation, Research, and Learning (MERL) in the humanitarian and international development fields. The MERL Tech network is multidisciplinary and includes researchers, evaluators, development practitioners, aid workers, technology developers, data analysts and data scientists, funders, and other key stakeholders.

One key goal of the MERL Tech conference and platform is to bring people from diverse backgrounds and practices together to learn from each other and to coalesce MERL Tech into a more cohesive field in its own right — a field that draws from the experiences and expertise of these various disciplines. MERL Tech tends to bring together six broad communities:

  • traditional M&E practitioners, who are interested in technology as a tool to help them do their work faster and better;
  • development practitioners, who are running ICT4D programs and beginning to pay more attention to the digital data produced by these tools and platforms;
  • business development and strategy leads in organizations who want to focus more on impact and keep their organizations up to speed with the field;
  • tech people who are interested in the application of newly developed digital tools, platforms and services to the field of development, but may lack knowledge of the context and nuance of that application
  • data people, who are focused on data analytics, big data, and predictive analytics, but similarly may lack a full grasp of the intricacies of the development field
  • donors and funders who are interested in technology, impact measurement, and innovation.

Since our first series of Technology Salons on ICT and M&E in 2012 and the first MERL Tech conference in 2014, the aim has been to create stronger bridges between these diverse groups and encourage the formation of a new field with an identity of its own — In other words, to move people beyond identifying as, say, an “evaluator who sometimes uses technology,” and towards identifying as a member of the MERL Tech space (or field or discipline) with a clearer understanding of how these various elements work together and play off one another and how they influence (and are influenced by) the shifts and changes happening in the wider ecosystem of international development.

By building and strengthening these divergent interests and disciplines into a field of their own, we hope that the community of practitioners can begin to better understand their own internal competencies and what they, as a unified field, offered to international development. This is a challenging prospect, as beyond their shared use of technology to gather, analyze, and store data and an interest in better understanding how, when, why, where, (etc.) these tools work for MERL and for development/humanitarian programming, there aren’t many similarities between participants.

At the MERL Tech London and MERL Tech DC conferences in 2017, we made a concerted effort to get to the next level in the process of creating a field. In London in February, participants created a timeline of technology and MERL and identified key areas that the MERL Tech community could work on strengthening (such as data privacy and security frameworks and more technological tools for qualitative MERL efforts). At MERL Tech DC, we began trying to understand what a ‘maturity model’ for MERL Tech might look like.

What do we mean by a ‘maturity model’?

Broadly, maturity models seek to qualitatively assess people/culture, processes/structures, and objects/technology to craft a predictive path that an organization, field, or discipline can take in its development and improvement.

Initially, we considered constructing a “straw” maturity model for MERL Tech and presenting it at the conference. The idea was that our straw model’s potential flaws would spark debate and discussion among participants. In the end, however, we decided against this approach because (a) we were worried that our straw model would unduly influence people’s opinions, and (b) we were not very confident in our own ability to construct a good maturity model.

Instead, we opted to facilitate a creative space over three sessions to encourage discussion on what a maturity model might look like, and what it might contain. Our vision for these sessions was to get participants to brainstorm in mixed groups containing different types of people- we didn’t want small subsets of participants to create models independently without the input of others.

In the first session, “Developing a MERL Tech Maturity Model”, we invited participants to consider what a maturity model might look like. Could we begin to imagine a graphic model that would enable self-evaluation and allow informed choices about how to best develop competencies, change and adjust processes and align structures in organizations to optimize using technology for MERL or indeed other parts of the development field?

In the second session, “Where do you sit on the Maturity Model?” we asked participants to use the ideas that emerged from our brainstorm in the first session to consider their own organizations and work, and compare them against potential maturity models. We encouraged participants to assess themselves using green (young sapling) to yellow (somewhere in the middle) and red (mature MERL Tech ninja!) and to strike up a conversation with other people in the breaks on why they chose that color.

In our third session, “Something old, something new”, we consolidated and synthesized the various concepts participants had engaged with throughout the conference. Everyone was encouraged to reflect on their own learning, lessons for their work, and what new ideas or techniques they may have picked up on and might use in the future.

The Maturity Models

As can be expected, when over 300 people take marker and crayons to paper, many a creative model emerges. We asked the participants to gallery walk the models over the next day during the breaks and vote on their favorite models.

We won’t go into detail of what all the 24 the models showed, but there were some common themes that emerged from the ones that got the most votes – almost all maturity models include dimensions (elements, components) and stages, and a depiction of potential progression from early stages to later stages across each dimension. They all also showed who the key stakeholders or players were, and some had some details on what might be expected of them at different stages of maturity.

Two of the models (MERLvana and the Data Appreciation Maturity Model – DAMM) depicted the notion that reaching maturity was never really possible and the process was an almost infinite loop. As the presenters explained MERLvana “it’s an impossible to reach the ideal state, but one must keep striving for it, in ever closer and tighter loops with fewer and fewer gains!”

MERLvana
MERLvana
Data
Data Appreciation Maturity Model

“MERL-tropolis” had clearly defined categories (universal understanding, learning culture and awareness, common principles, and programmatic strategy) and the structures/ buildings needed for those (staff, funding, tools, standard operating procedures, skills).

MERLTropolis
MERLTropolis

The most popular was “The Data Turnpike” which showed the route from the start of “Implementation with no data” to the finish line of “Technology, capacity and interest in data and adaptive management” with all the pitfalls along the way (misuse, not timely, low ethics etc) marked to warn of the dangers.

data turnpike
The Data Turnpike

As organizers of the session, we found the exercises both interesting and enlightening, and we hope they helped participants to begin thinking about their own MERL Tech practice in a more structured way. Participant feedback on the session was on polar extremes. Some people loved the exercise and felt that it allowed them to step back and think about how they and their organization were approaching MERL Tech and how they could move forward more systematically with building greater capacities and higher quality work. Some told us that they left with clear ideas on how they would work within their organizations to improve and enhance their MERL Tech practice, and that they had a better understanding of how to go about that. A few did not like that we had asked them to “sit around drawing pictures” and some others felt that the exercise was unclear and that we should have provided a model instead of asking people to create one. [Note: This is an ongoing challenge when bringing together so many types of participants from such diverse backgrounds and varied ways of thinking and approaching things!]

We’re curious if others have worked with “maturity models” and if they’ve been applied in this way or to the area of MERL Tech before. What do you think about the models we’ve shared? What is missing? How can we continue to think about this field and strengthen our theory and practice? What should we do at MERL Tech London in March 2018 and beyond to continue these conversations?

Five lessons learned from applying design thinking to data use

by Amanda Makulec, Data Visualization Lead, Excella Consulting and Barb Knittel, Research, Monitoring & Evaluation Advisor, John Snow Inc. Amanda and Barb led “How the Simpsons Make Data Use Happen” at MERL Tech DC.

MERL-DesignforDataUse 1

Workshopping ways to make data use happen.

Human centered design isn’t a new concept. We’ve heard engineers, from aerospace to software, quietly snicker as they’ve seen the enthusiasm for design thinking explode within the social good space in recent years. “To start with the end user in mind? Of course! How else would you create a product someone wants to use?”

However, in our work designing complex health information systems, dashboards, and other tools and strategies to improve data use, the idea of starting with the end user does feel relatively new.

Thinking back to graduate school nearly ten years ago, dashboard design classes focused on the functional skills, like how to use a pivot table in Excel, not on the complex processes of gathering user requirements to design something that could not only delight the end user, but be co-designed with them.

As part of designing for data use and data visualization design workshops, we’ve collaborated with design firms to find new ways to crack the nut of developing products and processes that help decisionmakers use information. Using design thinking tools like ranking exercises, journey maps, and personas has helped users identify and find innovative ways to address critical barriers to data use.

If you’re thinking about integrating design thinking approaches into data-centered projects, here are our five key considerations to take into account before you begin:

  1. Design thinking is a mindset, not a workshop agenda. When you’re setting out to incorporate design thinking into your work, consider what that means throughout the project lifecycle. From continuous engagement and touchpoints with your data users to
  1. Engage the right people – you need a diverse range of perspectives and experiences to uncover problems and co-create solutions. This means thinking of the usual stakeholders using the data at hand, but also engaging those adjacent to the data. In health information systems, this could be the clinicians reporting on the registers, the mid-level managers at the district health office, and even the printer responsible for distributing paper registers.
  1. Plan for the long haul. Don’t limit your planning and projections of time, resources, and end user engagement to initial workshops. Coming out of your initial design workshops, you’ll likely have prototypes that require continued attention to functionally build and implement.
  1. Focus on identifying and understanding the problem you’ll be solving. You’ll never be able to solve every problem and overcome every data use barrier in one workshop (or even in one project). Work with your users to develop a specific focus and thoroughly understand the barriers and challenges from their perspectives so you can tackle the most pressing issues (or choose deliberately to work on longer term solutions to the largest impediments).
  1. The journey matters as much as the destination. One of the greatest ah-ha moments coming out of these workshops has been from participants who see opportunities to change how they facilitate meetings or manage teams by adopting some of the activities and facilitation approaches in their own work. Adoption of the prototypes shouldn’t be your only metric of success.

The Designing for Data Use workshops were funded by (1) USAID and implemented by the MEASURE Evaluation project and (2) the Global Fund through the Data Use Innovations Fund. Matchboxology was the design partner for both sets of workshops, and John Snow Inc. was the technical partner for the Data Use Innovations sessions. Learn more about the process and learning from the MEASURE Evaluation workshops in Applying User Centered Design to Data Use Challenges: What we Learned and see our slides from our MERL Tech session “The Simpsons, Design, and Data Use” to learn more.

MERL Tech Round Up | October 2, 2017

We’ll be experimenting with a monthly round-up of MERL Tech related content (bi-weekly if there’s enough to fill a post). Let us know if it’s useful! We aim to keep it manageable and varied, rather than a laundry list of every possible thing. The format, categories, and topics will evolve as we see how it goes and what the appetite is.

If you have anything you’d like to share or see featured, feel free to send it on over or post on Twitter using the #MERLTech hashtag.

On the MERL Tech Blog:

Big Data in Evaluation – Michael Bamberger discusses the future of development evaluation in the age of Big Data and ways to build bridges between evaluators and Big Data analysis. Rick Davies (Monitoring and Evaluation News) raises some great points in the comments (and Michael replies).

Experiences with Mobile case management for multi-dimensional accountability from Oxfam and Survey CTO.

Thoughts on MERL Tech Maturity Models & Next Generation Transparency & Accountability from Megan Colner (Open Society Foundations) and Alison Miranda (Transparency and Accountability Initiative).

The best learning at MERL Tech DC came from sharing failures from Ambika Samarthya-Howard (Praekelt.org).

We’ll be posting more MERL Tech DC summaries and wrap-up posts over the next month or two. We’re also gearing up for MERL Tech London coming up in March 2018. Stay tuned for more information on that.

Stuff we’re reading / watching:

New research (Making All Voices Count research team) on ICT-mediated citizen engagement. What makes it transformative? 

Opportunities and risks in emerging technologies, including white papers on Artificial IntelligenceAlgorithmic Accountability; and Control of Personal Data (The Web Foundation).

Research on Privacy, Security, and Digital Inequality: How Technology Experiences and Resources Vary by Socioeconomic Status, Race, and Ethnicity in the United States from Mary Madden (Data & Society).

Tools, frameworks and guidance we’re bookmarking:

A framework for evaluating inclusive technology, technology for social impact and ICT4D programming (SIMLab) and an example of its application. The framework is open source, so you can use and adapt it!

survey tool and guidance for assessing women’s ICT access and use (FHI 360’s mSTAR project).  (Webinar coming up on Oct 10th)

Series on data management (DAI) covering 1) planning and collecting data; 2) managing and storing data; and 3) getting value and use out of the data that’s collected through analysis and visualization.

Events and training:

Webinar on using ICT in monitoring and evaluation of education programming for refugee populations (USAID and INEE). Recording and presentations from the Sep 28th event here.

Webinar on assessing women’s ICT access and use, Oct 10th (Nethope, USAID and mSTAR/FHI 360.

Let us know of upcoming events we should feature.

Jobs:

Send us vacancies for MERL Tech-related jobs, consultants, RFPs and we’ll help spread the word.

Failures are the way forward

By Ambika Samarthya-Howard, Head of Communications at Praekelt.org. This post also appears on the Praekelt.org blog.

Marc Mitchell, President of D-Tree International, gives his Lightning Talk: When the Control Group Wins.
Marc Mitchell, President of D-Tree International, gives his Lightning Talk: “When the Control Group Wins.”

Attending conferences often reminds me of dating: you put your best foot forward and do yourself up, and hide the rest for a later time. I always found it refreshing when people openly put their cards on the table.

I think that’s why I especially respected and enjoyed my experience at MERL Tech in DC last week. One of the first sessions I went to was a World Cafe style break out exploring how to be inclusive in M&E tech in the field. The organisations, like Global Giving and Keystone, posed hard questions about community involvement in data collection at scale, and how to get people less familiar or with less access to technology involved in the process. They didn’t have any of the answers. They wanted to learn from us.

This was followed by lightning talks after lunch where organisations gave short talks.  One organisation spoke very openly about how much money and time they were wasting on their data collection technologies. Another organisation confessed their lack of structure and organisation, and talked about collaborating with organisations like DataKind to make sense of their data. Anahi Ayala Iacucci from Internews did a presentation on the pitfalls and realities of M&E: “we all skew the results in order to get the next round of funding.” She fondly asked us to work together so we could “stop farting in the wind”. D-Tree International spoke about a trial around transport vouchers for pregnant women in Zanzibar, and how the control group that did not receive any funding actually did better.  They had to stop funding the vouchers.

The second day I attended an entire session where we looked at existing M&E reports available online to critique their deficiencies and identify where the field was lacking in knowledge dissemination. As a Communications person, looking at the write-ups of the data ironically gave me instant insight into ways forward and where gaps could be filled — which I believe is exactly what the speakers of the session intended. When you can so clearly see why and how things aren’t working, it actually inspires a different approach and way of working.

I was thoroughly impressed with the way people shared at MERL Tech. When you see an organisation able to talk so boldly about its learning curves or gaps, you respect their work, growth, and learnings.  And that is essentially the entire purpose of a conference.

Back to dating… and partnerships. Sooner or later, if the relationship works out, your partner is going to see you in the a.m. for who you really are. Why not cut to the chase, and go in with your natural look?  Then you can take the time to really do any of the hard work together, on the same footing.