Tag Archives: 2018

Technologies in monitoring and evaluation | 5 takeaways

Bloggers: Martijn Marijnis and Leonard Zijlstra. This post originally appeared on the ICCO blog on April 3, 2018.
.

Technologies in monitoring and evaluation | 5 takeaways

On March 19 and 20 ICCO participated in the MERL Tech 2018 in London. The conference explores the possibilities of technology in monitoring, evaluation, learning and research in development. About 200 like-minded participants from various countries participated. Key issues on the agenda were data privacy, data literacy within and beyond your organization, human-centred monitoring design and user-driven technologies. Interesting practices where shared, amongst others in using blockchain technologies and machine learning. Here are our most important takeaways:

1)  In many NGOs data gathering still takes place in silo’s

Oxfam UK shared some knowledgeable insights and practical tips in putting in place an infrastructure that combines data: start small and test, e.g. by building up a strong country use case; discuss with and learn from others; ensure privacy by design and make sure senior leadership is involved. ICCO Cooperation currently faces a similar challenge, in particular in combining our household data with our global result indicators.

2)  Machine learning has potential for NGOs

While ICCO recently started to test machine learning in the food security field (see this blog) other organisations showcased interesting examples: the Wellcome Trust shared a case where they tried to answer the following question: Is the organization informing and influencing policy and if so, how? Wellcome teamed up their data lab and insight & analysis team and started to use open APIs to pull data in combination with natural language processing to identify relevant cases of research supported by the organization. With their 8.000 publications a year this would be a daunting task for a human team. First, publications linked to Wellcome funding were extracted from a European database (EPMC) in combination with end of grant reports. Then WHO’s reference section was scraped to see if and to what extent WHO’s policy was influenced and to identify potential interesting cases for Wellcome’s policy team.

3)  Use a standardized framework for digital development

See digitalpinciples.org. It gives – amongst others – practical guidelines on how to use open standards and open data, how data can be reused, how privacy and security can be addressed, how users can and should be involved in using technologies in development projects. It is a useful framework for evaluating your design.

4)  Many INGOs get nervous these days about blockchain technology

What is it, a new hype or a real game changer? For many it is just untested technology with high risks and little upside for the developing world. But, for example INGOs working in agriculture value chains or in humanitarian relief operations, its potential is definitely consequential enough to merit a closer look. It starts with the underlying principle, that users of a so-called blockchain can transfer value, or assets, between each other without the need for a trusted intermediary. The running history of the transactions is called the blockchain, and each transaction is called a block. All transactions are recorded in a ledger that is shared by all users of a blockchain.

The upside of blockchain applications is the considerable time and money saving aspect of it. Users rely on this shared ledger to provide a transparent view into the details of the assets or values, including who owns them, as well as descriptive information such as quality or location. Smallholder farmers could benefit (e.g. real-time payment on delivery, access to credit), so can international sourcing companies (e.g. traceability of produce without certification), banks (e.g. cost-reductions, risk-reduction), as much as refugees and landless (e.g. registration, identification). Although we haven’t yet seen large-scale adoption of blockchain technology in the development sector, investors like the Bill and Melinda Gates Foundation and various venture capitalists are paying attention to this space.

But one of the main downsides or challenges for blockchain, like with agricultural technology at large, is connecting the technology to viable business models and compelling use cases. With or without tested technology, this is hard enough as it is and requires innovation, perseverance and focus on real value for the end-user; ICCO’s G4AW projects gain experience with blockchain.

5)  Start thinking about data-use incentives

Over the years, ICCO has made significant investments in monitoring & evaluation and data skills training. Yet limited measurable results of increased data use can be seen, like in many other organizations. US-based development consultant Cooper&Smith shared revealing insights into data usage incentives. It tested three INGOs working across five regions globally. The hypothesis was, that better alignment of data-use training incentives leads to increased data use later on. They looked at both financial and non-financial rewards that motivate individuals to behave in a particular way. Incentives included different training formats (e.g. individual, blended), different hardware (e.g. desktop, laptop, mobile phone), recognition (e.g. certificate, presentation at a conference), forms of feedback & support (e.g. one-on-one, peer group) and leisure time during the training (e.g. 2 hours/week, 12 hours/week). Data use was referred to as the practice of collecting, managing, analyzing and interpreting data for making program policy and management decisions.

They found considerable differences in appreciation of the attributes. For instance, respondents overwhelmingly prefer a certificate in data management, but instead they currently receive primarily no recognition or only recognition from their supervisor. Or  one region prefers a certificate while the other prefers attending an international conference as reward. Or that they prefer one-on-one feedback but instead they receive only peer-2-peer support. The lesson here is, that while most organizations apply a ‘one-size fits all’-reward system (or have no reward system at all), this study points at the need to develop a culturally sensitive and geographically smart reward system to see real increase in data usage.

For many NGOs the data revolution has just begun, but we are underway!

Present or lead a session at MERL Tech DC!

Please sign up to present, register to attend, or reserve a demo table for MERL Tech DC 2018 on September 6-7, 2018 at FHI 360 in Washington, DC.

We will engage 300 practitioners from across the development ecosystem for a two-day conference seeking to turn the theories of MERL technology into effective practice that delivers real insight and learning in our sector.

MERL Tech DC 2018, September 6-7, 2018

Digital data and new media and information technologies are changing monitoring, evaluation, research and learning (MERL). The past five years have seen technology-enabled MERL growing by leaps and bounds. We’re also seeing greater awareness and concern for digital data privacy and security coming into our work.

The field is in constant flux with emerging methods, tools and approaches, such as:

  • Adaptive management and developmental evaluation
  • Faster, higher quality data collection
  • Remote data gathering through sensors and self-reporting by mobile
  • Big data, data science, and social media analytics
  • Story-triggered methodologies

Alongside these new initiatives, we are seeing increasing documentation and assessment of technology-enabled MERL initiatives. Good practice guidelines are emerging and agency-level efforts are making new initiatives easier to start, build on and improve.

The swarm of ethical questions related to these new methods and approaches has spurred greater attention to areas such as responsible data practice and the development of policies, guidelines and minimum ethical standards for digital data.

Championing the above is a growing and diversifying community of MERL practitioners, assembling from a variety of fields; hailing from a range of starting points; espousing different core frameworks and methodological approaches; and representing innovative field implementers, independent evaluators, and those at HQ that drive and promote institutional policy and practice.

Please sign up to present, register to attend, or reserve a demo table for MERL Tech DC to experience 2 days of in-depth sharing and exploration of what’s been happening across this cross-disciplinary field, what we’ve been learning, complex barriers that still need resolving, and debate around the possibilities and the challenges that our field needs to address as we move ahead.

Submit Your Session Ideas Now

Like previous conferences, MERL Tech DC will be a highly participatory, community-driven event and we’re actively seeking practitioners in monitoring, evaluation, research, learning, data science and technology to facilitate every session.

Please submit your session ideas now. We are looking for a range of topics, including:

  • Experiences and learning at the intersection of MERL and tech
  • Ethics, inclusion, safeguarding, and data privacy
  • Data (big data, data science, data analysis)
  • Evaluation of ICT-enabled efforts
  • The future of MERL
  • Tech-enabled MERL Failures

Visit the session submission page for more detail on each of these areas.

Submission Deadline: Monday, April 30, 2018 (at midnight EST)

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early June and, if selected, you will be asked to submit the final session title, summary and outline by June 30.

Register Now

Please sign up to present or register to attend MERL Tech DC 2018 to examine these trends with an exciting mix of educational keynotes, lightning talks, and group breakouts, including an evening reception and Fail Fest to foster needed networking across sectors and an exploration of how we can learn from our mistakes.

We are charging a modest fee to better allocate seats and we expect to sell out quickly again this year, so buy your tickets or demo tables now. Event proceeds will be used to cover event costs and to offer travel stipends for select participants implementing MERL Tech activities in developing countries.

You can also submit session ideas for MERL Tech Jozi, coming up on August 1-2, 2018! Those are due on March 31st, 2018!

MERL Tech London: What’s Your Organisation’s Take on Data Literacy, Privacy and Ethics?

 It first appeared here on March 26th, 2018.

ICTs and data are increasingly being used for monitoring, evaluation, research and learning (MERL). MERL Tech London was an open space for practitioners, techies, researchers and decision makers to discuss their good and not so good experiences. This blogpost is a reflection of the debates that took place during the conference.

Is data literacy still a thing?

Data literacy is “the ability to consume for knowledge, produce coherently and think critically about data.” The perception of data literacy varies depending on the stakeholder’s needs. Being data literate for an M&E team, for example, means possessing statistics skills including collecting and combining large data sets. Program team requires different level of data literacy: the competence to carefully interpret and communicate meaningful stories using processed data (or information) to reach the target audiences.

Data literacy is – and will remain – a priority in development. The current debate is no longer about whether an organisation should use data or not. It’s rather how well the organisation can use data to achieve their objectives. Yet, organisation’s efforts are often concentrated in just one part of the information value chain, data collection. Data collection in itself is not the end goal. Data has to be processed into information and knowledge for making informed decisions and actions.

This doesn’t necessary imply that the decision making is purely based on data, nor that data can replace the role of decision makers. Quite the opposite: data-informed decision making strikes balance between expertise and information. It also takes data limitations into account. Nevertheless, one can’t become a data-informed organisation without being data literate.

What’s your organisation’s data strategy?

The journey of becoming a data-informed organisation can take some time. Poor data quality, duplication efforts and underinvestment are classic obstacles requiring a systematic solution (see Tweet). The commitment from senior management team should be secured for that. Data team has to be established. Staff members need access to relevant data platforms and training. More importantly, the organisation has to embrace the cultural change towards valuing evidence and acting on positive and negative findings

Marten Schoonman@mato74
Responsible data handling workgroup: mindmapping the relevant subjects @MERLTech

Organisations seek to balance between (data) demands and priorities. Some invest hundreds of thousands dollars for setting up a data team to articulate the organisation’s needs and priorities, as well as to mobilise technical support. A 3-5 years strategic plan is created to coordinate efforts between country offices.

Others take a more modest approach. They recruit few data scientists to support MERL activities of analysing particularly large amounts of project data. The data scientist role evolves along the project growth. In both cases, leadership is the key driver for shifting the culture towards becoming a data-informed organisation.

Should an organisation use certain data because it can?

The organisation working with data usually faces challenges around privacy, legality, ethics and grey areas, such as bias and power dynamics between data collectors and their target groups. The use of biometric data in humanitarian settings is an example where all these tensions collide. Biometric data, e.g. fingerprint, iris scan, facial recognition – is powerful, yet invasive. While proven beneficial, biometric data is vulnerable to data breach and misuse, e.g. profiling and tracking. The practice raises critical questions: does the target group, e.g. refugees, have the option to refuse handling over their sensitive personal data? If so, will they still be entitled to receive aid assistance? To what extent the target group is aware how their sensitive personal data will be used and shared, including in the unforeseen circumstances?

The people’s privacy, safety and security are main priorities in any data work. The organisation should uphold the highest standards and set an example. In those countries where regulatory frameworks are lagging behind data and technology, organisations shouldn’t abuse their power. When the risk of using a certain data outweighs the benefits, or in doubt, the organisation should take a pause and ask itself some necessary questions from the perspective of its target groups. Oxfam which dismissed – following two years of internal discussions and intensive research – the idea of using biometric data in any of their project should be seen as a positive example.

To conclude, the benefits of data can only be realised when an organisation enjoys visionary leadership, sufficient capacity and upholds its principles. No doubts, this is easier being said than done; it requires time and patience. All these efforts, however, are necessary for a high-achieving organisations.

More reading:

**Save the date for MERL Tech Jozi coming up on Aug 1-2!  Session ideas are due this Friday (March 31st).

Please Submit Session Ideas for MERL Tech Jozi

We’re thrilled to announce that we’re organizing MERL TEch Jozi for August of 2018!

Please submit your session ideas or reserve your demo table now, to explore what’s happening with innovation, digital data, and new technologies across the monitoring, evaluation, research, and learning (MERL) fields.

MERL Tech Jozi will be in Johannesburg, South Africa, August 1-2, 2018!

At MERL Tech Jozi, we’ll build on earlier MERL Tech conferences in DC and London, engaging 100 practitioners from across the development and technology ecosystems for a two-day conference seeking to turn theories of MERL technology into effective practices that deliver real insight and learning in our sector.

MERL Tech is a lively, interactive, community-driven conference.  We’re actively seeking a diverse set of practitioners in monitoring, evaluation, research, learning, program implementation, management, data science, and technology to lead every session.

Submit your session ideas now.

We’re looking for sessions that focus on:

  • Discussions around good practice and evidence-based review
  • Innovative MERL approaches that incorporate technology
  • Future-focused thought provoking ideas and examples
  • Conversations about ethics, inclusion, and responsible policy and practice in MERL Tech
  • Exploration of complex MERL Tech challenges and emerging good practice
  • Workshop sessions with practical, hands-on exercises and approaches
  • Lightning Talks to showcase new ideas or to share focused results and learning
Submission Deadline: Saturday, March 31, 2018.

Session submissions are reviewed and selected by our steering committee. Presenters and session leads will have priority access to MERL Tech tickets. We will notify you whether your session idea was selected in late April and if selected, you will be asked to submit the final session title, summary and detailed session outline by June 1st, 2018

If you’d prefer to showcase your technology tool or platform to MERL Tech participants, you can reserve your demo table here.

MERL Tech is dedicated to creating a safe, inclusive, welcoming and harassment-free experience for everyone through our Code of Conduct.

MERL Tech Jozi is organized by Kurante and supported by the following sponsors. Contact Linda Raftree if you’d like to be a sponsor of MERL Tech Jozi too.

 

 

 

MERL Tech London 2018 Agenda is out!

We’ve been working hard over the past several weeks to finish up the agenda for MERL Tech London 2018, and it’s now ready!

We’ve got workshops, panels, discussions, case studies, lightning talks, demos, community building, socializing, and an evening reception with a Fail Fest!

Topics range from mobile data collection, to organizational capacity, to learning and good practice for information systems, to data science approaches, to qualitative methods using mobile ethnography and video, to biometrics and blockchain, to data ethics and privacy and more.

You can search the agenda to find the topics, themes and tools that are most interesting, identify sessions that are most relevant to your organization’s size and approach, pick the session methodologies that you prefer (some of us like participatory and some of us like listening), and to learn more about the different speakers and facilitators and their work.

Tickets are going fast, so be sure to snap yours up before it’s too late! (Register here!)

View the MERL Tech London schedule & directory.

 

Submit your session ideas for MERL Tech London by Nov 10th!

MERL Tech London

Please submit a session idea, register to attend, or reserve a demo table for MERL Tech London, on March 19-20, 2018, for in-depth sharing and exploration of what’s happening across the multidisciplinary monitoring, evaluation, research and learning field.

Building on MERL Tech London 2017, we will engage 200 practitioners from across the development and technology ecosystems for a two-day conference seeking to turn the theories of MERL technology into effective practice that delivers real insight and learning in our sector.

MERL Tech London 2018

Digital data and new media and information technologies are changing MERL practices. The past five years have seen technology-enabled MERL growing by leaps and bounds, including:

  • Adaptive management and ‘developmental evaluation’
  • Faster, higher quality data collection.
  • Remote data gathering through sensors and self-reporting by mobile.
  • Big Data and social media analytics
  • Story-triggered methodologies

Alongside these new initiatives, we are seeing increasing documentation and assessment of technology-enabled MERL initiatives. Good practice guidelines and new frameworks are emerging and agency-level efforts are making new initiatives easier to start, build on and improve.

The swarm of ethical questions related to these new methods and approaches has spurred greater attention to areas such as responsible data practice and the development of policies, guidelines and minimum ethical frameworks and standards for digital data.

Please submit a session idea, register to attend, or reserve a demo table for MERL Tech London to discuss all this and more! You’ll have the chance to meet, learn from, debate with 150-200 of your MERL Tech peers and to see live demos of new tools and approaches to MERL.

Submit Your Session Ideas Now!

Like previous conferences, MERL Tech London will be a highly participatory, community-driven event and we’re actively seeking practitioners in monitoring, evaluation, research, learning, data science and technology to facilitate every session.

Please submit your session ideas now. We are particularly interested in:

  • Case studies: Sharing end-to-end experiences/learning from a MERL Tech process
  • MERL Tech 101: How-to use a MERL Tech tool or approach
  • Methods & Frameworks: Sharing/developing/discussing methods and frameworks for MERL Tech
  • Data: Big, large, small, quant, qual, real-time, online-offline, approaches, quality, etc.
  • Innovations: Brand new, untested technologies or approaches and their application to MERL(Tech)
  • Debates: Lively discussions, big picture conundrums, thorny questions, contentious topics related to MERL Tech
  • Management: People, organizations, partners, capacity strengthening, adaptive management, change processes related to MERL Tech
  • Evaluating MERL Tech: comparisons or learnings about MERL Tech tools/approaches and technology in development processes
  • Failures: What hasn’t worked and why, and what can be learned from this?
  • Demo Tables: to share MERL Tech approaches, tools, and technologies
  • Other topics we may have missed!

Session Submission Deadline: Friday, November 10, 2017.

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early December and, if selected, you will be asked to submit an updated and final session title, summary and outline by Friday, January 19th, 2018.

Register Now!

Please register to attend, or reserve a demo table for MERL Tech London 2018 to examine these trends with an exciting mix of educational keynotes, lightning talks, and group breakouts, including an evening Fail Festival reception to foster needed networking across sectors.

We are charging a modest fee to better allocate seats and we expect to sell out quickly again this year, so buy your tickets or demo tables now. Event proceeds will be used to cover event costs and to offer travel stipends for select participants implementing MERL Tech activities in developing countries.