Tag Archives: MERL Tech London

Feedback Report from MERL Tech London 2018

MERL Tech London happened on March 19-20, 2018. Here are some highlights from session level feedback and the post-conference survey on the overall MERL Tech London experience.

If you attended MERL Tech London, please get in touch if you have any further questions about the feedback report or if you would like us to send you detailed (anonymized) feedback about a session you led. Please also be sure to send us your blog posts & session summaries so that we can post them on MERL Tech News!

Background on the data

  • 54 participants (~27%) filled out the post-conference survey via Google Forms.
  • 59 (~30%) rated and/or commented on individual sessions via the Sched app. Participants chose from three ’emoji’ options: a happy face 🙂 , a neutral face 😐 , and a sad face 🙁 . Participants could also leave their comments on individual sessions.
  • We received 616 session ratings/comments via Sched. Some people rated the majority of sessions they attended; others only rated 1-2 sessions.
  • Some reported that they did not feel comfortable rating sessions in the Sched app because they were unclear about whether session leads and the public could see the rating. In future, we will let participants know that only Sched administrators can see the identity of commenters and the ratings given to sessions.
  • We do not know if there is an overlap between those who filled out Sched and those that fed back via Google Forms because the Google Forms survey is anonymous.

Overall feedback

Here’s how survey participants rated the overall experience:

Breakout sessions– 137 ratings: 69% 🙂 30% 😐 and 13% 🙁

Responses were fairly consistent across both Sched ratings and Google Forms (the form asked people to identify their favorite session). Big data and data science sessions stand out with the highest number of favorable ratings and comments. General Data Protection Regulation (GDPR) and responsible data made an important showing, as did the session on participatory video in evaluation.

Sessions with favorable comments tended to include or combine elements of:

  • an engaging format
  • good planning and facilitation
  • strong levels of expertise
  • clear and understandable language and examples
  • and strategic use of case studies to point at a bigger picture that is replicable to other situations.

Below are the breakout sessions that received the most favorable ratings and comments overall. (Plenty of other sessions were also rated well but did not make the “top-top.”)

Session title

Comments

Be it resolved: In the near future, conventional evaluation and big data will be successfully integrated Brilliant session! Loved the format! Fantastic to have such experts taking part. Really appreciated the facilitation and that there was a time at the end for more open questions/discussion.
Innovative Use of Theory-Based and Data Science Evaluation Approaches Most interesting talk of the day (maybe more for the dedicated evaluation practitioners), very practical and easy to understand and I’m really looking forward to hearing more about the results as the work progresses!
Unpacking How Change Happened (or Didn’t): Participatory Video and Most Significant Change Right amout of explanation and using case studies to illustrate points and respond to questions rather than just stand alone case studies.
GDPR – What Is It and What Do We Do About It? GDPR and what we do about it – Great presentation starting off with some historical background, explaining with clarity how this new legislation is a rights-based approach and concluding on how for Oxfam this is not a compliance project but a modification in data culture. Amazing, innovative and the speaker knew his area very well.
The Best of Both Worlds? Combining Data Science and Traditional M&E to Understand Impact I learned so much from this session and was completely inspired by the presenter and the content. Clear – well paced – honest – open – collaborative and packed with really good insight. Amazing.
Big Data, Adaptive Management, and the Future of MERL Quite a mixed bag of presenters, with focus on different pieces on the overall topic. Speakers from Novometrics was particularly engaging and stimulated some good discussion.
Blockchain: Getting Past the Hype and Considering its Value for MERL Great group with good facilitation. Open ended question left lots of room for discussion without bias towards particular outcome. Learned lots and not just about blockchain.
LEAP, and How to Bring Data to Life in Your Organization Really great session, highly interactive, rich in concepts clearly and convincingly explained. No questions were left unanswered. Very insightful suggestions shared between the presenters/facilitators and the audience. Should be on the agenda of next MERL Tech Conference as well.
Who Watches the Watchers? Good Practice for Ethical MERL(Tech)

 

I came out with some really helpful material. Collaborative session and good workshop participants willing to share and mind map. Perhaps the lead facilitator could have been a bit more contextual. Not always clear. But, our table session was really helpful and output useful.
The GDPR is coming! Now what?! Practical Steps to Help You Get Ready Good session. Appreciated the handouts….

What could session leads improve on?

We also had a few sessions that were ranked closer to 😐 (somewhere around a 6 or 6.5 on a scale of 1-10). Why did participants rate some sessions lower?

  • “Felt like a product pitch”
  • “Title was misleading”
  • Participatory activity was unclear
  • Poor time management
  • “Case studies did not expand to learning for the sector – too much ‘this is what we did’ and not enough ‘this is what it means.””
  • Poor facilitation/moderation
  • “Too unstructured, meandering”
  • Low energy
  • “Only a chat among panelists, very little time for Q&A. No space to engage”

Additionally, some sessions had participants with very diverse levels of expertise and varied backgrounds and expectations, which seemed to affect session ratings.

Lightning Talks– 182 ratings: 74% 🙂 22% 😐 4% 🙁

Lightning talks consistently get the highest ratings at MERL Tech, and this year was no exception. As one participant said, “my favorite sessions were the lightning talks because they gave a really quick overview of really concrete uses of technology in M&E work. This really helped in getting an overview of the type of activities and projects that were going on.”  Participants rated almost all the lightning talks positively.

Plenary sessions– 192 ratings: 77% 🙂 21% 😐 and 2% 🙁

Here we include:  the welcome, discussions on MERL Tech priorities on Day 1, opening talks on both days, summary on Day 1, panel with donors, the closing ‘fishbowl’, and the Fail Fest.

Opening Talks:

  • People appreciated André’ Clarke’s stage setting talk on Day 1. “Clear, accessible and thoughtful.” “Nice deck!”
  • Anahi Ayala Iacucci’s opening talk on Day 2 was a hit: “Great keynote! Anahi is very engaging but also her content was really rich. Useful that she used a lot of examples and provided a lot of history.” And “Like Anahi says ‘The question to ask is what does technology do _to_ development, rather than what can technology do _for_ development.'”

Deep Dive into Priorities for the Sector:

  • Most respondents enjoyed the self-directed conversations around the various topics.
  • “Great way to set the tone for the following sessions….” “Some really valuable and practical insights shared.” “Great group, very interesting discussion, good way to get to know a few people.”

Fail Fest:

  • The Fail Fest was enjoyed by virtually everyone. “Brilliantly honest! Well done for having created the space and thank you to those who shared so openly.” “Awesome! Anahi definitely stole the show. What an amzing way to share learning, so memorable. Again, one to steal….” “I thought this was fun way to end the first day. All the presenters were really good and the session was well framed and facilitated by Wayan.”

Fishbowl:

  • There were mixed reactions to the “Fish Bowl”
  • “Great session and way to close the event!” “Fascinating – especially insights from Michael and Veronica.” “Not enough people volunteered to speak.” “Some speakers went on too long.”

Lunchtime Demos – 23 ratings: 52% 🙂 34% 😐 and 13% 🙁

We know that many MERL Tech participants are wary of being “sold” to. Feedback from past conferences has been that participants don’t like sales pitches disguised as breakout sessions and lightning talks. So, this year we experimented with the idea of lunchtime demo sessions. The idea was that these optional sessions would allow people with a specific interest in a tool or product to have dedicated time with the tool creators for a demo or Q&A. We hoped that doing demo sessions separately from breakout sessions would make the nature of the sessions clear. Judging from the feedback, we didn’t hit the mark. We’ll try to do better next time!

What went wrong?

  • Timing: “The schedule was too tight.” “Give more time to the lunch time demo sessions or change the format. I missed the Impact Mapper session on day 1 as there was insufficient time to eat, go for a comfort break and network. This is really disappointing. I suggest a dedicated hour in the programme on the first day to visit all the ICT provider stalls.”
  • Content: “Demo sessions were more like advertising sessions by respective companies, while nicely titled as if they were to explore topical issues. Demo sessions were all promising the world to us while we know how much challenge technology application faces in real-world. Overall so many demo sessions within a short 2-day conference compromised the agenda”
  • Framing and intent: “I don’t know that framing the lunch sessions as ‘product demos’ makes a ton of sense. Maybe force people to have real case studies or practical (hands-on) sessions, and make them regular sessions? Not sure.” “I think more care needs to be taken to frame the sessions run by the software companies with a proper declarations of interests…. Sessions led by software reps were a little less transparent in that they pitched their product, but through some other topic that people would be interested in. I think that it would be wise to make it a DOI [declaration of intent] that is scripted when people who have an interest declare their interest up front for every panel discussion at the beginning, even if they did a previous one. I think that way the rules would be a little clearer.” 

General Comments

Because we bring such a diverse group together in terms of field, experience, focus and interest, expectations are varied, and we often see conflicting suggestions. Whereas some would like more MERL content, others want more Tech content. Where as some learn a lot, others feel they have heard much of this before. Here are a few of the overall comments from Sched and the Google Form.

Who else should be at MERL Tech?

  • More donors “EU, DFID, MCC, ADB, WB, SIDA, DANIDA, MFA Finland”
  • “People working with governments in developing countries”
  • “People from the ‘field’. It was mentioned in one of the closing comments that the term ‘field’ is outdated and we are not sure what we mean anymore. Wrong. There couldn’t be a more striking difference in discussions during those two days between those with solid field experience and those lacking in it.”
  • “More Brits? There were a lot of Americans that came in from DC…”

Content that participants would like to see in the future

  • More framing: “An opening session that explains what MERL Tech is and all the different ways it’s being or can be used”
  • More specifics on how to integrate technology for specific purposes and for new purposes: “besides just allowing quicker and faster data collection and analysis”
  • More big data/data science: “Anything which combines data science, stats and qualitative research is really interesting for me and seems to be the direction a lot of organisations are going in.”
  • Less big data/data science: “The big data stuff was less relevant to me”
  • More MERL-related sessions: “I have a tech background, so I would personally have liked to have seen more MERL-related sessions.”
  • More tech-related sessions: “It was my first MERLTech, so enjoyed it. I felt that many of the presentations could have been more on-point with respect to the Tech side, rather than the MERL end (or better focus on the integration of the two).”
  • More “R” (Research): Institutional learning and research (evaluations as a subset of research).
  • More “L” Learning treated as topic of it’s own. By this I mean, the capture of tacit knowledge and good practice, use of this learning for adaptive management. Compared to my last MERL Tech, I felt this meeting better featured evaluation, or at least spoke of ‘E’ as its own independent letter. I would like to see this for ‘L.’”
  • More opportunities for smaller organisations to get best practice lessons.
  • More ethics discussions: “Consequences/whether or not we should be using personal data held by privately owned companies (like call details records from telecomms companies)” “The conceptual issues around the power dynamics and biases in data and tech ownership, collection, analysis and use and what it means for the development sector.”
  • Hands-on tutorials “for applying some of the methods people have used would be amazing, although may be beyond the remit of this conference.”
  • Coaching sessions: “one-on-ones to discuss challenges in setting up good M&E systems in smaller organisations – the questions we had, and the challenges we face did not feel like they would have been of relevance to the INGOs in the room.”

Some “Ah ha! Moments”

  • “The whole tech discussion needs to be framed around evaluation practice and theory – it seems to me that people come at this being somewhat data obsessed and driven but not starting from what we want to know and why that might be useful.”
  • “There is still quite a large gap between the data scientist and the M&E world – we really need to think more on how to bridge that gap. Despite the fact that it is recognized I do feel that much of the tech stuff was ‘ because we can’ and not because it is useful and answers to a concrete problem. On the other hand some of the tech was so complex that I also couldn’t assess whether it was really useful and what possible risks could be”
  • “I was surprised to see the scale of the impact GDPR is apparently making. Before the conference, I usually felt that most people didn’t have much of an interest in data privacy and responsible data.”
  • “That people were being honest and critical about tech!”
  • “That the tech world remains data hungry and data obsessed!”
  • “That this group is seriously confused about how tech and MERL can be used effectively as a general business practice.”
  • “This community is learning fast!”
  • “Hot topics like Big Data and Block Chain are only new tools, not silver bullets. Like RCTs a few years ago, we are starting to understand their best use and specific added value.”

Kudos

  • “A v useful conference for a growing sector. Well done!”
  • “Great opportunity for bringing together different sectors – sometimes it felt we were talking across tech, merl, and programming without much clarity of focus or common language but I suppose that shows the value of this space to discuss and work towards a common understanding and debate.”
  • “Small but meaningful to me – watching session leads attend other sessions and actively participating was great. We have such an overlap in purpose and in some cases almost no overlap in skillsets. Really felt like MERLTech was a community taking turns to learn from each other, which is pretty different from the other conferences I’ve been to, where the same people often present the same idea to a slightly different audience each year.”
  • “I loved the vibe. I’m coming to this late in my career but was made to feel welcome. I did not feel like an idiot. I found it so informative and some sessions were really inspiring. It will probably become an annual must go to event for me.”
  • “I was fully blown away. I haven’t learnt so much in a long time during a conference. The mix of types of sessions helps massively make the most of the knowledge in room, so keep up that format in the future.”
  • “I absolutely loved it. It felt so good to be with like minded people who have similar concerns and values….”

Thanks again to everyone who filled out the feedback forms and rated their sessions. This really does help us to adapt and improve. We take your ideas and opinions seriously!

If you’d like to experience MERL Tech, please join us in Johannesburg August 1-2, 2018, or Washington, DC, September 6-7, 2018!  The call for session ideas for MERL Tech DC is open through April 30th – please submit yours now!

MERL Tech London: What’s Your Organisation’s Take on Data Literacy, Privacy and Ethics?

 It first appeared here on March 26th, 2018.

ICTs and data are increasingly being used for monitoring, evaluation, research and learning (MERL). MERL Tech London was an open space for practitioners, techies, researchers and decision makers to discuss their good and not so good experiences. This blogpost is a reflection of the debates that took place during the conference.

Is data literacy still a thing?

Data literacy is “the ability to consume for knowledge, produce coherently and think critically about data.” The perception of data literacy varies depending on the stakeholder’s needs. Being data literate for an M&E team, for example, means possessing statistics skills including collecting and combining large data sets. Program team requires different level of data literacy: the competence to carefully interpret and communicate meaningful stories using processed data (or information) to reach the target audiences.

Data literacy is – and will remain – a priority in development. The current debate is no longer about whether an organisation should use data or not. It’s rather how well the organisation can use data to achieve their objectives. Yet, organisation’s efforts are often concentrated in just one part of the information value chain, data collection. Data collection in itself is not the end goal. Data has to be processed into information and knowledge for making informed decisions and actions.

This doesn’t necessary imply that the decision making is purely based on data, nor that data can replace the role of decision makers. Quite the opposite: data-informed decision making strikes balance between expertise and information. It also takes data limitations into account. Nevertheless, one can’t become a data-informed organisation without being data literate.

What’s your organisation’s data strategy?

The journey of becoming a data-informed organisation can take some time. Poor data quality, duplication efforts and underinvestment are classic obstacles requiring a systematic solution (see Tweet). The commitment from senior management team should be secured for that. Data team has to be established. Staff members need access to relevant data platforms and training. More importantly, the organisation has to embrace the cultural change towards valuing evidence and acting on positive and negative findings

Marten Schoonman@mato74
Responsible data handling workgroup: mindmapping the relevant subjects @MERLTech

Organisations seek to balance between (data) demands and priorities. Some invest hundreds of thousands dollars for setting up a data team to articulate the organisation’s needs and priorities, as well as to mobilise technical support. A 3-5 years strategic plan is created to coordinate efforts between country offices.

Others take a more modest approach. They recruit few data scientists to support MERL activities of analysing particularly large amounts of project data. The data scientist role evolves along the project growth. In both cases, leadership is the key driver for shifting the culture towards becoming a data-informed organisation.

Should an organisation use certain data because it can?

The organisation working with data usually faces challenges around privacy, legality, ethics and grey areas, such as bias and power dynamics between data collectors and their target groups. The use of biometric data in humanitarian settings is an example where all these tensions collide. Biometric data, e.g. fingerprint, iris scan, facial recognition – is powerful, yet invasive. While proven beneficial, biometric data is vulnerable to data breach and misuse, e.g. profiling and tracking. The practice raises critical questions: does the target group, e.g. refugees, have the option to refuse handling over their sensitive personal data? If so, will they still be entitled to receive aid assistance? To what extent the target group is aware how their sensitive personal data will be used and shared, including in the unforeseen circumstances?

The people’s privacy, safety and security are main priorities in any data work. The organisation should uphold the highest standards and set an example. In those countries where regulatory frameworks are lagging behind data and technology, organisations shouldn’t abuse their power. When the risk of using a certain data outweighs the benefits, or in doubt, the organisation should take a pause and ask itself some necessary questions from the perspective of its target groups. Oxfam which dismissed – following two years of internal discussions and intensive research – the idea of using biometric data in any of their project should be seen as a positive example.

To conclude, the benefits of data can only be realised when an organisation enjoys visionary leadership, sufficient capacity and upholds its principles. No doubts, this is easier being said than done; it requires time and patience. All these efforts, however, are necessary for a high-achieving organisations.

More reading:

**Save the date for MERL Tech Jozi coming up on Aug 1-2!  Session ideas are due this Friday (March 31st).

DataDay TV: MERL Tech Edition

What data superpower would you ask for? How would you describe data to your grandparents? What’s the worst use of data you’ve come across? 

These are a few of the questions that TechChange’s DataDay TV Show tackles in its latest episode.

The DataDay Team (Nick Martin, Samhir Vasdev, and Priyanka Pathak) traveled to MERL Tech DC last September to ask attendees some tough data-related questions. They came away with insightful, unusual, and occasionally funny answers….

If you’re a fan of discussing data, technology and MERL, join us at MERL Tech London on March 19th and 20th. 

Tickets are going fast, so be sure to register soon if you’d like to attend!

If you want to take your learning to the next level with a full-blown course, TechChange has a great 2018 schedule, including topics like blockchain, AI, digital health, data visualization, e-learning, and more. Check out their course catalog here.

What about you, what data superpower would you ask for?

 

Self-service data collection with the most vulnerable

This is a summary of a Lightning Talk presented by Salla Mankinen, Good Return, at MERL Tech London in 2017. 

When collecting data from the most vulnerable target groups, organizations often use methods such as guesstimating, interviewing done by enumerators, SMS, or IVR. The organization Good Return created a smart phone and tablet app that allowed vulnerable groups to interact directly with the data collection tool, without training or previous exposure to any technology.

At MERL Tech London in February 2017, Salla Mankinen shared Good Return’s experiences with using tablets for self-service check in at village training centers in Cambodia.

“Our challenge was whether we could have app-based, self-service data collection for the most vulnerable and in the most remote locations,” she said. “And could there be a journey from technology illiteracy to technology confidence” in the process?

The team created a voice and image based application that worked even for those who had little technology knowledge. It collected data from village participants such as “Why did you miss the last training session?” or “Do you have any money left this week?”

By the end of the exercise, 72% of participants felt confident with the app and 83% said they felt a lot more confident with technology in general.

Watch Salla’s presentation here or take a look at her slides here!

Register now for MERL Tech London, March 19-20, 2018!

Thoughts from MERL Tech UK

merltech_uk-2016Post by Christopher Robert, Dobility (Survey CTO)

MERL Tech UK was held in London this week. It was a small, intimate gathering by conference standards (just under 100 attendees), but jam-packed full of passion, accumulated wisdom, and practical knowledge. It’s clear that technology is playing an increasingly useful role in helping us with monitoring, evaluation, accountability, research, and learning – but it’s also clear that there’s plenty of room for improvement. As a technology provider, I walked away with both more inspiration and more clarity for the road ahead.

Some highlights:

  • I’ve often felt that conferences in the ICT4D space have been overly-focused on what’s sexy, shiny, and new over what’s more boring, practical, and able to both scale and sustain. This conference was markedly different: it exceeded even the tradition of prior MERL Tech conferences in shifting from the pathology of “pilotitus” to a more hard-nosed focus on what really works.
  • There was more talk of data responsibility, which I took as another welcome sign of maturation in the space. This idea encompasses much beyond data security and the honoring of confidentiality assurances that we at Dobility/SurveyCTO have long championed, and it amounted to a rare delight: rather than us trying to push greater ethical consideration on others, for once we felt that our peers were pushing us to raise the bar even further. My own ideas in terms of data responsibility were challenged, and I came to realize that data security is just one piece of a larger ethical puzzle.
  • There are far fewer programs and projects re-inventing the wheel in terms of technology, which is yet another welcome sign of maturation. This is helping more resources to flow into the improvement and professionalization of a small but diverse set of technology platforms. Too much donor money still seems to be spent on technologies that have effective, well-established, and sustainable options available, but it’s getting better.
  • However, it’s clear that there are still plenty of ways to re-invent the wheel, and plenty of opportunities for greater collaboration and learning in the space. Most organizations are having to go it alone in terms of procuring and managing devices, training and supporting field teams, designing and monitoring data-collection activities, organizing and managing collected data, and more. Some larger international organizations who adopted digital technologies early have built up some impressive institutional capacity – but every organization still has its gaps and challenges, later adopters don’t have that historical capacity from which to draw, and smaller organizations don’t have the same kind of centralized institutional capacity.
  • Fortunately, MERL Tech organizers and participants like Oxfam GB and World Bank DIME have not only built tremendous internal capacity, but also been extremely generous in thinking through how to share that capacity with others. They share via their blogs and participation in conferences like this, and they are always thinking about new and more effective ways to share. That’s both heartening and inspiring.

I loved the smaller, more intimate nature of MERL Tech UK, but I have quickly come to somewhat regret that it wasn’t substantially larger. My first London day post-MERL-Tech was spent visiting with some other SurveyCTO users, including a wonderfully-well-attended talk on data quality at the Zoological Society of London, a meeting with some members of Imperial College London’s Schistosomiasis Control Initiative, and a discussion about some new University of Cambridge efforts to improve data and research on rare diseases in the UK. Later today, I’ll meet with some members of the TUMIKIA project team at the London School of Hygiene and Tropical Medicine, and in retrospect I now wish that all of these others had been at MERL Tech. I’m trying to share lessons as best I can, but it’s obvious that so many other organizations could both contribute to and profit from the kinds of conversations and sharing that were happening at MERL Tech.

Personally, I’ve always been distrustful of product user conferences as narrow, ego-driven, sales-and-marketing kinds of affairs, but I’m suddenly seeing how a SurveyCTO user conference could make real (social) sense. Our users are doing such incredible things, learning so much in the process, building up so much capacity – and so many of them are also willing to share generously with others. The key is providing mechanisms for that sharing to happen. At Dobility, we’ve just kept our heads down and stayed focused on providing and supporting affordable, accessible technology, but now I’m seeing that we could play a greater role in facilitating greater progress in the space. With thousands of SurveyCTO projects now in over 130 countries, the amount of learning – and the potential social benefits to sharing more – is enormous. We’ll have to think about how we can get better and better about helping. And please comment here if you have ideas for us!

Thanks again to Oxfam GB, Comic Relief, and everybody else who made MERL Tech UK possible. It was a wonderful event.