Tag Archives: Session

Feedback Report from MERL Tech London 2018

MERL Tech London happened on March 19-20, 2018. Here are some highlights from session level feedback and the post-conference survey on the overall MERL Tech London experience.

If you attended MERL Tech London, please get in touch if you have any further questions about the feedback report or if you would like us to send you detailed (anonymized) feedback about a session you led. Please also be sure to send us your blog posts & session summaries so that we can post them on MERL Tech News!

Background on the data

  • 54 participants (~27%) filled out the post-conference survey via Google Forms.
  • 59 (~30%) rated and/or commented on individual sessions via the Sched app. Participants chose from three ’emoji’ options: a happy face 🙂 , a neutral face 😐 , and a sad face 🙁 . Participants could also leave their comments on individual sessions.
  • We received 616 session ratings/comments via Sched. Some people rated the majority of sessions they attended; others only rated 1-2 sessions.
  • Some reported that they did not feel comfortable rating sessions in the Sched app because they were unclear about whether session leads and the public could see the rating. In future, we will let participants know that only Sched administrators can see the identity of commenters and the ratings given to sessions.
  • We do not know if there is an overlap between those who filled out Sched and those that fed back via Google Forms because the Google Forms survey is anonymous.

Overall feedback

Here’s how survey participants rated the overall experience:

Breakout sessions– 137 ratings: 69% 🙂 30% 😐 and 13% 🙁

Responses were fairly consistent across both Sched ratings and Google Forms (the form asked people to identify their favorite session). Big data and data science sessions stand out with the highest number of favorable ratings and comments. General Data Protection Regulation (GDPR) and responsible data made an important showing, as did the session on participatory video in evaluation.

Sessions with favorable comments tended to include or combine elements of:

  • an engaging format
  • good planning and facilitation
  • strong levels of expertise
  • clear and understandable language and examples
  • and strategic use of case studies to point at a bigger picture that is replicable to other situations.

Below are the breakout sessions that received the most favorable ratings and comments overall. (Plenty of other sessions were also rated well but did not make the “top-top.”)

Session title

Comments

Be it resolved: In the near future, conventional evaluation and big data will be successfully integrated Brilliant session! Loved the format! Fantastic to have such experts taking part. Really appreciated the facilitation and that there was a time at the end for more open questions/discussion.
Innovative Use of Theory-Based and Data Science Evaluation Approaches Most interesting talk of the day (maybe more for the dedicated evaluation practitioners), very practical and easy to understand and I’m really looking forward to hearing more about the results as the work progresses!
Unpacking How Change Happened (or Didn’t): Participatory Video and Most Significant Change Right amout of explanation and using case studies to illustrate points and respond to questions rather than just stand alone case studies.
GDPR – What Is It and What Do We Do About It? GDPR and what we do about it – Great presentation starting off with some historical background, explaining with clarity how this new legislation is a rights-based approach and concluding on how for Oxfam this is not a compliance project but a modification in data culture. Amazing, innovative and the speaker knew his area very well.
The Best of Both Worlds? Combining Data Science and Traditional M&E to Understand Impact I learned so much from this session and was completely inspired by the presenter and the content. Clear – well paced – honest – open – collaborative and packed with really good insight. Amazing.
Big Data, Adaptive Management, and the Future of MERL Quite a mixed bag of presenters, with focus on different pieces on the overall topic. Speakers from Novometrics was particularly engaging and stimulated some good discussion.
Blockchain: Getting Past the Hype and Considering its Value for MERL Great group with good facilitation. Open ended question left lots of room for discussion without bias towards particular outcome. Learned lots and not just about blockchain.
LEAP, and How to Bring Data to Life in Your Organization Really great session, highly interactive, rich in concepts clearly and convincingly explained. No questions were left unanswered. Very insightful suggestions shared between the presenters/facilitators and the audience. Should be on the agenda of next MERL Tech Conference as well.
Who Watches the Watchers? Good Practice for Ethical MERL(Tech)

 

I came out with some really helpful material. Collaborative session and good workshop participants willing to share and mind map. Perhaps the lead facilitator could have been a bit more contextual. Not always clear. But, our table session was really helpful and output useful.
The GDPR is coming! Now what?! Practical Steps to Help You Get Ready Good session. Appreciated the handouts….

What could session leads improve on?

We also had a few sessions that were ranked closer to 😐 (somewhere around a 6 or 6.5 on a scale of 1-10). Why did participants rate some sessions lower?

  • “Felt like a product pitch”
  • “Title was misleading”
  • Participatory activity was unclear
  • Poor time management
  • “Case studies did not expand to learning for the sector – too much ‘this is what we did’ and not enough ‘this is what it means.””
  • Poor facilitation/moderation
  • “Too unstructured, meandering”
  • Low energy
  • “Only a chat among panelists, very little time for Q&A. No space to engage”

Additionally, some sessions had participants with very diverse levels of expertise and varied backgrounds and expectations, which seemed to affect session ratings.

Lightning Talks– 182 ratings: 74% 🙂 22% 😐 4% 🙁

Lightning talks consistently get the highest ratings at MERL Tech, and this year was no exception. As one participant said, “my favorite sessions were the lightning talks because they gave a really quick overview of really concrete uses of technology in M&E work. This really helped in getting an overview of the type of activities and projects that were going on.”  Participants rated almost all the lightning talks positively.

Plenary sessions– 192 ratings: 77% 🙂 21% 😐 and 2% 🙁

Here we include:  the welcome, discussions on MERL Tech priorities on Day 1, opening talks on both days, summary on Day 1, panel with donors, the closing ‘fishbowl’, and the Fail Fest.

Opening Talks:

  • People appreciated André’ Clarke’s stage setting talk on Day 1. “Clear, accessible and thoughtful.” “Nice deck!”
  • Anahi Ayala Iacucci’s opening talk on Day 2 was a hit: “Great keynote! Anahi is very engaging but also her content was really rich. Useful that she used a lot of examples and provided a lot of history.” And “Like Anahi says ‘The question to ask is what does technology do _to_ development, rather than what can technology do _for_ development.'”

Deep Dive into Priorities for the Sector:

  • Most respondents enjoyed the self-directed conversations around the various topics.
  • “Great way to set the tone for the following sessions….” “Some really valuable and practical insights shared.” “Great group, very interesting discussion, good way to get to know a few people.”

Fail Fest:

  • The Fail Fest was enjoyed by virtually everyone. “Brilliantly honest! Well done for having created the space and thank you to those who shared so openly.” “Awesome! Anahi definitely stole the show. What an amzing way to share learning, so memorable. Again, one to steal….” “I thought this was fun way to end the first day. All the presenters were really good and the session was well framed and facilitated by Wayan.”

Fishbowl:

  • There were mixed reactions to the “Fish Bowl”
  • “Great session and way to close the event!” “Fascinating – especially insights from Michael and Veronica.” “Not enough people volunteered to speak.” “Some speakers went on too long.”

Lunchtime Demos – 23 ratings: 52% 🙂 34% 😐 and 13% 🙁

We know that many MERL Tech participants are wary of being “sold” to. Feedback from past conferences has been that participants don’t like sales pitches disguised as breakout sessions and lightning talks. So, this year we experimented with the idea of lunchtime demo sessions. The idea was that these optional sessions would allow people with a specific interest in a tool or product to have dedicated time with the tool creators for a demo or Q&A. We hoped that doing demo sessions separately from breakout sessions would make the nature of the sessions clear. Judging from the feedback, we didn’t hit the mark. We’ll try to do better next time!

What went wrong?

  • Timing: “The schedule was too tight.” “Give more time to the lunch time demo sessions or change the format. I missed the Impact Mapper session on day 1 as there was insufficient time to eat, go for a comfort break and network. This is really disappointing. I suggest a dedicated hour in the programme on the first day to visit all the ICT provider stalls.”
  • Content: “Demo sessions were more like advertising sessions by respective companies, while nicely titled as if they were to explore topical issues. Demo sessions were all promising the world to us while we know how much challenge technology application faces in real-world. Overall so many demo sessions within a short 2-day conference compromised the agenda”
  • Framing and intent: “I don’t know that framing the lunch sessions as ‘product demos’ makes a ton of sense. Maybe force people to have real case studies or practical (hands-on) sessions, and make them regular sessions? Not sure.” “I think more care needs to be taken to frame the sessions run by the software companies with a proper declarations of interests…. Sessions led by software reps were a little less transparent in that they pitched their product, but through some other topic that people would be interested in. I think that it would be wise to make it a DOI [declaration of intent] that is scripted when people who have an interest declare their interest up front for every panel discussion at the beginning, even if they did a previous one. I think that way the rules would be a little clearer.” 

General Comments

Because we bring such a diverse group together in terms of field, experience, focus and interest, expectations are varied, and we often see conflicting suggestions. Whereas some would like more MERL content, others want more Tech content. Where as some learn a lot, others feel they have heard much of this before. Here are a few of the overall comments from Sched and the Google Form.

Who else should be at MERL Tech?

  • More donors “EU, DFID, MCC, ADB, WB, SIDA, DANIDA, MFA Finland”
  • “People working with governments in developing countries”
  • “People from the ‘field’. It was mentioned in one of the closing comments that the term ‘field’ is outdated and we are not sure what we mean anymore. Wrong. There couldn’t be a more striking difference in discussions during those two days between those with solid field experience and those lacking in it.”
  • “More Brits? There were a lot of Americans that came in from DC…”

Content that participants would like to see in the future

  • More framing: “An opening session that explains what MERL Tech is and all the different ways it’s being or can be used”
  • More specifics on how to integrate technology for specific purposes and for new purposes: “besides just allowing quicker and faster data collection and analysis”
  • More big data/data science: “Anything which combines data science, stats and qualitative research is really interesting for me and seems to be the direction a lot of organisations are going in.”
  • Less big data/data science: “The big data stuff was less relevant to me”
  • More MERL-related sessions: “I have a tech background, so I would personally have liked to have seen more MERL-related sessions.”
  • More tech-related sessions: “It was my first MERLTech, so enjoyed it. I felt that many of the presentations could have been more on-point with respect to the Tech side, rather than the MERL end (or better focus on the integration of the two).”
  • More “R” (Research): Institutional learning and research (evaluations as a subset of research).
  • More “L” Learning treated as topic of it’s own. By this I mean, the capture of tacit knowledge and good practice, use of this learning for adaptive management. Compared to my last MERL Tech, I felt this meeting better featured evaluation, or at least spoke of ‘E’ as its own independent letter. I would like to see this for ‘L.’”
  • More opportunities for smaller organisations to get best practice lessons.
  • More ethics discussions: “Consequences/whether or not we should be using personal data held by privately owned companies (like call details records from telecomms companies)” “The conceptual issues around the power dynamics and biases in data and tech ownership, collection, analysis and use and what it means for the development sector.”
  • Hands-on tutorials “for applying some of the methods people have used would be amazing, although may be beyond the remit of this conference.”
  • Coaching sessions: “one-on-ones to discuss challenges in setting up good M&E systems in smaller organisations – the questions we had, and the challenges we face did not feel like they would have been of relevance to the INGOs in the room.”

Some “Ah ha! Moments”

  • “The whole tech discussion needs to be framed around evaluation practice and theory – it seems to me that people come at this being somewhat data obsessed and driven but not starting from what we want to know and why that might be useful.”
  • “There is still quite a large gap between the data scientist and the M&E world – we really need to think more on how to bridge that gap. Despite the fact that it is recognized I do feel that much of the tech stuff was ‘ because we can’ and not because it is useful and answers to a concrete problem. On the other hand some of the tech was so complex that I also couldn’t assess whether it was really useful and what possible risks could be”
  • “I was surprised to see the scale of the impact GDPR is apparently making. Before the conference, I usually felt that most people didn’t have much of an interest in data privacy and responsible data.”
  • “That people were being honest and critical about tech!”
  • “That the tech world remains data hungry and data obsessed!”
  • “That this group is seriously confused about how tech and MERL can be used effectively as a general business practice.”
  • “This community is learning fast!”
  • “Hot topics like Big Data and Block Chain are only new tools, not silver bullets. Like RCTs a few years ago, we are starting to understand their best use and specific added value.”

Kudos

  • “A v useful conference for a growing sector. Well done!”
  • “Great opportunity for bringing together different sectors – sometimes it felt we were talking across tech, merl, and programming without much clarity of focus or common language but I suppose that shows the value of this space to discuss and work towards a common understanding and debate.”
  • “Small but meaningful to me – watching session leads attend other sessions and actively participating was great. We have such an overlap in purpose and in some cases almost no overlap in skillsets. Really felt like MERLTech was a community taking turns to learn from each other, which is pretty different from the other conferences I’ve been to, where the same people often present the same idea to a slightly different audience each year.”
  • “I loved the vibe. I’m coming to this late in my career but was made to feel welcome. I did not feel like an idiot. I found it so informative and some sessions were really inspiring. It will probably become an annual must go to event for me.”
  • “I was fully blown away. I haven’t learnt so much in a long time during a conference. The mix of types of sessions helps massively make the most of the knowledge in room, so keep up that format in the future.”
  • “I absolutely loved it. It felt so good to be with like minded people who have similar concerns and values….”

Thanks again to everyone who filled out the feedback forms and rated their sessions. This really does help us to adapt and improve. We take your ideas and opinions seriously!

If you’d like to experience MERL Tech, please join us in Johannesburg August 1-2, 2018, or Washington, DC, September 6-7, 2018!  The call for session ideas for MERL Tech DC is open through April 30th – please submit yours now!

Present or lead a session at MERL Tech DC!

Please sign up to present, register to attend, or reserve a demo table for MERL Tech DC 2018 on September 6-7, 2018 at FHI 360 in Washington, DC.

We will engage 300 practitioners from across the development ecosystem for a two-day conference seeking to turn the theories of MERL technology into effective practice that delivers real insight and learning in our sector.

MERL Tech DC 2018, September 6-7, 2018

Digital data and new media and information technologies are changing monitoring, evaluation, research and learning (MERL). The past five years have seen technology-enabled MERL growing by leaps and bounds. We’re also seeing greater awareness and concern for digital data privacy and security coming into our work.

The field is in constant flux with emerging methods, tools and approaches, such as:

  • Adaptive management and developmental evaluation
  • Faster, higher quality data collection
  • Remote data gathering through sensors and self-reporting by mobile
  • Big data, data science, and social media analytics
  • Story-triggered methodologies

Alongside these new initiatives, we are seeing increasing documentation and assessment of technology-enabled MERL initiatives. Good practice guidelines are emerging and agency-level efforts are making new initiatives easier to start, build on and improve.

The swarm of ethical questions related to these new methods and approaches has spurred greater attention to areas such as responsible data practice and the development of policies, guidelines and minimum ethical standards for digital data.

Championing the above is a growing and diversifying community of MERL practitioners, assembling from a variety of fields; hailing from a range of starting points; espousing different core frameworks and methodological approaches; and representing innovative field implementers, independent evaluators, and those at HQ that drive and promote institutional policy and practice.

Please sign up to present, register to attend, or reserve a demo table for MERL Tech DC to experience 2 days of in-depth sharing and exploration of what’s been happening across this cross-disciplinary field, what we’ve been learning, complex barriers that still need resolving, and debate around the possibilities and the challenges that our field needs to address as we move ahead.

Submit Your Session Ideas Now

Like previous conferences, MERL Tech DC will be a highly participatory, community-driven event and we’re actively seeking practitioners in monitoring, evaluation, research, learning, data science and technology to facilitate every session.

Please submit your session ideas now. We are looking for a range of topics, including:

  • Experiences and learning at the intersection of MERL and tech
  • Ethics, inclusion, safeguarding, and data privacy
  • Data (big data, data science, data analysis)
  • Evaluation of ICT-enabled efforts
  • The future of MERL
  • Tech-enabled MERL Failures

Visit the session submission page for more detail on each of these areas.

Submission Deadline: Monday, April 30, 2018 (at midnight EST)

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early June and, if selected, you will be asked to submit the final session title, summary and outline by June 30.

Register Now

Please sign up to present or register to attend MERL Tech DC 2018 to examine these trends with an exciting mix of educational keynotes, lightning talks, and group breakouts, including an evening reception and Fail Fest to foster needed networking across sectors and an exploration of how we can learn from our mistakes.

We are charging a modest fee to better allocate seats and we expect to sell out quickly again this year, so buy your tickets or demo tables now. Event proceeds will be used to cover event costs and to offer travel stipends for select participants implementing MERL Tech activities in developing countries.

You can also submit session ideas for MERL Tech Jozi, coming up on August 1-2, 2018! Those are due on March 31st, 2018!

Please Submit Session Ideas for MERL Tech Jozi

We’re thrilled to announce that we’re organizing MERL TEch Jozi for August of 2018!

Please submit your session ideas or reserve your demo table now, to explore what’s happening with innovation, digital data, and new technologies across the monitoring, evaluation, research, and learning (MERL) fields.

MERL Tech Jozi will be in Johannesburg, South Africa, August 1-2, 2018!

At MERL Tech Jozi, we’ll build on earlier MERL Tech conferences in DC and London, engaging 100 practitioners from across the development and technology ecosystems for a two-day conference seeking to turn theories of MERL technology into effective practices that deliver real insight and learning in our sector.

MERL Tech is a lively, interactive, community-driven conference.  We’re actively seeking a diverse set of practitioners in monitoring, evaluation, research, learning, program implementation, management, data science, and technology to lead every session.

Submit your session ideas now.

We’re looking for sessions that focus on:

  • Discussions around good practice and evidence-based review
  • Innovative MERL approaches that incorporate technology
  • Future-focused thought provoking ideas and examples
  • Conversations about ethics, inclusion, and responsible policy and practice in MERL Tech
  • Exploration of complex MERL Tech challenges and emerging good practice
  • Workshop sessions with practical, hands-on exercises and approaches
  • Lightning Talks to showcase new ideas or to share focused results and learning
Submission Deadline: Saturday, March 31, 2018.

Session submissions are reviewed and selected by our steering committee. Presenters and session leads will have priority access to MERL Tech tickets. We will notify you whether your session idea was selected in late April and if selected, you will be asked to submit the final session title, summary and detailed session outline by June 1st, 2018

If you’d prefer to showcase your technology tool or platform to MERL Tech participants, you can reserve your demo table here.

MERL Tech is dedicated to creating a safe, inclusive, welcoming and harassment-free experience for everyone through our Code of Conduct.

MERL Tech Jozi is organized by Kurante and supported by the following sponsors. Contact Linda Raftree if you’d like to be a sponsor of MERL Tech Jozi too.

 

 

 

Submit your session ideas for MERL Tech London by Nov 10th!

MERL Tech London

Please submit a session idea, register to attend, or reserve a demo table for MERL Tech London, on March 19-20, 2018, for in-depth sharing and exploration of what’s happening across the multidisciplinary monitoring, evaluation, research and learning field.

Building on MERL Tech London 2017, we will engage 200 practitioners from across the development and technology ecosystems for a two-day conference seeking to turn the theories of MERL technology into effective practice that delivers real insight and learning in our sector.

MERL Tech London 2018

Digital data and new media and information technologies are changing MERL practices. The past five years have seen technology-enabled MERL growing by leaps and bounds, including:

  • Adaptive management and ‘developmental evaluation’
  • Faster, higher quality data collection.
  • Remote data gathering through sensors and self-reporting by mobile.
  • Big Data and social media analytics
  • Story-triggered methodologies

Alongside these new initiatives, we are seeing increasing documentation and assessment of technology-enabled MERL initiatives. Good practice guidelines and new frameworks are emerging and agency-level efforts are making new initiatives easier to start, build on and improve.

The swarm of ethical questions related to these new methods and approaches has spurred greater attention to areas such as responsible data practice and the development of policies, guidelines and minimum ethical frameworks and standards for digital data.

Please submit a session idea, register to attend, or reserve a demo table for MERL Tech London to discuss all this and more! You’ll have the chance to meet, learn from, debate with 150-200 of your MERL Tech peers and to see live demos of new tools and approaches to MERL.

Submit Your Session Ideas Now!

Like previous conferences, MERL Tech London will be a highly participatory, community-driven event and we’re actively seeking practitioners in monitoring, evaluation, research, learning, data science and technology to facilitate every session.

Please submit your session ideas now. We are particularly interested in:

  • Case studies: Sharing end-to-end experiences/learning from a MERL Tech process
  • MERL Tech 101: How-to use a MERL Tech tool or approach
  • Methods & Frameworks: Sharing/developing/discussing methods and frameworks for MERL Tech
  • Data: Big, large, small, quant, qual, real-time, online-offline, approaches, quality, etc.
  • Innovations: Brand new, untested technologies or approaches and their application to MERL(Tech)
  • Debates: Lively discussions, big picture conundrums, thorny questions, contentious topics related to MERL Tech
  • Management: People, organizations, partners, capacity strengthening, adaptive management, change processes related to MERL Tech
  • Evaluating MERL Tech: comparisons or learnings about MERL Tech tools/approaches and technology in development processes
  • Failures: What hasn’t worked and why, and what can be learned from this?
  • Demo Tables: to share MERL Tech approaches, tools, and technologies
  • Other topics we may have missed!

Session Submission Deadline: Friday, November 10, 2017.

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early December and, if selected, you will be asked to submit an updated and final session title, summary and outline by Friday, January 19th, 2018.

Register Now!

Please register to attend, or reserve a demo table for MERL Tech London 2018 to examine these trends with an exciting mix of educational keynotes, lightning talks, and group breakouts, including an evening Fail Festival reception to foster needed networking across sectors.

We are charging a modest fee to better allocate seats and we expect to sell out quickly again this year, so buy your tickets or demo tables now. Event proceeds will be used to cover event costs and to offer travel stipends for select participants implementing MERL Tech activities in developing countries.

MERL Tech DC: Session ideas due by May 12th!

Don’t forget to sign up to present, register to attend, or reserve a demo table for MERL Tech DC on September 7-8, 2017 at FHI 360 in Washington, DC.

Submit Your Session Ideas by Friday, May 12th!

Like previous conferences, MERL Tech DC will be a highly participatory, community-driven event and we’re actively seeking practitioners in monitoring, evaluation, research, learning, data science and technology to facilitate every session.

Please submit your session ideas now. We are particularly interested in:

  • Discussions around good practice and evidence-based review
  • Workshops with practical, hands-on exercises
  • Discussion and sharing on how to address methodological aspects such as rigor, bias, and construct validity in MERL Tech approaches
  • Future-focused thought provoking ideas and examples
  • Conversations about ethics, inclusion and responsible policy and practice in MERL Tech

Session leads receive priority for the available seats at MERL Tech and a discounted registration fee. You will hear back from us in early June and, if selected, you will be asked to submit the final session title, summary and outline by June 30.

If you have questions or are unsure about a submission idea, please get in touch with Linda Raftree.

Submit your ideas here!