Recap: NLP-CoP 2024 year in review meeting

Our last Natural Language Processing Community of Practice (NLP-CoP) meeting of the year happened on December 12^th.

We started off by sharing some reflections on 2024, then Working Group leads gave their year-end updates and shared ideas for future priorities for the NLP-CoP.

Access the slides and meeting recording here.

AI Trends and Challenges

The MERL Tech Initiative (MTI) has been seeing three main buckets of AI applications for MERL (Monitoring, Evaluation, Research, and Learning): back-office AI for efficiency, frontline staff focused AI to assist people to do their jobs better, and community-facing AI for direct engagement with community members

Over the past 2 years, generative AI use by MERL practitioners grew rapidly, however many practitioners still hesitate to share their experiments publicly out of fear that their pilots or test efforts are not good enough to share with others. We’ve emphasized the importance of sharing and learning out loud to enhance learning in the sector. Read this blog post on the history of MERL Tech and the role of GenAI in MERL since 2022.

Working Group Updates

Working group leads outlined their 2024 accomplishments and plans for 2025. Key themes included ethics, capacity building, and practical applications:

1. Ethics & Governance Working Group (Isabelle Amazon-Brown):
   • Explored topics such as AI procurement, child-centered AI, and using AI for qualitative analysis.
   • Plans for 2025 include prioritizing grassroots perspectives, exploring AI labor practices, and discussing survey applications.
2. Social and Behavior Change (SBC) Working Group (Isabelle Amazon-Brown)
   • Focused on advancing an SBC and AI research agenda, kickstarted at the ICT4D conference in Accra in March, hosted a successful Slack session on using AI for SBC strategy design, and participated in a roundtable with SBC and AI experts in Nairobi in December, continuing to gather insights and refine approaches.
   • For 2025 the group will aim to drill down into AI-powered chatbots for SBC, flesh out the research agenda, continue to center grassroots, majority world perspectives; and broaden efforts from the current health focus to other thematic areas like Climate.
3. AI+MERL in Africa (Linda Raftree on behalf of Mutsa Chinyamakobvu):
   • Emphasized “Made in Africa” approaches, AI literacy, and policy development.
   • We’ll be renewing efforts in 2025 thanks to a grant from Hewlett Foundation that will allow us to bring on someone to work more closely with the AI in Africa working group and develop some focused AI+MERL training for folks on the continent.
4. Sandbox Working Group (Zach Tilton):
   • Hosted webinars and hackathons to foster experimentation with tools like chatbots and retrieval-augmented generation.
   • Plans include “bite-sized collaborative projects” like development of a “Consumer Reports” on Gen AI tools.
   • Zach suggested a “project idea parking lot” for collaborative brainstorming, which received strong support from meeting participants in the chat.
5. Humanitarian AI (Quito Tsui):
   • This is a newer group focused on co-creating a research agenda for AI in humanitarian contexts.
   • Plans for 2025 include collaborations with the CDAC Network, presenting at RightsCon 2025 and other humanitarian conferences, and inviting humanitarian organizations to share what they are working on as well as their concerns about AI in the humanitarian space, including how AI intersects with localization efforts.
6. Philanthropy and AI (Linda Raftree):
   • Explored how philanthropy organizations use AI for efficiency and drawing out insights from large repositories of text and from grantee reports. There’s a particularly strong interest in climate-related initiatives.
   • For 2025 we are hoping to continue exploring how foundations are using GenAI, potentially host a face-to-face event for foundations to share what they are doing with GenAI.
   • We’ll continue to seek foundation funding for the CoP to allow working groups to deepen and expand their work. We were able to secure funding from Packard Foundation to hire a Community Manager for 2025 and from Hewlett (as noted above) to support the AI+MERL in Africa work
7. Climate MERL + AI Working Group (Cathy Richards)
   • This group will focus on using Ai to advance Climate Action in the MERL Space. It will also seek to address the environmental impacts of AI tools, including energy consumption and resource demands and encourage exploration of alternatives to reduce AI’s environmental footprint.
   • In 2025 it will aim to developing guidance and tools for climate-conscious AI practices, facilitate discussions on how AI can support climate-focused monitoring, evaluation, and research; and create connections among practitioners interested in the intersection of AI and climate
8. Gender, AI and MERL (Linda Raftree on behalf of Allison Sambo)
   • This working group will kick off on January 29^th (register now – more details coming soon!). It was launched at the recent AWID forum in Bangkok. It will explore ethical use of AI in MERL with a gender lens, feminist approaches, participatory methods, and empowerment strategies within AI-enabled evaluations. It will also look at gender bias in AI products and promoting equitable tech
   • Upcoming plans include workshops on adapting and adopting AI tools for gender-responsive programming; engaging grassroots organizations and feminist AI initiatives to shape a participatory agenda; highlighting mitigation strategies for gender inequities in AI use and development.

Future Plans

2025 priorities for the CoP can be found in the slide deck, but generally we are working to:

• Improve community member experiences (we’re so excited that our Community Manager will join in January!)
• Revamp member charter, onboarding, etc.
• Pursue funding for specific working groups and their projects
• Organize a MERL Tech AI Conference in DC in 2025 and in an African country in 2026
• Develop a vendor vetting framework and ‘Consumer Reports’ initiative for AI+MERL applications
• More peer-peer learning opportunities (what are people doing and can they share?)
• Develop a variety of more in-depth training offerings

Meeting Participants’ Suggestions for 2025

Participants in the meeting also gave their suggestions including:

• More peer-to-peer learning: Participants discussed creating mentorship programs and more structured frameworks for collaboration. One person emphasized that, “Even if only 10% of our 1,100 members have learning to share, that’s a lot of people and learning.”
• More in-person connections: Early results from our year-end survey showed that people’s favorite CoP activities where face-to-face and people want more of this!
• Sharing failures: There was strong support for platforms to share “unsuccessful use cases.” As one person commented: “Failure is part of the learning process, and it would be nice to learn from missteps in a deliberative way.”
• Deliberative tech: Another area suggested was more exploration of the potential of AI for participatory governance, noting other CoPs exploring similar avenues.

Participant Recognition and Closing

We closed out by celebrating participants (and AI tools) that were nominated by their peers for outstanding contributions in 2024.

• Nominations for impactful individuals included Jewlya Lynn for her thoughtful AI evaluation work and Christopher Robert for his open-source AI experimentation.
• NotebookLM got a shout out as “mind-blowing” in the way it can create alternative formats for dense reports, democratizing access to evaluation insights. However, there was strong critique of its very American, California accents, underscoring localization challenges with AI.

Congratulations to the nominees and we can’t wait to see you all in 2025!

📌 Interested in taking part in similar discussions in the future? Make sure to join the NLP CoP, a community of over 1100 development and humanitarian practitioners working at the intersection of AI, digital development, and MERL.