Menu Close

Wrapping up: A year in the NLP-CoP

by Linda Raftree

This was a foundational year for the MERL Tech Sector, with a number of advances in the area of Natural Language Processing. The excitement around this emerging AI technology lead us to kick off a Natural Language Processing Community of Practice (NLP-CoP) in January with the aim of exploring the potential and challenges NLP brings to MERL.

Here’s a glimpse into what the NLP-CoP achieved in its first year, guided by the five goals we set out early in the year.

Connect and build a diverse network

The NLP-CoP and network now has 380 members. Sixty percent of members consider themselves beginners in NLP and 30% intermediate. Most (76%) are intermediate or advanced MERL practitioners. Approximately 39% are from the USA or Canada, 30% from Europe, 21% from African countries, and the remaining 10% are from Asian Pacific, Latin American and Middle Eastern/North African countries. Slightly over half identify as female.

We initiated five working groups to enhance understanding and application of NLP and GenAI in MERL in specific areas:

  1. Customizing Large Language Models for MERL: This group explores how to fine-tune LLM models and adapt LLM tools for MERL. It develops technical resources on LLMs and NLP for MERL applications.
  2. NLP in African Contexts: This group focuses on NLP tools for low-resource and multilingual contexts, the inclusion of African languages, capacity development for African MERL practitioners, and relevant use cases in African countries.
  3. Participation and Accountability in NLP for MERL: This group examines frameworks and processes to ensure meaningful participation and accountability in NLP design and application for MERL.
  4. NLP and MERL for Social and Behavior Change Communications (SBCC): This group explores ways to use NLP and GenAI to implement, understand, and measure the impact of SBCC.
  5. Ethics and Privacy Considerations for NLP + MERL: This group addresses the ethical aspects of using NLP in MERL, including privacy protection and harm prevention and wider impacts of NLP on people, communities, and societies. It compiles and develops guidance for the sector in these areas.

Democratize understanding of NLP and Generative AI

We hosted 9 NLP-CoP meetings and 3 in-person Technology Salons in 2023 (read the re-caps below):

Identify, explore, develop and/or test responsible NLP tools, applications, code or models for MERL

We hosted lively discussion on Slack about new and emerging AI tools and technologies and learned how different CoP members are experimenting with and testing them.

Co-develop guidance, and support materials as public goods

We shared good practices and guidance documents and orientation on Slack and via our monthly newsletter. We summarized all our meetings and shared emerging good practices via blog posts as well.

Influence good practice in the wider sector

We led sessions at multiple conferences, wrote articles and opinion pieces and engaged with various networks and partners.

  • In October, NLP-CoP members presented at 3 different sessions at the American Evaluation Association (AEA) conference (including the Presidential Strand and a session on how evaluators can use NLP). We hosted a lunchtime meet up in collaboration with the AEA’s Integrating Technology into Evaluation Technical Interest Group (ITE-TIG). Read more about What’s next for Emerging AI in Evaluation.
  • In October, members presented on opportunities and challenges with NLP at the UK Evaluation Society conference. This talk shared key points from a journal article that several NLP-CoP members authored for the New Directions for Evaluation journal: Large language model applications for evaluation: Opportunities and ethical implications
  • We joined a session at NetHope’s annual summit to discuss how INGOs are addressing policy and guidance on the use of NLP and Generative AI and spoke at NetHope’s Data Innovations Working Group and NetHope’s AI Working Group. (NetHope is a network of technology companies and large INGOs). 
  • Linda did a keynote talk on A Just Transition: What does it mean for AI and Evaluation? at the European Evaluation Society’s Autumn Virtual Event. She urged the wider evaluation community to think beyond emerging AI as a tool for evaluation and to consider it from a systems level with more attention to politics, economics, job markets, and societal implications.

What’s next for the NLP-CoP?

We’ll be back in 2024 with more detailed plans on where we’re headed once we complete in-depth consultations with CoP members and Working Group leads. Some key areas that have been flagged already for 2024 include:

  1. A Name Change? As the terminology in our field shifts from “NLP” to “Generative AI” (GenAI), should we change our name to (GenAI-CoP) and slightly expand the focus of the CoP? (Members – we’ll ask you to vote on this in a survey coming your way soon!)
  2. Seeking Funding! We did a lot this year with limited resources. We’re seeking core funding to sustain the CoP. We’re also looking for funding for working group activities such as research, development and testing of new models and tools, and the creation of guidance, tool kits, and public goods focusing on NLP, GenAI, and MERL.
  3. Expanding: We aim to grow CoP membership and strengthen partnerships in majority world countries. Based on our members’ needs and sector demands, we’re considering new working groups, such as one for young and emerging evaluators, a funders’ working group, and one dedicated to NLP/GenAI and humanitarian MERL.
  4. Developing Public Goods: Possible initiatives include creating a comprehensive list of NLP and GenAI tools for MERL, developing a set of tipsheets for creative participatory MERL methods using GenAI, and compiling guidance materials on the responsible use of NLP and GenAI in MERL (covering aspects like internal organizational guidance, emerging AI approaches for MERL, bias assessment, and ethical considerations).
  5. A GenAI+MERL Sandbox: We want to establish a sandbox environment for testing and experimenting with models and applications that integrate NLP and GenAI, specifically tailored for MERL use cases and with close attention to quality of the experience for end users, accuracy and reliability, privacy, and responsible data management and use.
  6. Active Conference Participation: We’ll attend various conferences and events. At ICT4D Accra we’re coordinating the Data Innovations for Impact stream and planning workshops on GenAI and MERL and GenAI and SBCC. We’ll be at the African Evaluation Association (AfrEA) Conference in March. We hope to be involved with the South African M&E Association (SAMEA) Conference, the American Evaluation Association (AEA) Conference and the European Evaluation Society (EES) Conference in the latter part of the year as well,

Once we have results back from our year-end survey and wider consultations, we’ll share our plans more widely!

A huge thank you to everyone who participated in the CoP this year!

We appreciate the ideas, energy and inputs from NLP-CoP members, the NLP-CoP Steering Committee (Cari-Beth Head, Grace Higdon, Jonas Norén, Kerry Bruce, Stephanie Coker, Mark Gachara Iruru, Matthew McConnachie, Megan Colnar, Paul Jasper, Susanna Morrison-Metois, and Teresa Perosa); Working Group Leads (Mutsa Chinyamakobvu, Zach Tilton, Chioma Agwuegbo, Jacqueline Hart, Nicola Harford, Louis Davison), and the NLP-CoP Secretariat (Cathy Richards and Talitha Hlaka). Thanks also to The Mastercard Foundation for providing core support for our first year.

Note: Illustrations in this post were created using ChatGPT4/DALL-E.

Leave a Reply

Your email address will not be published. Required fields are marked *

You might also like