Exploring NLP and MERL: Customization and Collaboration

Natural Language Processing (NLP) is revolutionizing the Monitoring, Evaluation, Research, and Learning (MERL) landscape. The NLP Community of Practice (NLP-CoP) is exploring how advances in NLP can be applied to data collection, analysis, use and other MERL tasks.

At our June 21 meeting, CoP members spoke about key advances in NLP, providing examples and use cases and sharing emerging learning. We also launched five Working Groups (WGs) within the NLP-CoP, where members can dive deeper into specific topics. 

Experimentions with custom content

Three members of the CoP shared some different ways that they are applying NLP in their MERL practice.

• Custom Knowledge Bases: Abheejit presented on the topic of custom knowledge bases and introduced Mobius, an application that combines the power of OpenAI models with carefully curated data from companies. Mobius stores text embeddings to efficiently retrieve and match user queries with relevant information. Abheejit’s demonstrated this via a Q&A WhatsApp chatbot that provided text and audio responses from a custom knowledge base. 

• Retrieval Augmentation: Christopher Robert focused on retrieval augmentation and the utilization of Large Language Models (LLMs) as back-office assistants. Retrieval augmentation involves integrating additional context into queries in strategic chunks in order to accommodate the limited context windows of LLMs. Chris also highlighted the iterative process of refining system prompts using GPT 3.5, which has proven more effective than traditional approaches. 

• Atlas.ti: Cari-Beth Head showcased the integration of AI capabilities into Atlas.ti, a tool widely used for qualitative data analysis. By leveraging AI, Atlas.ti is better enabling features like topic modeling, sentiment analysis, automated focus group coding, and survey coding. Beth demonstrated the ease of importing social network commentary, analyzing text using the platform’s features, and utilizing AI to code content. 

New NLP-CoP Working Groups are kicking off!

We’ve established five new Working Groups (WGs) that offer members opportunities for collaboration and innovation. At the meeting, WG leads shared draft visions and goals and invited CoP members to join and to help finalize group objectives and activities. We are seeking funding for WGs to implement joint projects, pilot test potential applications, and develop useful materials and/or guidance for the wider MERL sector.

1. Customizing Large Language Models for MERL (Leads: Chris + Abheejit): This WG aims to explore the customization of Large Language Models for MERL purposes. It will focus on ways that existing organizational data can be better harnessed by NLP applications. The WG welcomes individuals interested in fine-tuning models and developing open-source tools that can be utilized by nonprofit organizations.

2. NLPs in African Contexts (Lead: Mutsa Chinyamakobvu): This WG will be focusing on NLP tools and applications that are suitable for low-resource contexts. It will also address themes such as inclusion/exclusion of African languages, capacity strengthening needs for African MERL practitioners and young evaluators, and specific use cases relevant to MERL professionals working in African countries.

3. Participation and Accountability in NLP for MERL (Leads: Megan Colnar + Jacqueline Hart): With a focus on participation and accountability, this group aims to explore the responsible use of NLP in the MERL domain. What does meaningful participation by civil society organizations and communities look like when we apply NLP to MERL? How is accountability addressed — accountabilty to whom and for what? The group aims to create frameworks that guide the future use of these tools.

4. NLP and MERL for Social and Behavior Change Communications (SBCC) (Leads: Nicola Harford + Louis Davison): This WG aims to leverage AI-powered text analytics to understand and measure the impacts of Social and Behavior Change Communications. It will explore the use of generative AI chatbots for storytelling and consider privacy, ethics, trust, safety, and user perspectives.

5. Ethics and Privacy Considerations for NLP + MERL: (Lead: Linda Raftree): This working group will explore ethical aspects of NLP as applied to MERL, including a) practical ethics — how to protect privacy and prevent harm to individuals and groups when NLP is used for MERL with or about them — and b) the role of MERL professionals in assessing wider, systems aspects such as supply chains, environment, political, information systems, and societal effects of NLPs. The group aims to create (or compile) guidance on these elements for the MERL sector.

The NLP-CoP secretariat will help cross-pollinate learning across the various working groups and support their functioning.

Join a Working Group!

To join a Working Group, sign up for the NLP-CoP by agreeing to the charter and completing the survey here.

Save the date!

Our next meeting will be July 27, 2023, 9am ET. We will explore the use of NLPs for Qualitative Analysis in African contests. Register here!