What role is NLP playing in Social and Behavior Change Communication?
By Louis Davison & Stephanie Coker
On the 28th November, 2023, MERL Tech’s Natural Language Processing Community of Practice (NLP-CoP) hosted a presentation by Shujaaz and Girl Effect, two NGOs using NLP to effectively improve their communications campaigns, chatbot technology and audience segmentation in order to improve young women’s Sexual and Reproductive Health (SRH).
Our speakers were Karina Michel and Soma Mitra-Behura from Girl Effect, and Anastasia Mirzoyants from Shujaaz and Bryson Mwamburi Mwakuwona from iMedia. The meeting was chaired by Stephanie Coker with support from Louis Davison; both co-leads of the NLP-CoP Social and Behavior Change Communication (SBCC) Working Group. Topics discussed ranged from the use of NLP in precision messaging and sentiment analysis, to the construction of chatbots to circumvent social taboos.
The event was attended by MERL professionals, consultants, researchers and academics, as well as business professionals and other interested parties. Audiences joined the call from a broad range of locations including Uganda, Nigeria, Ghana, Edinburgh, London, New York, Minneapolis, and South Africa. Professionals from key organizations such as NIRAS, GIZ, IRI, Save the Children, UNHCR & UNICEF were also present.
Access the recording and slides here.
Girl Effect
Karina began the discussion by presenting Girl Effect’s AI and machine learning roadmap focused on Big Sis, a chatbot launched in South Africa in 2018. (Slides here). Created to circumvent social taboos and get young people talking about SRH, Karina talked about significant impact and achievements, forecasting that 2,000 girls will have access to life saving services by the end of 2023 because of Big Sis. Following success of the South Africa launch, Girl Effect have also started two new chatbots in India and Kenya named Bol Behen and Wazzii, respectively.
Karina went on to explain the role that the chatbot takes as a confidante in the ‘Big Sis’ role, providing real time answers to questions that ‘inspire, equip, and prepare a young person to take action, like book an appointment for HIV Test’.
Following Karina, Soma talked about the languages of Big Sis. After expanding to India and Kenya, Big Sis has had to adapt its language and lexicon to suit local needs. The bot is trained in both Sheng (Swahili-English) and Hinglish (Hindi-English), which was made possible by the advent of GPT 3.5 & 4. Soma went further to say that Big Sis technology goes beyond language: their chatbots are engaging because they speak with the tone and personality of a teenager. Soma noted that this replication of tone might also be one of the biggest challenges to maintain over time and as they expand.
Soma went on to discuss the difficulties and responsibilities of Big Sis chatbots, particularly focusing on hallucinations, mistakes that an AI might make when asked a question to which it may not know the answer. Girl Effect has been careful to ensure that the bot is built for safety, safeguarding all data and protecting against hallucinations by constant monitoring.
After the presentation, a Q&A session facilitated by Stephanie and Louis yielded some key insights for the audience. These insights include having chatbot teams largely based in-country, using data integration to understand changes in individual girls to take action after receiving advice, hiring trained human staff to facilitate intent to action, and ensuring that informed consent is obtained from users. Other themes highlighted were Girl Effect’s use of Theories of Change (based on 8 psychological drivers for adolescent decision making) to guide research into behavior changes and monitoring tone and style of bots to account for code-switching behavior.
Shujaaz
Following Girl Effect was Shujaaz, presented by Anastasia Mirzoyants and Bryson Mwakuwona. Shujaaz is a multimedia platform that aims to help improve the lives and livelihoods of young people in East Africa. It was launched in Kenya in February 2010 and is written and presented in Sheng. (Slides here)
The central question of their presentation was: How can NLP help us with digital segmentation?
Bryson stated that NLP can revolutionize the way we monitor, understand and modify how we engage with audience needs at scale. Shujaaz is using NLP and LLMs to gather data which informs their segmentation of their audience and maximizes relatability. Using these clear segments enables precision messaging, enhanced relevance and a clearer picture of sentiments within East African youth culture.
Bryson outlined Shujaaz’ process for using NLP for digital segmentation. The first step was to translate youth conversations. After testing multiple NLP tools, the team used ChatGPT 3.5 and 4 because it handled translation well, including euphemisms and understanding different tone implications.However, there remain challenges with interpreting emojis and other non-text forms of conversation. The second step involved topic modeling and sentiment analysis, which sought to obtain meaningful topics from translated user discussions. Bryson shared several key lessons that emerged during this step, such as using a multilingual model to provide a safety check against ChatGPT erroneous behavior and using qualitative analysis to double-check and contextualize topics generated. After this step, Shujaaz used topics formed to create different audience personas which could then be augmented by new data. Anastasia concluded the presentation by outlining several opportunities and challenges that NLP brings to the social sector, such as the ability to quickly scale digital implementation of new programs and the need to have safety checks for each model’s behavior.
After the presentation, audience members asked questions on learning from actors in the commercial world and the cost-effectiveness of implementing NLP digital segmentation. Key points highlighted were the need to be attentive to dubious ethics of models generated by commercial actors who are primarily concerned with a financial return and difficulties estimating the time to completion for such projects as well as the costs of upskilling professionals in the social sector to use NLP.
What do people wish to explore in 2024?
To wrap up the session, Stephanie and Louis shared key trends occurring in the SBBC space (slides here) and solicited input from members about topics of interest for 2024. Audience members were keen to explore more about the cost-effectiveness, cost-benefit and impact measurement of SBBC projects. In particular, people were interested in understanding what it costs to develop more sophisticated systems and staff expertise in NLP. Some members of the audience were also interested in exploring SBBC models for groups that are excluded from or difficult to reach in web and digital spaces.
Finally, audiences were interested in creating problem-solving sessions where individuals could come with questions related to SBBC and crowdsource solutions.