Designing for safety: A down-to-earth conversation about AI safety in practice

The AI Design & UX Learning Group is bringing together designers and researchers working at the frontlines of AI for social impact for a focused session on what safety means in practice. Across three short, discussion-led segments, we’ll move from definitions to practice: exploring how actors in the social sector are identifying specific AI safety risks in the tools they build, how that thinking is shaping concrete design decisions, and what rigorous safety testing looks like, including why automated tools alone are not enough.

Register here to join us on June 18th at 8AM BST / 12.30 IST

AI safety concerns range from immediate harms already visible in deployed tools, including biased outputs, training data that excludes the communities a tool is meant to serve, and systems that quietly render vulnerable populations economically invisible, to existential risks  from future superintelligent systems. 

On May 25th at the Vatican, Pope Leo XIV presented Magnifica Humanitas, a formal letter to Catholic bishops and the church on “safeguarding the human person in the time of artificial Intelligence”. Standing alongside Anthropic co-founder Chris Olah, the Pope said that “very troubling accounts” had reached him “ of algorithms that can block access to healthcare, employment and security on the basis of data tainted by prejudice and injustice,” and “the silence of those who have no voice when decisions are made—decisions likely to generate new forms of exclusion and suffering.” (Summary of Bulletin, Holy See Press Office, 25 May 2026). Chris Olah named risks he believes the tech industry cannot confront alone, including AI displacing human labour at massive scale, with development concentrated in “a handful of wealthy nations” and no existing mechanism to share its gains with poorer countries (Anthropic).

Last month, India’s Finance Minister Nirmala Sitharaman convened an emergency meeting with the country’s banking sector over Anthropic’s Claude Mythos model, warning of “unprecedented” cybersecurity risks. And just last week, 1.8 million patient records, including fingerprints, diagnoses, and precise geolocation data, were confirmed stolen in a months-long breach at NYC Health + Hospitals (Malwarebytes)

Olah’s conclusion at the Vatican was unambiguous: every frontier AI lab, including his own, “operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing,” which is why the industry needs people with “Moral voices that incentives cannot bend… who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful, critics….It is through dialogue and mutual effort, through the push and pull, that humanity will achieve great things.” (Anthropic). 

Those voices exist,  including among practitioners building and deploying AI tools in contexts where the stakes are immediate and the users are among the world’s most vulnerable populations. For those of us designing AI tools for social impact in low- and middle-income countries, the risks the Pope named are not hypothetical. They show up in whether a health chatbot offers relevant, viable referral pathways, in whether a farming tool designed to reach marginalized women actually does, and in how we test for harm in communities who are underrepresented or missing from training data.

Join us to be part of this conversation. We’ll start with hearing from you about your safety concerns, provide a brief overview of AI safety definitions, and then move onto a series of curated discussions. We’re excited to be joined by:

• Dr Namita Singh, Sr. Director of User Research and Insights at Digital Green, will present the evolving risk taxonomy for Farmer.Chat. Namita and Tetyana Zalenska, Director of Monitoring, Evaluation and Learning, will then share findings from Digital Green’s ongoing research to assess these risks in the field, and how they hope this evidence will influence new distribution, product and growth decisions. 
• Stephen Obonyo, Machine Learning Engineering Manager at Jacaranda Health, will share how they ensure safe maternal messaging through a rigorous data-design pipeline, a smart triage system that instantly escalates emergencies, and an automated AI auditor that evaluates every single response for medical accuracy before it reaches a mother.
• Deepali Mittal, Founder of UserTrace, which allows advanced testing of AI models at scale via synthetic personas,  discussing how AI itself can be used to support safe practices.

These conversations will be followed by a discussion of participants’ reactions, and their own experiences in Safety by Design.

Attendees can expect a grounded, practical conversation about what AI safety actually looks like in the tools we design: from risk registers and design decisions through to testing methodologies and real-world safety mechanisms.

Olah’s call at the Vatican was addressed to religious communities, scholars, and civil society, including people like us, working at the intersection of AI and social impact, asking hard questions about harm, and insisting on design that puts the most vulnerable users first. We hope to see you there.

Not yet part of The Design of UX and UI for AI Learning Group? Use the link here to register and join a growing community of practitioners designing AI tools for social impact.