Agentic AI: Governance, risks, and responsible deployment in the social sector

At our December 8th Technology Salon NYC session, we unpacked the topic of Agentic AI – what is it?, how does it differ from GenAI and AI Agents? What does Agentic AI mean for organizations working in the social sector? The conversation surfaced critical questions about governance, accountability, consent, and how we, as a sector committed to equity and justice, should approach this rapidly evolving technology.

Our lead discussants were Faisal Lalani, Head of Global Partnerships, Collective Intelligence Project; Leah Ferentinos, Product Manager, AI Governance Intelligence, Credo AI; and Angeline Lee, Product Policy Manager, Office of Ethical and Humane Use, Salesforce. We welcomed some 40 Salon participants from various sectors and points of view, from organizations wondering about Agentic AI to those developing policy and governance around it to those who are building it to those who are skeptical of it, meaning it was a great opportunity to try and understand varying perspectives. We were graciously hosted at the Rockefeller Foundation’s Convening Lab.

The conversation was wide-ranging. The two hour session flew by as we bounced from theoretical ideas to specific examples. A big takeaway for me was that we don’t have collective clarity about what Agentic AI is. Like with any other AI, it is really important to be specific about the kind of AI (and in this case, the kind of Agentic AI) we are talking about, what we want to use it for and with whom, if we want to be precise about its risks and benefits. We should start with questions of why we want to adopt Agentic AI and at what cost? rather than rushing into adoption out of #FOMO (fear of missing out) and vague promises of its efficiency gains. 

Here are key themes and takeaways from the session:

1. What do we mean by “Agentic AI?”

Agentic AI refers to AI systems that can plan, decide, and act autonomously toward a high-level goal. Unlike traditional or generative AI—which responds directly to prompts—an AI agent is goal-driven. It can perceive its environment (read an email, scan a case record, check a database), reason about the goal and break it into smaller sub-tasks, act by calling external tools, APIs, enterprise systems, or other AI systems, and reflect and adapt by learning from intermediate results to improve its plan.

The core technology combines a large language model as the “brain” for reasoning, planning, and interpreting context, with tools, memory, and an execution framework as the “body” that lets it take multi-step actions. In terms of what differentiates Agentic AI from other chatbots, one participant noted, “A traditional bot has a simple decision tree. Generally, if it doesn’t know, it will not say. With agentic systems, the goal is autonomous decision-making. If it doesn’t have an answer, it will invent one.”

The more data AI agents can access, the more useful they are. An agent may need access to your emails to pull information about an event and access to your calendar to confirm your availability and add it to your schedule. This capability is both the promise and the concern. Issues arise with regard to autonomy, personhood, liability, and accountability when agents perform tasks “autonomously” yet also on behalf of a company or user. Privacy and security concerns arise if an agent is doing something on your behalf—sifting through your inbox, making purchases, accessing sensitive organizational data.

2. Accountability and Agentic AI

Agentic AI distributes responsibility in a new way, creating shared risks and shared accountability across multiple actors. As one person put it, there are at least 3 points of accountability:

• The Provider who sets foundational guardrails—technical, policy, governance, and safety constraints as well as platform-level protections and required oversight patterns.
• Model vendors (e.g., OpenAI, Anthropic, etc.) who determine the underlying model quality, training data sources, safety classifications, and fine-tuning policies that shape downstream privacy, legality, and bias risk.
• Customer/implementer who decides how the agent is deployed, what data it touches, what actions it is allowed to take and who is responsible for human review, operational checks, and responsible use of organizational data.

With Agentic AI, no single entity controls the system end-to-end, which means risks must be actively managed across all three layers. As one participant put it, “So many players—if something goes wrong, who is at fault? This challenges the UN Guiding Principles on Business and Human Rights.”

For example, if a finance agent incorrectly denies loan pre-qualification based on ZIP code, the deployer is responsible for using ZIP code inappropriately, the platform may be responsible if guardrails failed, and the model vendor may be accountable if the model reasoned in an unsafe or biased way. This is why providers require human oversight for high-impact decisions and why customers/implementers need to validate constraints before deployment.

A major theme at the Salon was the inadequacy of current consent models and the fact that they are largely unenforceable and need to be updated. As one participant noted, “We are missing a few governance instruments, including how to ensure agency in the development of agents…. The data governance regime we have is all around consent but this is a flawed consent. It’s never informed, and it’s individual, so it doesn’t work.” The idea of a “social license” is emerging but not yet common practice. “Consent is currently binary—how do we make it less binary and more collective? Are there multiple options for consent for our data?”

3. New risks and concerns

In addition to higher level concerns (we will discuss these later in this post) immediate concerns around Agentic AI included misuse for illicit activities (e.g., CSAM, fraud); prompt injections, loss of control/manipulation; bias and cybersecurity. In terms of cybersecurity, one person laid out three ways risks manifest:

• Systems working incorrectly: These kinds of inaccuracies and errors can be addressed by the developer at a technical level.
• Systems working correctly: These represent catastrophic risks, when the system does exactly what it was asked to do, but the ask itself was flawed or harmful.
• Malign users: Loss of control, illicit use, fraud.

As one Salon participant explained, “There are concerns about autonomous decision-making when an organization gives an agent increasing access. If you want to roll out a large-scale program that gives credit card access, that might be a good thing, but it can also go wrong. The more information you give it, the more surface area for it to go wrong.” Another person added, “If it needs 80% of your organization’s data to function, maybe don’t use it.” An additional security and privacy risk is that it’s becoming easier to reconstitute data even if anonymized.

4. So you want to deploy some agents…

Participants emphasized the importance of conducting thorough risk assessments before deploying agents at the organizational level. This includes:

• Use case definition: What exactly are you trying to accomplish?
• Risk scoring: What could go wrong, and how severe would the consequences be?
• Data scoping and minimization: What data does the agent actually need access to?

As one participant shared, before deploying ¨we went through all our low stakes tasks, obviously not feeding in PII or customer data, iterating that way. That’s a good place to start. Use a risk framework approach. Think about the annoying things you want to automate and what is low and high risk. What could possibly go wrong? Then try the low-risk, high-efficiency things first.” 

The question “how do we even know what is ‘low risk’?” came up. One person suggested asking, “If it hallucinates, what is the worst that can happen?” (I like to think of this as a “dystopian theory of change”). Document formatting or admin work such as calendar management were suggested as a good place to start piloting. Another area mentioned was triaging and routing. “But there needs to be a human that reviews it and hits send.”

One organization shared their approach: “Use Agentic AI to help develop insights. Be very deliberate about it. Pay attention to change management and have lots of internal discussions. Be clear on the use case. Get consent. How will we use it? If we plan to use it with partners, we will have to have these same conversations with them. Test before deploying. What are the SOPs and guardrails. What will the law be in six months?”

Multiple participants emphasized human oversight. “You need to understand the tech, check if the outcome makes sense—you are the subject matter expert.” Others warned that “human-in-the-loop” can only go so far, and felt that human-in-the-loop is more relevant in the AI assurance research field as a possible solution for catastrophic risk, but it’s not a solution to consent or broader equity concerns. (“Human-in-the-lead” may be safer and more appropriate!’)

5. Coordination and governance of the multi-agentic future

The future will be ever more complex, however, said one person. It will be multi-agentic. “There won’t be just one AI model operating at scale, but multiple agents. That means that as the cost of intelligence becomes near zero, coordination is what becomes important, both in society and in organizations. How do you interact with all these AI agents all at once? This is the ‘handshake problem’: the more agents, the more interaction among them. What guardrails do we need?” 

Another critical gap emerged as related to “agent-on-agent” behaviors. “A lot of the use cases for agentic systems are based on the agent interacting with the human layer of the internet. We don’t have good consent models for agent-on-agent behavior. You assume your agent will interact with another human. But we are going to be in an environment where agents are acting with agents. How do you know who you (or your agents) are interacting with and what their own guardrails and controls are? This future of the internet is coming quickly.”

The concept of digital twins emerged as well. As one person explained, we will have AI agents that represent us and act on our behalf, reflecting our goals and personal values. This raises profound questions about how we capture individual and collective preferences and expectations.

As we hung around chatting post Salon, this topic came up again – and people wondered whether it would be annoying to be dealing with a bunch of different agents for different tasks, highlighting the coordination challenge. (And personally, making me want to avoid stepping into that future!) Consider several agents making autonomous decisions on your behalf, interacting with other autonomous agents, and little human knowledge or oversight of who or what is behind these other agents.

6. Evaluating agents and dynamic governance

The conversation turned to evaluations as a governance mechanism. As one participant explained, “Evaluations are basically tests for your AI. They are your students and you are the teacher. They are distributed and created by experts who know about LLMs but not about your domains. AI agents are normally benchmarked this way, but context and variability matter.”Ongoing monitoring is needed. “Just as you check in with employees, you need to check in with your agents.”

A platform called weval.org, for example, can be helpful for setting up and sharing your own parameters and testing AI models to see if they meet expectations. The need for dynamic governance and evaluation was emphasized. “Lots of AI evaluations are static. What if everyone could enter criteria of what matters to them? We had Sri Lankan civil society contribute criteria to test an AI model to see if it could answer questions about Sri Lanka.”

7. What happens to human relationships when we add Agents?

One participant raised the familiar hammer looking for a nail phenomenon, “We are seeing the same mindset as before when people wanted to make an app for everything, whether or not it’s even needed.” (AI hype frequently reminds me of the blockchain days, when blockchain — with its huge climate costs — was pushed as a solution, yet most of the time an excel spreadsheet would have been simpler and sufficient).

Multiple participants raised concerns about what happens to human relationships and human work when we bring in AI. As one person put it, “What do any of these agents do to the human experience of the action? Taking out a step a human may have done in a workflow can create a disruption in the thought processes and introduce a sense of alienation. It creates gaps in how teams function and work together.” We should be calculating not not just a business case or a financial case, but looking at the loss or cost for humans and the human experience.

For example, said one person, “if we have AI write code, people are inheriting that code which is not theirs. It might be faster on the front end, but not on the back end, because they have to review and familiarize themselves with someone else’s code. It does not allow them to be involved and grow with the work and the task.” Others wondered what AI is doing to relationships and thinking more broadly. Normally as humans we are “swimming in data, sleeping and dreaming in the work,” and when the AI does that work and hands back a set of bullet points, the connection goes missing. How does introducing AI affect human relationships and how teams work together and relate to one another?

8. AI as a band-aid for dysfunction

Several participants pushed back on the rush to implement agents without addressing underlying organizational issues. The point was made that there is no digital or AI transformation without cultural transformation. “You end up building for the sake of building and it just magnifies technical and cultural debt. If you have siloization between two factions in your organization and you drop in some agents so that marketing can talk to finance, but they are fighting each other for budget, you will just weaponize AI.”

In any case, nonprofits often lack the data sets to do any of this. “Agentic AI requires discipline. You need to be so clear on what you want it to do. It can get really uncomfortable if you are a nonprofit and don’t know what you are doing or what you want to measure. A lot of nonprofits will struggle implementing Agentic AI because they don’t know what they are trying to do or what they are trying to solve with an AI Agent.” Adopting AI for the sake of it or in order to “keep up” will lead to wasted investment and potential harm.

9. Critical AI literacy

Critical AI literacy was a point raised multiple times throughout the conversation. This included policy makers understanding how AI works and the forces driving it, as well as tech developers understanding how policy works. One person explained that “we often approach this from a culture of ‘policy makers need to understand what we are doing,’ but we also need to understand the government context to channel things into policy…. How do we make this translation of tech concepts to non-tech concepts? Don’t make people speak your language.” The need for more cross-disciplinary conversations is clear.

Another participant introduced the concept of question literacy. “We have an issue with question literacy. If you don’t know what the problem is, don’t build the solution. What is the question that even matters that Agentic AI can act on? What is the problem you are trying to solve? What is the question that will make the difference if you have an answer to it? We need to think about question science.”

Final thoughts

Several themes emerged from the conversation:

• Slow, measured adoption is not anti-innovation. We should not let fear of missing out drive us to deploy technologies that could harm the communities we serve.
• Governance needs to be multi-layered and adaptive. No single entity controls agentic systems end-to-end, which means we need robust governance across providers, model vendors, and implementers.
• Human relationships and organizational culture matter. Technology cannot fix broken processes or dysfunctional teams. We need to address underlying issues before layering on automation.
• Question literacy is critical. Before we build solutions, we need to understand what problems we’re actually trying to solve and what questions we need to answer.
• Consent models need fundamental rethinking. Binary, individual consent doesn’t work in a world of agent-to-agent interactions and collective data implications.
• Critical literacy around AI is key. Without it we risk jumping into something due to fear of missing out and with little understanding of the why and the potential immediate and long-term costs.

Technology Salons run under Chatham House Rule, so no attribution has been made in this post. If you’d like to join us for a Salon, sign up here. If you’d like to suggest a topic, please get in touch! Please contact us if you would like to discuss sponsoring a Salon or offering financial support for our work!