Authors: Sara Chamberlain, Jona Repishti

An LLM evaluation of FarmerChat, run across a sample of 116,411 message–bot pairs, found negligible gender bias in the quality of the chatbot’s advice. The conversations that women and men had with Digital Green’s AI-enabled agricultural advisory service were graded across eighteen separate quality measures. To assess bias in response quality researchers looked at pertinence, actionability and complexity. The scores were almost identical, indicating no identifiable gender gaps in bot functionality. 

Yet, women still stopped using FarmerChat faster than men. After controlling for quality dimensions, the women farmers in the study were 18% less likely to return after seven days, and 26% less likely after thirty. The barrier was not in the AI model. 

Investment in addressing AI Agricultural advisory risks has prioritized critical upstream factors, such as improving training data and model responsiveness, and conducting algorithmic audits to improve accuracy and remove bias. But a landmark, 10-month program of design research into FarmerChat suggests that downstream risks matter just as much. 

FarmerChat has already been downloaded 2.4 million times in 16 languages across 6 countries, and the research is extensive: 

• an LLM evaluation of 116,411 message-bot pairs, 
• a baseline survey of 2,995 farm households across 599 villages in Kenya for an RCT, 
• a survey of nearly 428 households with 822 respondents in Ethiopia to estimate the addressable market, 
• qualitative fieldwork with more than 300 farmers across India, Kenya and Ethiopia, 
• engagement-log analysis of over half a million FarmerChat messages,
• analysis of  paid social media marketing campaigns in India, Ethiopia, Kenya, and Nigeria,
• and randomised A/B tests of FarmerChat’s UX and UI involving hundreds of thousands of users. 

Drawing on this research and two decades of institutional experience of delivering agricultural guidance to smallholder farmers, Digital Green created a Taxonomy of Risks: a diagnostic framework to make these risks visible and measurable, and to help practitioners build more gender-inclusive AI tools and programs

A Taxonomy of Risk 

The taxonomy moves the conversation from “can we generate a technically accurate response?” to “can farmers access, navigate, understand, trust, implement, and benefit from advice within their real social, economic, and ecological contexts?” It identifies ten ‘gates’ that sit between deploying a model and achieving equitable impact on the ground: 

• Access 
• Usability
• Relevance
• Accuracy 
• Adoption 
• Norms
• Income
• Accountability
• Environment
• Data rights and privacy

Each represents a place where an AI tool can lose a user. 

What makes the taxonomy stand out from existing risk taxonomies is that it has been developed by practitioners who are designing and implementing AI Ag advisory tools at scale, and is based on practical design research that seeks to solve real-world problems. Although the underlying research is agricultural, the risks the taxonomy identifies are relevant to other domains. They apply to any AI advisory service in health, finance, or education. 

This blog shares evidence of risks for five gates, starting with the accuracy gate, where the LLM evaluation described earlier found no meaningful gender gap in the quality of FarmerChat’s responses. The following discussion dives deep into four more gates: access, usability, adoption, and norms, where Digital Green’s research currently offers the richest insights and the clearest picture of what practitioners can act on today. Research across the remaining gates continues, and our understanding of the risks continues to evolve.

Explore the Risk Taxonomy here

 Access 

The first gate in Digital Green’s risk taxonomy is foundational: do digital marketing campaigns and face-to-face onboarding efforts reach farmers? And if they do, can she install the app? AI-enabled advisory fails when the answer is no, either because she never hears about the app or because she can’t download it on day one.

Digital Green analyzed the two-month performance of social media marketing campaigns across Bihar (India), Ethiopia, Kenya, and Nigeria. The advertising was very successful at capturing women’s attention: female users were responsible for 79% of impressions and clicks on just 67% of the budget. But at the critical step of application download, three to four times as many men installed the apps as women.  Women were curious, but something prevented them from downloading the apps on their own – especially onto shared phones. 

In Ethiopia, Digital Green found that only 13% of spouses in male-headed farming households own a smartphone compared to 50% of male heads. In Nakuru, Kenya, only 10% of primary smartphone users in male-headed farming households are women. Women’s access is therefore largely mediated. 

Digital Green’s qualitative research suggests that in rural farming contexts, the household, not the individual, may be the effective adoption unit. Women reported navigating layers of household permission to install the app without friction, including securing approval from whoever shares her phone, and justifying data use. As one woman in Bona Kebele, Ethiopia, shared: “I sometimes avoid installing new apps because my husband might suspect I am communicating with travel agencies… if I install new apps, he thinks I will leave him soon.”

Women’s lack of digital access, skill and agency has a measurable cost. Analysis of Digital Green’s social media marketing campaign data revealed that the advertising spend required to generate one installation of the app by a woman was three to four times higher than for a man. The spend required to generate one first query was nine times higher.

Opening the gate: 

• Start with research: Impressions, clicks, and cost per click may look encouraging at first, but mean little. Track the click-to-install rate and the cost per first query by gender instead. Those are the numbers that show whether the campaign was effective.
• Acting on it: In contexts with steep gender divides, inclusivity may lie in partnerships with organizations already reaching women at scale. In Ethiopia, qualitative research indicated that the most credible endorsement of FarmerChat was a Development Agent or neighbor who had used it, pointed to a positive result, and provided hands-on help. Yet Digital Green has found that digital acquisition of women users is considerably less expensive than face-to-face, and much faster to scale. These insights highlight that there is no one size fits all approach, and digital marketing and face-to-face outreach should be complementary channels.

Usability 

The second gate in Digital Green’s risk taxonomy is usability: can farmers navigate the app to ask a question, and if they can, are they able to understand the answer? AI advisory can fail when the interface is too complex to navigate, when answers are too complex to understand, when they’re in a language the user doesn’t speak or a modality they cannot use. 

The Kenya baseline survey for FarmerChat found that women perform significantly worse than men on every digital skill measured, from basic browser use to more advanced actions such as managing tabs, bookmarks, or navigating app stores. These gaps are large in magnitude and persist even when women are the primary users of smartphones or the heads of households. This gap matters because analysis of the baseline data, cross referenced with usage engagement statistics, found that digital literacy predicts how many questions a farmer asks FarmerChat more reliably than anything else measured. Each step up in digital literacy came with a 16% jump in query volume. 

The quality of questions also matters, since it shapes the quality of answers. One study found eighteen percent of FarmerChat queries arrive with no crop, location or growth stage attached, limiting how targeted an answer can be, regardless of model quality. Women’s text queries were shorter and less explicit than men’s. The research also found that voice queries are typically shorter and more implicit than text, and female users rely on voice nearly 10 percentage points more than male users (47.6% vs. 37.7%), producing less precisely targeted, more verbose responses from the app. Digital Green’s research indicates that voice use is a signal of lower text literacy or a preference for oral communication. 

Opening the gate:

• Start with research: Track query completeness (the share of queries that arrive with crop, location, and growth stage attached) as a standing product metric, not a one-off analysis. Track modality split by gender too. And treat digital literacy as something that actively shapes engagement, not just a fact about who’s using the app. 
• Acting on it: Digital Green is experimenting with prompts that help farmers supply more context, enabling the model to provide more specific answers. These are ready-made questions a farmer can tap rather than compose, lowering the bar for knowing what to ask in the first place. Usage data shows women use starter prompts twice as much as men. Some prompts appear as tappable images. Digital Green tested which image style resonates most with farmers. Both female and male farmers preferred real photographs of crops and livestock to illustrated or photorealistic cards. Authenticity beat aesthetics in every country where the cards were tested. 

Adoption 

Adoption of agricultural advice is another critical, often overlooked gate in Digital Green’s risk taxonomy. The advice is correct. But can the farmer act on it? AI advice can be technically sound and still fail. Not because the recommendation is wrong, but because farmers, especially women, face constraints that stand between advice and action. 

Quantitative data from the Kenya baseline survey shows that husbands retain final authority over major agricultural decisions, dominating crop choice (62.4%), fertilizer purchase (61.2%), and trying new advice (58.9%). Spouses’ agricultural agency ranges from more collaborative and consultative decision-making arrangements to highly unequal structures in which women have limited or no meaningful participation in agricultural decisions. However, even in a consultative context, male heads often retain final authority over decisions, reflecting the persistent influence of traditional gender norms and divisions of labor in rural agricultural contexts. Hence, even when spouses receive and understand advisory content, they may not have the authority to act on it independently.

How to open the gate: 

• Start with research: Measure decision-making power at baseline, by domain of decision, and report advisory outcomes conditional on it. A service that counts “advice delivered” or “advice understood” as its outcome will show impact among exactly the users who already had the authority to act.
• Acting on it: In a recent 60 Decibel study in Kenya, 82% of active women users who participated in the phone survey reported applying FarmerChat’s advice. Among these women, 81% reported that their farm income “very much increased” (as opposed to 60% of men), with 86% reporting greater yield, and 63% reporting improved crop/ livestock health. These findings suggest real benefits for women who are able to apply the app’s advice. Among women without this agency, systematic reviews of SHGs found positive effects on independence in financial decision-making. Digital Green has partnered with women’s collectives since launch, embedding FarmerChat into networks already expanding women’s decision-making power. 

Norms 

In Digital Green’s risk taxonomy, gender is not just a separate “inclusion” section tacked onto a technical risk framework. It is woven into every gate, based on gender disaggregation of most research data. However social and gender norms also introduce risks that deserve separate attention. 

Although digital literacy is the single strongest predictor of how much a farmer uses the app, it does not explain the whole story. The gender gap in usage, where women initiate far fewer sessions than men (7 vs 10),  persists even after controlling for digital literacy, personal autonomy and risk-taking. The evidence points to access constraints. 

Interpretation of gender disaggregated engagement logs suggests a gendered daily rhythm consistent with women’s time poverty rather than preference. While men’s usage spreads across a broad evening window (18:00–21:00 EAT), female usage is earlier in the day (10:00–13:00 EAT), with voice queries peaking from 6 to 8am. This is plausibly before women’s household work and caring responsibilities begin, in advance of their farming work. Yet once women begin a FarmerChat session, they ask an identical number of queries as men (2.44 vs 2.48).  The gap is in how often she can return to the app, not in how deeply she uses it once she’s there. 

Women also navigate deep, socially embedded digital anxieties. During qualitative interviews in Ethiopia and Kenya, they express persistent worries about consuming shared mobile data, taking up phone storage, or triggering suspicious notifications. 

Opening the gate: 

• Start with research: Separate usage statistics on how often women start a chatbot session from how many questions they ask once they’re using it. If these statistics are combined into a single engagement number, the gender gap looks like women disengaging from the app. Split them apart, and the real story is different: women ask just as many questions per session as men, they just can’t return as often.
• Acting on it: Digital Green has partnered with mobile network operators in Kenya to zero rate FarmerChat, so no data is consumed using the app. This powerful approach has led to a 27% jump in questions asked by women users, pointing to how industry partnerships can lower the bar to entry for disadvantaged women. The 60 Decibels study in Kenya also found that FarmerChat has a strong multiplier effect offline. The research suggests that nearly eight in ten farmers share the app’s advice with others, with both women and men sharing with at least 3 people.

A practical evidence map, not a theoretical framework

The combination of transparency and specificity is what makes the risk taxonomy a practical tool, not just a conceptual one. Designers gain a checklist of evidenced risks. Program managers gain a basis for developing more inclusive implementation strategies. Monitoring, Evaluation and Learning teams gain a defined set of constructs and metrics, spanning model performance, AI products, intended user populations and their contexts, and differential outcomes for women and men farmers, that are ready to be operationalized into indicators. And investors gain a framework for stress-testing funding proposals, beyond the technical robustness of the model. 

An invitation to test the taxonomy

We invite practitioners building AI advisory services, and researchers evaluating them, to apply this taxonomy directly to their own work. If your program has data that speaks to any of the taxonomy’s gates, we’d welcome hearing about it. A taxonomy of this kind gains its value only through use: tested, challenged, and reported on across the wider research community, not treated as a finished instrument.

Explore the Risk Taxonomy here