Indaba is an isiZulu and isiXhosa word for an important meeting held by leaders in South Africa to discuss critical matters. This past week, I’ve been listening in at the Africa Evaluation Indaba, organized by The University of the Witwatersrand and CLEAR Anglophone Africa. Critical matters have indeed been discussed!
At the upcoming session, we will discuss ways that M&E practitioners can improve data management and how they can play a role in improving data governance practices at the institutional and national levels. We will open the floor for discussion and consultation on priority areas and gaps in the practical aspects of responsible data management as well as in data governance processes that improve accountability.
Following the Indaba, will draft up a plan that lays out how we can best offer training, guidance and support for the M&E community with relation to responsible data management and data governance. We also plan to develop a set of guidance documents on responsible data governance and M&E together with the RDiME Working Group, which is made up of a group of experts who have data governance, data protection, and evaluation-related expertise and experience. We hope the RDiME Alliance’s work will support government evaluation efforts as well as civil society organizations and evaluation firms.
What makes the Africa Indaba Evaluation conversations so exciting (for me, at least!) is that they are framed within a lens of decolonization and transformation. This past week, topics included:
“Transforming Evaluation: The Race, Power, Gender and Class Struggle,” with speakers covering questions like: how do we locate evaluation within the historical context of asymmetrical global power relations and aid dependency? What needs to be done to dismantle systems and structures so that evaluation does not become complicit in entrenching existing inequalities? (Monday 16 November)
The Made in Africa (MAE) Evaluation approachwhich arose out of the quest for contextually relevant approaches, methods that emphasize the centrality of contextual relevance and place importance on indigenous knowledge systems. (Tuesday 17 November)
The launch of the Global Evaluation Initiative, (GEI) which aims to offer better coordination of evaluation resources and to support local and international organizations working in the area of evaluation. (Wednesday 18 November)
(Recordings of these sessions will be available soon).
CartONG has just released a new study on “Program Data: The silver bullet of the humanitarian and aid sectors? Panorama of the practices and needs of francophone CSOs“.
What place for program data management in a sector in the throes of a digital revolution?
Mirroring our society, the Humanitarian Aid and International Development (HAID) sector is in the throes of a digital revolution. Whilst the latter is undeniably impacting day-to-day management of Civil Society Organisations (CSOs) – whether in their administrative duties or in those related to fundraising – it is also generating radical changes in actions being implemented for the benefit of populations.
Although it has become a key element in the coordination of operations, data management remains somewhat invisible from the perspective of the sector, in spite of its many ethical, financial and human implications, and above all its impact on project quality. In the field and at headquarters, project teams are therefore devoting an increasing amount of time to data management, often at the expense of other activities. Poorly trained and ill-equipped, these teams can produce substandard performances with regards to these tasks, and without the topic necessarily being regarded as an operational issue by most CSOs.
Program data management – also known as Information Management (IM) – is both a topical issue and the source of numerous debates within francophone Humanitarian Aid and International Development CSOs.
A unique study in the world of French-speaking CSOs
At present and to our knowledge, no equivalent study with a view to examining, as a whole, the practices of (francophone) CSOs, or to identifying their needs in terms of program data management, has yet been carried out. A number of analyses and articles do exist, yet these generally approach the subject either from a technical standpoint or as if these were still innovations for the sector and thus with limited constructive hindsight.
The organisational dimension is moreover relatively unexplored and very little consolidated data at the inter-CSO level is available. Lastly, although CSOs have been handling large amounts of data for almost 20 years, there remains much debate: what level of attention and investment should data management be subject to? Does the activity require a dedicated person in-house and, if so, which profile should be given priority? In fact, where does the scope of data management begin and where does it end? Do CSOs working in humanitarian situations have different needs than those working in a development context? Do differences in approach exist between francophone and anglophone CSOs, the latter often deemed more advanced in the field?
Based on a survey of CSOs, a literature review and interviews with key stakeholders, this study designed by CartONG aims to explore and provide preliminary answers to these questions. It also aims to make a valuable contribution to bolster the debate on data management. To this end, we have thereupon sought to synthesise and formalise often scattered and at times contradictory considerations.
Based on the concept of Information Management (IM), program data management is a term whose scope of application continues to fluctuate and whose definition remains unclear. With a view to facilitating its ownership, readers of this new study will be given an accessible definition (synthesised in the diagram below) and a relatively small scope of application (see illustration below), at the juncture of Monitoring & Evaluation (M&E), Information and Communications Technologies for Development (ICT4D), information systems and knowledge management.
Main components of Information Management
Simplified diagram of the place of Information Management vis-à-vis related topics
Program data management & Francophone CSOs: an overview of the main stakes and of the existing relationships by categories of CSOs
Despite studies still being relatively sparse as to the link between project data management and project quality, the available evidence shows that good data project management makes for greater efficiency and transparency in organisations. The evidence gathered suggests, however, that project data management is widely used today for the benefit of bottom-up accountability – towards decision-makers and financial backers – rather than for day-to-day project steering.
The reasons for this state of affairs are manifold, but it appears that chief amongst them is a significant lack of maturity from francophone CSOs in matters relating to data and digital issues. Six main weaknesses and levers for action have thus been identified (see illustration):
an insufficient data literacy within CSOs
unduly fragile, siloed and insufficiently funded program data management strategies
a lack of leadership and often overly vague responsibilities;
a technological environment that is neither controlled nor influenced by CSOs
the use of approaches that foster information overload and neglect qualitative data; and
an under-estimation of the responsibilities carried by CSOs and of the ethical issues at stake with regard to the data they manipulate.
Confronted with these challenges, it appears that francophone CSOs are somewhat lagging behind – at least in terms of awareness and strategic positioning – compared to their anglophone counterparts. Moreover, program data management continues to be approached by the various CSOs in an inconsistent manner: the study therefore proposes a classification of CSOs and reflects on the main existing differences – between types, sectors and sizes – and in particular points out the difficulties encountered by the smallest organisations.
What types of IM support are expected by Francophone CSOs and on what priority themes?
This study was also an opportunity to identify both the type of materials and on which priority program data management themes a support is expected by francophone CSOs (see below); especially to enable specialized organisations, including H2H/Support CSOs such as CartONG, to better define their priorities of support toward CSOs.
The study also reveals that CSOs are mainly waiting for accompaniment on the following topics (in this order):
selection of solutions
responsible data management
data quality control
data sharing and, for the smaller ones also
database design and
simple map visualization.
What follow-up does CartONG intend to give to this study?
The study closes with a series of some fifteen recommendations to the various international aid and development actors, especially CSOs, who would benefit from being more proactive on the topic, as well as to donors and network heads who play a pivotal role to advance these issues.
By clarifying the various elements feeding the debate along with the issues at stake, we hope that this document – which remains a first for CartONG – will help feed current discussions. Many of them should actually be taken up again during the next GeOnG Forum that will be held online from November 2-3, 2020.
Carried out as part of the project Strengthening program data management within francophone CSOs carried out by CartONG (and co-financed by the French Development Agency – AFD over the 2020-2022 period), this study should be the subject of presentations during face-to-face or remote events before the year is out. It will also be enriched in the coming months by the release of many other resources.
Do not hesitate to follow us on social media or to write to us to be added to the project mailing list to stay informed.
Key aspects coming out of the events were the need for 1) guidance on data governance and 2) orientation on responsible data practices. Both policy and practice need to be contextualized for the African context and aimed at supporting African monitoring, evaluation, research and learning (MERL) practitioners in their work.
As a follow-on activity, CLEAR Anglophone Africa is calling on M&E practitioners to join up to be a part of this responsible data project for African MERL Practitioners. CLEAR Anglophone Africa and MERL Tech will be collaborating on this responsible data initiative.
The four panelists described the kinds of administrative or “routine” data they are using in their work. For example, in Kenya educational records, client information from financial institutions, hospital records of patients, and health outcomes are being used to plan and implement actions related to COVID-19 and to evaluate the impact of different COVID-related policies that governments have put in place or are considering. In Malawi, administrative data is combined with other sources such as Google mobility data to understand how migration might be affecting the virus’ spread. COVID-19 is putting a spotlight on weaknesses and gaps in existing administrative data systems.
Watch the video here:
Listen to just the audio from the event here:
Benefits of administrative data include that:
Data is generated through normal operations and does not require an additional survey to create it
It can be more relevant than a survey because it covers a large swath of the entire population
It is an existing data source during COVID when it’s difficult to collect new data
It can be used to create dashboards for decision-makers at various levels
Data sits in silos and the systems are not designed to be interoperable
Administrative data may leave out those who are not participating in a government program
Data sets are time-bound to the life of the program
Some administrative data systems are outdated and have poor quality data that is not useful for decision-making or analysis
There is a demand for beautiful dashboards and maps but there is insufficient attention to the underlying data processes that would be needed to produce this information so that it can be used
Real-time data is not possible when there is no Internet connectivity
There is insufficient attention to data privacy and protection, especially for sensitive data
Institutions may resist providing data if weakness are highlighted through the data or they think it will make them look bad
Recommendations for better use of administrative data in the public sector:
Understand the data needs of decision-makers and build capacity to understand and use data systems
Map the data that exists, assess its quality, and identify gaps
Design and enact policies and institutional arrangements, tools, and processes to make sure that data is organized and interoperable.
Automate processes with digital tools to make them more seamless.
Focus on enhancing underlying data collection processes to improve the quality of administrative data; this includes making it useful for those who provide the data so that it is not yet another administrative burden with no local value.
Assign accountability for data quality across the entire system.
Learn from the private sector, but remember that the public sector has different incentives and goals.
Rather than fund more research on administrative data, donors should put funds into training on data quality, data visualization, and other skills related to data use and data literacy at different levels of government.
Determine how to improve data quality and use of existing administrative data systems rather than building new ones.
Make administrative data useful to those who are inputting it to improve data quality.
Data is not always available, and it can be costly to produce. One challenge is generating data cheaply and quickly to meet the needs of decision-makers within the operational constraints that enumerators face. Another is ensuring that the process is high quality and also human-centered, so that we are not simply extracting data. This can be a challenge when there is low connectivity and reach, poor networks capacity and access, and low smartphone access. Enumerator training is also difficult when it must be done remotely, especially if enumerators are new to technology and more accustomed to doing paper-based surveys.
Watch the video below.
Listen to just the audio from the session here.
Some recommendations arising from the session included:
Learn and experiment as you try new things. For example, tracking when and why people are dropping off a survey and finding ways to improve the design and approach. This might be related to the time of the call or length of the survey.
It’s not only about phone surveys. There are other tools. For example, WhatsApp has been used successfully during COVID-19 for collecting health data.
Don’t just put your paper processes onto a digital device. Instead, consider how to take greater advantage of digital devices and tools to find better ways of monitoring. For example, could we incorporate sensors into the monitoring from the start? At the same time, be careful not to introduce technologies that are overly complex.
Think about exclusion and access. Who are we excluding when we move to remote monitoring? Children? Women? Elderly people? We might be introducing bias if we are going remote. We also cannot observe if vulnerable people are in a safe place to talk if we are doing remote monitoring. So, we might be exposing people to harm or they could be slipping through the cracks. Also, people self-select for phone surveys. Who is not answering the phone and thus left out of the survey?
Consider providing airtime but make sure this doesn’t create perverse incentives.
Ethics and doing no harm are key principles. If we are forced to deliver programs remotely, this involves experimentation. And we are experimenting with people’s lives during a health crisis. Consider including a complaints channel where people can report any issues.
Ensure data is providing value at the local level, and help teams see what the whole data process is and how their data feeds into it. That will help improve data quality and reduce the tendency to ‘tick the box’ for data collection or find workarounds.
Design systems for interoperability so that the data can overlap, and the data can be integrated with other data for better insights or can be automatically updated. Data standards need to be established so that different systems can capture data in the same way or the same format;
Create a well-designed change management program to bring people on board and support them. Role modeling by leaders can help to promote new behaviors.
Further questions to explore:
How can we design monitoring to be remote from the very start? What new gaps could we fill and what kinds of mixed methods could we use?
What two-way platforms are most useful and how can they be used effectively and ethically?
Can we create a simple overview of opportunities and threats of remote monitoring?
How can we collect qualitative data, e.g., focus groups and in-depth interviews?
How can we keep respondents safe? What are the repercussions of asking sensitive questions?
How can we create data continuity plans during the pandemic?
Over the past decade, monitoring, evaluation, research and learning (MERL) practices have become increasingly digitalized. The COVID-19 pandemic has caused that the process of digitalization to happen with even greater speed and urgency, due to travel restrictions, quarantine, and social distancing orders from governments who are desperate to slow the spread of the virus and lessen its impact.
Data is a necessary and critical part of COVID-19 prevention and response efforts to understand where the virus might appear next, who is most at risk, and where resources should be directed for prevention and response. However we need to be sure that we are not putting people at risk of privacy violations or misuse of personal data and to ensure that we are managing that data responsibly so that we don’t unnecessarily create fear or panic.
Watch the video below:
Listen to the audio from the session here:
MERL Practitioners have clear responsibilities when sharing, presenting, consuming and interpreting data. Individuals and institutions may use data to gain prestige, and this can allow bias to creep in or to justify government decisions. Data quality is critical for informing decisions, and information gaps create the risk of misinformation and flawed understanding. We need to embrace uncertainty and the limitations of the science, provide context and definitions so that our sources are clear, and ensure transparency around the numbers and the assumptions that are underpin our work.
MERL Practitioners should provide contextual information and guidance on how to interpret the data so that people can make sense of it in the right way. We should avoid cherry picking data to prove a point, and we should be aware that data visualization carries power to sway opinions and decisions. It can also influence behavior change in individuals, so we need to take responsibility for that. We also need to find ways to visualize data for lay people and non-technical sectors.
Critical data is needed, yet it might be used in negative or harmful ways, for example, COVID-related stigmatization that can affect human dignity. We must not override ethical and legal principles in our rush to collect data. Transparency around data collection processes and use are also needed, as well as data minimization. Some might be taking advantage of the situation to amass large amounts of data for alternative purposes, which is unethical. Large amounts of data also bring increased risk of data breaches. When people are scared, such as in COVID times, they will be willing to hand over data. We need to ensure that we are providing oversight and keeping watch over government entities, health facilities, and third-party data processors to ensure data is protected and not misused.
MERL Practitioners are seeking more guidance and support on: aspects of consent and confidentiality; bias and interference in data collection by governments and community leaders; overcollection of data leading to fatigue; misuse of sensitive data such as location data; potential for re-identification of individuals; data integrity issues; lack of encryption; and some capacity issues.
Good practices and recommendations include ethical clearance of data and data assurance structures; rigorous methods to reduce bias; third party audits of data and data protection processes; localization and contextualization of data processes and interpretation; and “do no harm” framing.
Sometimes we are not clear on why we are collecting data. ‘Just because we can’ is not a valid reason to collect or use data and technology. What purposes are driving our data collection and use of technology? What is the problem we are trying to solve? A lack of specificity can allow us stray into speculative data collection — if we’re collecting data on X, then it’s a good opportunity to collect data on Y “in case we need it in the future”. Do we ever really need it in the future? And if we do go back to it, we often find that because we didn’t collect the data on Y with a specific purpose, it’s not the “right” data for our needs. So, let’s always ask ourselves why are we collecting this data, do we really need it?
Projects are increasingly under pressure to be more efficient and cost-effective in their data collection, yet the need or desire to conduct more robust assessments can requires the collection of data on multiple dimensions within a community. These two dynamics are often in conflict with each other. Here are three questions that can help guide our decision making:
Are there existing data sets that are “good enough” to meet the M&E needs of a project? Often there are, and they are collected regularly enough to be useful. Lean on partners who understand the data space to help map out what exists and what really needs to be collected. Leverage partners who are innovating in the data space – can machine learning and AI-produced data meet 80% of your needs? If so, consider it.
What data are we critically in need of to assess a project? Build an efficient data collection methodology that considers respondent burden and potentially includes multiple channels for receiving responses to increase inclusivity.
What will the data be used for? Sensitive contexts and life or death decisions require a different level of specificity and periodicity than less sensitive projects. Think about data from this lens when deciding which information to collect, how often to collect it, and who to collect it from.
It is worth exploring questions of access in our data collection practices. Who has access to the data and the technology? Do the people about whom the data is, have access to it? Have we considered the harms that could come from the collection, storage, and use of data? For instance, while it can be useful to know where all the clients are who are accessing a pregnancy clinic to design better services, an unintended consequence may involve others having the ability to identify people who are pregnant, which pregnant people might not like these others to know. What can we do to protect the privacy of vulnerable populations? Also, going digital can be helpful, but if a person or community implicated in a data collection endeavour does not have access to technology or to a charging point – are we not just increasing or reinforcing inequality?
While we often advocate for transparency in many parts of our industry, we are not always transparent about our data practices. Are we willing to tell others, to tell community members, why we are collecting data, using technology, and how we are using information? If we are clear on our purpose, but not willing for it to be transparent, then it might be a good reason to reconsider. Yet, transparency does not equate accountability, so what are the mechanisms for ensuring greater accountability towards the people and communities we seek to serve?
Power and patience
One of the issues we’re facing is power imbalances. The demands that are made of us from donors about data, and the technology solutions that are presented to us, all make us feel like we’re not in control. But the rules haven’t been written yet — we get to write them.
One of the lessons from the responsible data workshop leading up to the conference was that organisations can get out in front of demands for data by developing their own data management and privacy policies. From this position it is easier to enter into dialogues and negotiations, with the organisational policy as your backstop. Therefore, it is worth asking, Who has power? For what? Where does it reside and how can we rebalance it?
Literacy underpins much of this – linguistic, digital, identity, ethical literacy. Often when it comes to ‘digital’ we immediately fall under the spell of the tyranny of the urgent. Therefore, in what ways can we adopt a more ‘patient’ or ‘reflective’ practice with respect to digital?
MERL and development practitioners have long wrestled with complex ethical, regulatory, and technical aspects of adopting new data approaches and technologies. The topic of responsible data has gained traction over the past 5 years or so, and a handful of early adopters have developed and begun to operationalize institutional RD policies. Translating policy into practical action, however, can feel daunting to organizations. Constrained budgets, complex internal bureaucracies, and ever-evolving technology and regulatory landscapes make it hard to even know where to start.
We don’t think organizations should do that anyway, given that each organization’s context and operating approach is different, and policy means nothing if it’s not rolled out through actual practice and behavior change!
In September, we hosted a MERL Tech pre-workshop on Operationalizing Responsible Data to discuss and share different ways of turning responsible data policy into practice. Below we’ve summarized some tips shared at the workshop. RD champions in organizations of any size can consider these when developing and implementing RD policy.
1. Understand Your Context & Extend Empathy
Before developing policy, conduct a non-punitive assessment (a.k.a. a landscape assessment, self-assessment or staff research process) on existing data practices, norms, and decision-making structures . This should engage everyone who will using or affected by the new policies and practices. Help everyone relax and feel comfortable sharing how they’ve been managing data up to now so that the organization can then improve. (Hint: avoid the term ‘audit’ which makes everyone nervous.)
Create ‘safe space’ to share and learn through the assessment process:
Allow staff to speak anonymously about their challenges and concerns whenever possible
Highlight and reinforce promising existing practices
Involve people in a ‘self-assessment’
Use participatory workshops (e.g. work with a team to map a project’s data flows or conduct a Privacy Impact Assessment or a Risk-Benefits Assessment) – this allows everyone who participates to gain RD awareness while also learning new practical tools along with highlighting any areas that need attention. The workshop lead or “RD champion” can also then get a better sense of the wider organizations knowledge, attitudes and practices as related to RD
Acknowledge (and encourages institutional leaders to affirm) that most staff don’t have “RD expert” written into their JDs; reinforce that staff will not be ‘graded’ or evaluated on skills they weren’t hired for.
Identify organizational stakeholders likely to shape, implement, or own aspects of RD policy and tailor your engagement strategies to their perspectives, motivations, and concerns. Some may feel motivated financially (avoiding fines or the cost of a data breach); others may be motivated by human rights or ethics; whereas some others might be most concerned with RD with respect to reputation, trust, funding and PR.
Map organizational policies, major processes (like procurement, due diligence, grants management), and decision making structures to assess how RD policy can be integrated into these existing activities.
2. Consider Alternative Models to Develop RD Policy
There is no ‘one size fits all’ approach to developing RD policy. As the (still small, but promising) number of organizations adopting policy grows, different approaches are emerging. Here are some that we’ve seen:
Top-down: An institutional-level policy is developed, normally at the request of someone on the leadership team/senior management. It is then adapted and applied across projects, offices, etc.
Works best when there is strong leadership buy-in for RD policy and a focal point (e.g. an ‘Executive Sponsor’) coordinating policy formation and navigating stakeholders
Bottom-up: A group of staff are concerned about RD but do not have support or interest from senior leadership, so they ‘self-start’ the learning process and begin shaping their own practices, joining together, meeting, and communicating regularly until they have wider buy-in and can approach leadership with a use case and budget request for an organization-wide approach.
Good option if there is little buy-in at the top and you need to build a case for why RD matters.
Project- or Team-Generated: Development and application of RD policies are piloted within a targeted project or projects or on one team. Based on this smaller slice of the organization, the project or team documents its challenges, process, and lessons learned to build momentum for and inform the development of future organization-wide policy.
Promising option when organizational awareness and buy-in for RD is still nascent and/or resources to support RD policy formation and adoption (staff, financial, etc.) are limited.
Hybrid approach: Organizational policy/policies are developed through pilot testing across a reasonably-representative sample of projects or contexts. For example, an organization with diverse programmatic and geographical scope develops and pilots policies in a select set of country offices that can offer different learning and experiences; e.g., a humanitarian-focused setting, a development-focused setting, and a mixed setting; a small office, medium sized office and large office; 3-4 offices in different regions; offices that are funded in various ways; etc.
Promising option when an organization is highly decentralized and works across a diverse country contexts and settings. Supports the development of approaches that are relevant and responsive to diverse capacities and data contexts.
3. Couple Policy with Practical Tools, and Pilot Tools Early and Often
In order to translate policy into action, couple it with practical tools that support existing organizational practices.
Make sure tools and processes empower staff to make decisions and relate clearly to policy standards or components; for example:
If the RD policy includes a high-level standard such as, “We ensure that our partnerships with technology companies align with our RD values,” give staff tools and guidance to assess that alignment.
When developing tools and processes, involve target users early and iteratively. Don’t worry if draft tools aren’t perfectly formatted. Design with users to ensure tools are actually useful before you sink time into tools that will sit on a shelf at best, and confuse or overburden staff at worst.
4. Integrate and “Right-Size” Solutions
As RD champions, it can be tempting to approach RD policy in a silo, forgetting it is one of many organizational priorities. Be careful to integrate RD into existing processes, align RD with decision-making structures and internal culture, and do not place unrealistic burdens on staff.
When building tools and processes, work with stakeholders to develop responsibility assignment charts (e.g. RACI, MOCHA) and determine decision makers.
When developing responsibility matrices, estimate the hours each stakeholder (including partners, vendors, and grantees) will dedicate to a particular tool or process. Work with anticipated end users to ensure that processes:
Can realistically be carried out within a normal workload
Will not excessively burden staff and partners
Are realistically proportionate to the size, complexity, and risk involved in a particular investment or project
5. Bridge Policy and Behavior Change through Accompaniment & Capacity Building
Integrating RD policy and practices requires behavior change and can feel technically intimidating to staff. Remember to reassure staff that no one (not even the best resourced technology firms!), has responsible data mastered, and that perfection is not the goal.
In order to feel confident using new tools and approaches to make decisions, staff need knowledge to analyze information. Skills and knowledge required will be different according to role, so training should be adapted accordingly. While IT staff may need to know the ins and outs of network security, general program officers certainly do not.
Accompany staff as they integrate RD processes into their work. Walk alongside them, answering questions along the way, but more importantly, helping staff build confidence to develop their own internal RD compass. That way the pool of RD champions will grow!
What approaches have you seen work in your organization?
by Alvaro Cobo-Santillan, Catholic Relief Services (CRS); Jeff Lundberg, CRS; Paul Perrin, University of Notre Dame; and Gillian Kerr, LogicalOutcomes Canada.
In the year 2017, with all of us holding a mini-computer at all hours of the day and night, it’s probably not too hard to imagine that “A teenager in Africa today has access to more information than the President of United States had 15 years ago”. So it also stands to reason that the ability to appropriately and ethically grapple with the use of that immense amount information has grown proportionately.
What do we mean when we say that the world of development—particularly evaluation—data is murky? A major factor in this sentiment is the ambiguous polarity between research and evaluation data.
“Research seeks to prove; evaluation seeks to improve.” – CDC
“Research studies involving human subjects require IRB review. Evaluative studies and activities do not.”
This has led to debates as to the actual relationship between research and evaluation. Some see them as related, but separate activities, others see evaluation as a subset of research, and still others might posit that research is a specific case of evaluation.
But regardless, though motivations of the two may differ, research and evaluation look the same due to their stakeholders, participants, and methods.
If that statement is true, then we must hold both to similar protections!
What are some ways to make the waters less murky?
Deeper commitment to informed consent
Reasoned use of identifiers
Need to know vs. nice to know
Data security and privacy protocols
Data use agreements and protocols for outside parties
Revisit NGO primary and secondary data IRB requirements
Alright then, what can we practically do within our individual agencies to move the needle on data protection?
In short, governance. Responsible data is absolutely a crosscutting responsibility, but can be primarily championed through close partnerships between the M&E and IT Departments
Think about ways to increase usage of digital M&E – this can ease the implementation of R&D
Can existing agency processes and resources be leveraged?
Plan and expect to implement gradual behavior change and capacity building as a pre-requisite for a sustainable implementation of responsible data protections
Think in an iterative approach. Gradually introduce guidelines, tools and training materials
Plan for business and technical support structures to support protections
Is anyone doing any of the practical things you’ve mentioned?
Yes! Gillian Kerr from LogicalOutcomes spoke about highlights from an M&E system her company is launching to provide examples of the type of privacy and security protections they are doing in practice.
As a basis for the mindset behind their work, she notably presented a pretty fascinating and simple comparison of high risk vs. low risk personal information – year of birth, gender, and 3 digit zip code is unique for .04% of US residents, but if we instead include a 5 digit zip code over 50% of US residents could be uniquely identified. Yikes.
In that vein, they are not collecting names or identification and only year of birth (not month or day) and seek for minimal sensitive data defining data elements by level of risk to the client (i.e. city of residence – low, glucose level – medium, and HIV status – high).
In addition, asking for permission not only in the original agency permission form, but also in each survey. Their technical system maintains two instances – one containing individual level personal information with tight permission even for administrators and another with aggregated data with small cell sizes. Other security measures such as multi-factor authentication, encryption, and critical governance; such as regular audits are also in place.
It goes without saying that we collectively have ethical responsibilities to protect personal information about vulnerable people – here are final takeaways:
If you can’t protect sensitive information, don’t collect it.
If you can’t keep up with current security practices, outsource your M&E systems to someone who can.
Your technology roadmap should aspire to give control of personal information to the people who provide it (a substantial undertaking).
In the meantime, be more transparent about how data is being stored and shared
For this year’s MERL Tech DC, we teamed up to do a session on Responsible Data. Based on feedback from last year, we knew that people wanted less discussion on why ethics, privacy and security are important, and more concrete tools, tips and templates. Though it’s difficult to offer specific do’s and don’ts, since each situation and context needs individualized analysis, we were able to share a lot of the resources that we know are out there.
To kick off the session, we quickly explained what we meant by Responsible Data. Then we handed out some cards from Oxfam’s Responsible Data game and asked people to discuss their thoughts in pairs. Some of the statements that came up for discussion included:
Being responsible means we can’t openly share data – we have to protect it
We shouldn’t tell people they can withdraw consent for us to use their data when in reality we have no way of doing what they ask
Biometrics are a good way of verifying who people are and reducing fraud
Following the card game we asked people to gather around 4 tables with a die and a print out of the data lifecycle where each phase corresponded to a number (Planning = 1, collecting = 2, storage = 3, and so on…). Each rolled the die and, based on their number, told a “data story” of an experience, concern or data failure related to that phase of the lifecycle. Then the group discussed the stories.
For our last activity, each of us took a specific pack of tools, templates and tips and rotated around the 4 tables to share experiences and discuss practical ways to move towards stronger responsible data practices.
Responsible data policy, practices and evaluation of their roll-out
Oxfam released its Responsible Program Data Policy in 2015. Since then, they have carried out six pilots to explore how to implement the policy in a variety of countries and contexts. Emily shared information on these these pilots and the results of research carried out by the Engine Room called Responsible Data at Oxfam: Translating Oxfam’s Responsible Data Policy into practice, two years on. The report concluded that the staff that have engaged with Oxfam’s Responsible Data Policy find it both practically relevant and important. One of the recommendations of this research showed that Oxfam needed to increase uptake amongst staff and provide an introductory guide to the area of responsible data.
In response, Oxfam created the Responsible Data Management pack, (available in English, Spanish, French and Arabic), which included the game that was played in today’s session along with other tools and templates. The card game introduces some of the key themes and tensions inherent in making responsible data decisions. The examples on the cards are derived from real experiences at Oxfam and elsewhere, and they aim to generate discussion and debate. Oxfam’s training pack also includes other tools, such as advice on taking photos, a data planning template, a poster of the data lifecycle and general information on how to use the training pack. Emily’s session also encouraged discussion with participants about governance and accountability issues like who in the organisation manages responsible data and how to make responsible data decisions when each context may require a different action.
Nina shared early results of four case studies mSTAR is conducting together with Sonjara for USAID. The case studies are testing a draft set of responsible data guidelines, determining whether they are adequate for ‘on the ground’ situations and if projects find them relevant, useful and usable. The guidelines were designed collaboratively, based on a thorough review and synthesis of responsible data practices and policies of USAID and other international development and humanitarian organizations. To conduct the case studies, Sonjara, Nina and other researchers visited four programs which are collecting large amounts of potentially sensitive data in Nigeria, Kenya and Uganda. The researchers interviewed a broad range of stakeholders and looked at how the programs use, store, and manage personally identifiable data (PII). Based on the research findings, adjustments are being made to the guidelines. It is anticipated that they will be published in October.
Linda mentioned that a literature review of responsible data policy and practice has been done as part of the above mentioned mSTAR project (which she also worked on). The literature review will provide additional resources and analysis, including an overview of the core elements that should be included in organizational data guidelines, an overview of USAID policy and regulations, emerging legal frameworks such as the EU’s General Data Protection Regulation (GDPR), and good practice on how to develop guidelines in ways that enhance uptake and use. The hope is that both the Responsible Data Literature Review and the of Responsible Data Guidelines will be suitable for adopting and adapting by other organizations. The guidelines will offer a set of critical questions and orientation, but that ethical and responsible data practices will always be context specific and cannot be a “check-box” exercise given the complexity of all the elements that combine in each situation.
Check out this responsible data resource list, which includes additional tools, tips and templates. It was developed for MERL Tech London in February 2017 and we continue to add to it as new documents and resources come out. After a few years of advocating for ‘responsible data’ at MERL Tech to less-than-crowded sessions, we were really excited to have a packed room and high levels of interest this year!