Tag Archives: danielle de garcia

Qualitative Coding: From Low Tech to High Tech Options

by Daniel Ramirez-Raftree, MERL Tech volunteer

In their MERL Tech DC session on qualitative coding, Charles Guedenet and Anne Laesecke from IREX together with Danielle de Garcia of Social Impact offered an introduction to the qualitative coding process followed by a hands-on demonstration on using Excel and Dedoose for coding and analyzing text.

They began by defining content analysis as any effort to make sense of qualitative data that takes a volume of qualitative material and attempts to identify core consistencies and meanings. More concretely, it is a research method that uses a set of procedures to make valid inferences from text. They also shared their thoughts on what makes for a good qualitative coding method.

Their belief is that: it should

  • consider what is already known about the topic being explored
  • be logically grounded in this existing knowledge
  • use existing knowledge as a basis for looking for evidence in the text being analyzed

With this definition laid out, they moved to a discussion about the coding process where they elaborated on four general steps:

  1. develop codes and a codebook
  2. decide on a sampling plan
  3. code your data
  4. go back and do it again!
  5. test for reliability

Developing codes and a codebook is important for establishing consistency in the coding process, especially if there will be multiple coders working on the data. A good way to start developing these codes is to consider what is already known. For example, you can think about literature that exists on the subject you’re studying. Alternatively, you can simply turn to the research questions the project seeks to answer and use them as a guide for creating your codes. Beyond this, it is also useful to go through the content and think about what you notice as you read. Once a codebook is created, it will lend stability and some measure of objectivity to the project.

The next important issue is the question of sampling. When determining sample size, though a larger sample will yield more robust results, one must of course consider the practical constraints of time, cost and effort. Does the benefit of higher quality results justify the additional investment? Fortunately, the type of data will often inform sampling. For example, if there is a huge volume of data, it may be impossible to analyze it all, but it would be prudent to sample at least 30% of it. On the other hand, usually interview and focus group data will all be analyzed, because otherwise the effort of obtaining the data would have gone to waste.

Regarding sampling method, session leads highlighted two strategies that produce sound results. One is systematic random sampling and the other is quota sampling–a method employed to ensure that the proportions of demographic group data are fairly represented.

Once these key decisions have been made, the actual coding can begin. Here, all coders should work from the same codebook and apply the codes to the same unit of analysis. Typical units of analysis are: single words, themes, sentences, paragraphs, and items (such as articles, images, books, or programs). Consistency is essential. A way to test the level of consistency is to have a 10% overlap in the content each coder analyzes and aim for 80% agreement between their coding of that content. If the coders are not applying the same codes to the same units this could either mean that they are not trained properly or that the code book needs to be altered.

Along a similar vein, the fourth step in the coding process is to test for reliability. Challenges in producing stable and consistent results in coding could include: using a unit of analysis that is too large for a simple code to be reliably applied, coding themes or concepts that are ambiguous, and coding nonverbal items. For each of these, the central problem is that the units of analysis leave too much room for subjective interpretation that can introduce bias. Having a detailed codebook can help to mitigate against this.

After giving an overview of the coding process, the session leads suggested a few possible strategies for data visualization. One is to use a word tree, which helps one look at the context in which a word appears. Another is a bubble chart, which is useful if one has descriptive data and demographic information. Thirdly, correlation maps are good for showing what sorts of relationships exist among the data. The leads suggested visiting the website stephanieevergreen.com/blog for more ideas about data visualization.

Finally, the leads covered low-tech and high-tech options for coding. On the low-tech end of the spectrum, paper and pen get the job done. They are useful when there are few data sources to analyze, when the coding is simple, and when there is limited tech literacy among the coders. Next up the scale is Excel, which works when there are few data sources and when the coders are familiar with Excel. Then the session leads closed their presentation with a demonstration of Dedoose, which is a qualitative coding tool with advanced capabilities like the capacity to code audio and video files and specialized visualization tools. In addition to Dedoose, the presenters mentioned Nvivo and Atlas as other available qualitative coding software.

Despite the range of qualitative content available for analysis, there are a few core principles that can help ensure that it is analyzed well, these include consistency and disciplined methodology. And if qualitative coding will be an ongoing part of your organization’s operations, there are several options for specialized software that are available for you to explore. [Click here for links and additional resources from the session.]

Visualizing what connects us: Social Network Analysis (SNA) in M&E

by Anne Laesecke (IREX) and Danielle de García (Social Impact). This post also appears on the Social Impact blog and  the IREX blog.

SNA, or Social Network Analysis, continues to gain momentum in the M&E space. This year at MERL Tech, we held an SNA 101 session, giving a quick-and-dirty overview of what it is, how it can contribute to M&E, and useful tips and tools for conducting an SNA. If you missed it, here’s what you need to know:

What is SNA?

SNA is a way to analyze social systems through relationships. Analyzing and visualizing networks can reveal critical insights for understanding relationships between organizations, supply chains; social movements; and/or between individuals. It’s a very versatile tool which can be used throughout the program cycle to measure things like trust and social capital, information flows, resources, collaboration, and disease spread, among other things.

SNA uses a different vocabulary than other types of analyses. For example, the relationships we measure are called ties or links, and the entities that make up a network are called nodes or actors. These can be organizations, people, or even whole networks themselves. We can study nodes more closely by looking at their attributes – things that characterize them (like demographic information), and we can learn more about how nodes interact and cluster by studying communities or modalities within networks. Various measures of the roles nodes play in a network, as well as measures that characterize the networks themselves, can reveal a lot about the systems and hidden relationships at play. For example, we can determine who has the most ties with other actors; who is relatively cut off from the network, or who is connected to the most well-connected actors.

Why would you use SNA for M&E?

The term “social network analysis” often triggers associations with social media, but SNA uses data from a variety of platforms (including but not limited to social media!). For instance, SNA can identify key influencers in systems – important for programs that rely on thinking politically. SNA can also be a useful tool in processing big data with applications for cybersecurity as well as creating biological and epidemiological projections. Beyond looking at networks of individuals, SNA can explore relationships with concepts through analysis of qualitative data and concept mapping. It can also look at organizational risks and processes (think about comparing an organizational chart with who people actually go to within an organization for information).

How do you do SNA?

Conducting SNA mostly follows the same procedure as other analysis.

  1. Determine your purpose and questions. What decisions do you need to make based on the data you get? Who is your audience and what do they need to know? Answering these questions can help you decided what you are trying to measure and how.
  2. Collect your data. SNA can incorporate lots of different data forms, including customized surveys and interviews asking actors in your network about the links they have, external data such as census information or other public records to further inform attributes or triangulate your custom data; and mapping of key locations or concepts. One thing to consider while conducting an SNA – data cleaning is usually a heavier lift than for other types of analysis.
  3. Crunch those numbers. SNA uses matrices to calculate various measures – from types of centrality to network density and beyond. Lucky for us, there are plenty of tools that take on both the analysis and visualization portions of SNA. However, another consideration as you analyze your data is that network data is often not generalizable in the same way as some statistical analysis. If you miss a key node in the network, you may miss an entire portion that is only linked through that node.
  4. Visualize the network. Network visualizations are one of the most distinctive features of SNA and can be incredibly useful as tools to engage partners about your findings. There is a wealth of analysis and visualization tools that can help you do this. We created a worksheet that outlines several, but a few of the most popular are UCINet, Gephi, and NodeXL.
  5. Interpret your results. You now have a beautiful graph that shows what nodes are important in your network. So what? How does it relate to your program? Your interpretation should answer the questions around the purpose of your analysis, but beyond interpretation can serve to improve your programming. Often, SNA results can help make projections for program sustainability based on who key players are and who can continue championing work, or projecting where trends seem to be going and anticipating activities around those areas.

Conclusions and resources

We barely scratched the surface of what SNA can do and there are so many more applications! Some great resources to learn more are the SNA TIG of the American Evaluation Association, Stephen Borgatti’s course website on SNA, and a site of his dedicated completely to designing surveys for SNA.