Using analytics as a validation tool: rethinking quality and reliability of data collection

Rebecca Rumbul, the Head of Research at My Society, gave a Lightning Talk at MERL Tech London in which she described the potential for using Google Analytics as a tool for informing and validating research.

First, she explained her organization’s work. Broadly speaking, My Society is a non-profit social enterprise with a mission to invent and popularize digital tools that enable citizens to exert power over institutions and decision makers. She noted that her organization exists solely online, and that as a result it gathers a significant amount of data from their software’s users in the 44 countries where they operate.

My Society is currently using this data to research and examine whether it is worth continuing to pursue civic technology. To do this, they are taking rational and measured approaches designed to help them evaluate and compare their products and to see to what extent they have valuable real world effects.

One tool that Rebecca’s organization makes extensive use of is Google Analytics. Google Analytics allows My Society’s research team to see who is using their software, where they are from, if they are returning users or new ones, and the number of sessions happening at one time. Beyond this, it also provides basic demographic information. Basically, Google Analytics alone gives them ample data to work with.

One application of this data is to take trends that emerge and use them to frame new research questions. For example, if more women than men are searching for a particular topic on a given day, this phenomenon could merit further exploration.

Additionally, it can act as a validation tool. For example, if the team wants to conduct a new survey, Google Analytics provides a set of data that can complement the results from that survey. It enables one to cross-check the survey results with Google’s data to determine the extent to which the survey results may or may not have suffered from errors like self-selection bias. With it, one can develop a better sense on whether there are issues with the research or if the data can be relied upon.

Google Analytics, despite having its flaws, enables one to think more deeply about their data, have frank discussions and frame research questions. All of this is can be very valuable to evaluation efforts in the development sector.

For more, see Rebecca’s Lightning Talk below!