As a social impact company that builds civic technology, CitizenLab recently took some important steps in how we evaluate our impact. The impact measurement framework that we designed can be used by other civic tech initiatives too, so today we’re sharing it here to help amplify impact collectively.
Measuring impact in civic tech, it’s not a straight-forward job. In 2018, Matt Stempeck from Civic Hall wrote about the 10 common problems with impact measurement in civic tech. To name just a few problems, we’re all using different metrics, sharing is irregular, purely quantitative metrics can miss objectives, and case studies are often too biased.
At CitizenLab, we’ve also been struggling for a long time to find meaningful metrics beyond how many community members we have engaged. Because no one seems to have cracked this before, or at least not shared it publicly, we had to find our own way. After a long process of reflection and ideation, we’ve finally come to a framework we’re proud of – and we’re happy to share it with the world.
Introducing our Theory of Change
To proactively plan for issues that might arise in using the framework, we put the following principles at the heart of the framework to guide us. In using the framework, we aim to:
1. Build on the Sustainable Development Goals (SDGs), the best universal standards out there, for our Theory of Change. The SDGs, as developed by the United Nations, are a global compass for sustainable development but are also meaningful when it comes to governance. We found that SDG 16.7 to “ensure responsive, inclusive, participatory and representative decision-making at all levels” was exactly the type of impact we’re trying to create at CitizenLab. We further broke it into individual, longer-term impact milestones, which can also be broken down into multiple, near-term outcomes.
2. Look beyond quantity, and also measure the quality of participation. Civic tech initiatives can’t stop at bragging about the number of citizens who participated in a certain project. While that is a key impact indicator – one that we use widely, too – only optimizing for reach or engagement is not enough. Metrics that look at the number of participants tell us how many people have had a say, but don’t tell us anything about the influence they had on the actual decision. Therefore, we concluded that we should complement the quantity metrics, at whatever price, with quality metrics. This is a time commitment that currently takes manual annotation and assessments to come to good quality indices, but we find it to be worthwhile in measuring our impact.
3. Track progress continuously and use it as our company’s compass. Last year, we started an annual impact measurement initiative to survey both community members and governments who use our platform, resulting in an annual impact report. While we will continue this initiative to capture in-depth, qualitative feedback, we can’t stop at that. To get more insights that can provide us with enough direction to guide our company and maximize impact on a more frequent basis, we’ve started measuring impact at the outcomes level on a monthly basis to continuously track progress.
With these principles in mind, we came to the following impact measurement framework:
By looking at public decision-making as an integrated process, we break it into three blocks: input, process/throughput, and output. These blocks reflect our mission to make public decision-making more inclusive, participatory, and responsive. So what do these blocks mean for how we look at outcomes and impact?
Impact 1: more inclusive decision-making
Outcome 1.1: More participants in the process. The most obvious indicator, and also the easiest one to measure. We also add depth by looking at who was Aware v. Informed v. Engaged.
Outcome 1.2: More representative group of participants.
Measuring the diversity of participants is not an easy job. Age and gender can be a bit more straight-forward to gather, but going beyond those personal identity metrics can be sensitive. Whether an initiative aims to include more community members from different ethnicities, religions, socio-economic status, etc. we always want to be conscientious of the fact that these are personal data points and should not always be gathered, despite our best intentions to design for optimal inclusion.
Outcome 1.3: Higher engagement among participants. We don’t just want to know whether more people make their voice heard because of our platform, we also want to know whether the engagement is deeper, also referred to as “thick engagement”.
Impact 2: More participatory decision-making
“What’s the quality of the participatory process?”
Outcome 1.1: Higher quality of input. Each month, we take a sample of 100 new inputs from our client’s platforms to evaluate these each against a set of quality indicators:
- Relevance: Does the input respond to the project prompt?
- Argumentation: Do participants provide supporting arguments?
- Specificity: Do they provide specifics of how it should be done?
- Language: Is the language productive and oriented toward positive change?
Outcome 1.2: More participatory agenda-setting. An open democracy is characterized by the possibility for the wider public to influence the actual political agenda, not just participate in the agenda being set. Therefore, we also look at how many proposals get successfully launched from the bottom-up.
Outcome 1.3: Higher process quality. Similar to how we approach input assessment, we evaluate all new projects on a monthly basis across a set of quality indicators:
- Timeline: The process has a clear start and end.
- Time and resources: The project has a dedicated project manager for follow-up.
- Stake: The project’s issue area is of high interest to participants.
- Responsiveness: Feedback is built into the participatory process.
- Transparency: Enough information is given to participate effectively. How input is used in the process, and what the value-add of the participatory process looks like is explained.
- Outreach: Communication efforts are in place and residents are incentivized to participate.
- Commitment: The public officials are committed to following up on what’s happening in the process and taking the input gathered into account for actual decision-making.
Impact 3: More responsive decision-making
“To what extent did participants influence the actual decision?”
Outcome 3.1: Better feedback after participation. Feedback is essential to affirming that one’s voice is heard. We measure what percentage of input has received feedback within three months of submission, and whether the project’s final decisions and outcomes were clearly communicated to participants.
Outcome 3.2: More officials listening to their community. Internal adoption is a good indicator of how many public officials are actually listening to the opinions of their community.
Outcome 3.3 More efficient input processing. Lastly, with the help of Natural Language Processing(NLP) technology the time needed to process and summarize input is drastically reduced, freeing up more time for in-depth analysis and discussion.
The impact you can create
We hope this framework has inspired you to think about how you measure your own impact. While monitoring and evaluation are constantly evolving, including at CitizenLab, we’re hopeful that by placing a focus on social impact we can create more positive change, together. Don’t hesitate to get in touch should you have any suggestions or questions.
