CitizenLab's Blog
  • Platform
  • Services
  • Resources
    • Blog
    • Guides
  • About
  • Contact
  • Schedule a demo
  • English
    • Français
    • Nederlands
    • Español
  • Home
  • >
  • 🏛️ Blog
  • >
  • Measuring impact: our theory of change at CitizenLab

Measuring impact: our theory of change at CitizenLab

By Wietse Van Ransbeeck Civic Engagement, Civic Engagement 25/05/2021
CitizenLab's Blog
  • Platform
  • Services
  • Resources
    • Blog
    • Guides
  • About
  • Contact
  • Schedule a demo
  • English
    • Français
    • Nederlands
    • Español
Graphs and data

As a social impact company that builds civic technology, CitizenLab recently took some important steps in how we evaluate our impact. The impact measurement framework that we designed can be used by other civic tech initiatives too, so today we’re sharing it here to help amplify impact collectively. 

Measuring impact in civic tech, it’s not a straight-forward job. In 2018, Matt Stempeck from Civic Hall wrote about the 10 common problems with impact measurement in civic tech. To name just a few problems, we’re all using different metrics, sharing is irregular, purely quantitative metrics can miss objectives, and case studies are often too biased.

At CitizenLab, we’ve also been struggling for a long time to find meaningful metrics beyond how many community members we have engaged. Because no one seems to have cracked this before, or at least not shared it publicly, we had to find our own way. After a long process of reflection and ideation, we’ve finally come to a framework we’re proud of – and we’re happy to share it with the world.

Introducing our Theory of Change

To proactively plan for issues that might arise in using the framework, we put the following principles at the heart of the framework to guide us. In using the framework, we aim to:

1. Build on the Sustainable Development Goals (SDGs), the best universal standards out there, for our Theory of Change. The SDGs, as developed by the United Nations, are a global compass for sustainable development but are also meaningful when it comes to governance. We found that SDG 16.7 to “ensure responsive, inclusive, participatory and representative decision-making at all levels” was exactly the type of impact we’re trying to create at CitizenLab. We further broke it into individual, longer-term impact milestones, which can also be broken down into multiple, near-term outcomes.

2. Look beyond quantity, and also measure the quality of participation. Civic tech initiatives can’t stop at bragging about the number of citizens who participated in a certain project. While that is a key impact indicator – one that we use widely, too – only optimizing for reach or engagement is not enough. Metrics that look at the number of participants tell us how many people have had a say, but don’t tell us anything about the influence they had on the actual decision. Therefore, we concluded that we should complement the quantity metrics, at whatever price, with quality metrics. This is a time commitment that currently takes manual annotation and assessments to come to good quality indices, but we find it to be worthwhile in measuring our impact.

3. Track progress continuously and use it as our company’s compass. Last year, we started an annual impact measurement initiative to survey both community members and governments who use our platform, resulting in an annual impact report. While we will continue this initiative to capture in-depth, qualitative feedback, we can’t stop at that. To get more insights that can provide us with enough direction to guide our company and maximize impact on a more frequent basis, we’ve started measuring impact at the outcomes level on a monthly basis to continuously track progress.

With these principles in mind, we came to the following impact measurement framework:

By looking at public decision-making as an integrated process, we break it into three blocks: input, process/throughput, and output. These blocks reflect our mission to make public decision-making more inclusive, participatory, and responsive. So what do these blocks mean for how we look at outcomes and impact?


Impact 1: more inclusive decision-making

 

Outcome 1.1: More participants in the process. The most obvious indicator, and also the easiest one to measure. We also add depth by looking at who was Aware v. Informed v. Engaged.

Outcome 1.2: More representative group of participants.
Measuring the diversity of participants is not an easy job. Age and gender can be a bit more straight-forward to gather, but going beyond those personal identity metrics can be sensitive. Whether an initiative aims to include more community members from different ethnicities, religions, socio-economic status, etc. we always want to be conscientious of the fact that these are personal data points and should not always be gathered, despite our best intentions to design for optimal inclusion.

Outcome 1.3: Higher engagement among participants. We don’t just want to know whether more people make their voice heard because of our platform, we also want to know whether the engagement is deeper, also referred to as “thick engagement”.

 

on off switches


Impact 2: More participatory decision-making


“What’s the quality of the participatory process?”

 

Outcome 1.1: Higher quality of input. Each month, we take a sample of 100 new inputs from our client’s platforms to evaluate these each against a set of quality indicators:

  • Relevance: Does the input respond to the project prompt?
  • Argumentation: Do participants provide supporting arguments?
  • Specificity: Do they provide specifics of how it should be done?
  • Language: Is the language productive and oriented toward positive change? 

Outcome 1.2: More participatory agenda-setting. An open democracy is characterized by the possibility for the wider public to influence the actual political agenda, not just participate in the agenda being set. Therefore, we also look at how many proposals get successfully launched from the bottom-up.

Outcome 1.3: Higher process quality. Similar to how we approach input assessment, we evaluate all new projects on a monthly basis across a set of quality indicators:

  • Timeline: The process has a clear start and end.
  • Time and resources: The project has a dedicated project manager for follow-up.
  • Stake: The project’s issue area is of high interest to participants.
  • Responsiveness: Feedback is built into the participatory process.
  • Transparency: Enough information is given to participate effectively. How input is used in the process, and what the value-add of the participatory process looks like is explained.
  • Outreach: Communication efforts are in place and residents are incentivized to participate.
  • Commitment: The public officials are committed to following up on what’s happening in the process and taking the input gathered into account for actual decision-making.
 


Impact 3: More responsive decision-making

“To what extent did participants influence the actual decision?”

Outcome 3.1: Better feedback after participation. Feedback is essential to affirming that one’s voice is heard. We measure what percentage of input has received feedback within three months of submission, and whether the project’s final decisions and outcomes were clearly communicated to participants.

Outcome 3.2: More officials listening to their community. Internal adoption is a good indicator of how many public officials are actually listening to the opinions of their community.

Outcome 3.3 More efficient input processing. Lastly, with the help of Natural Language Processing(NLP) technology the time needed to process and summarize input is drastically reduced, freeing up more time for in-depth analysis and discussion. 

 

The impact you can create

We hope this framework has inspired you to think about how you measure your own impact. While monitoring and evaluation are constantly evolving, including at CitizenLab, we’re hopeful that by placing a focus on social impact we can create more positive change, together. Don’t hesitate to get in touch should you have any suggestions or questions.

 

measuring impact in civic tech guide

You May Also Like

CitizenLab Team USA
Civic Engagement 16/06/2022
Introducing CitizenLab’s Team USA
AI for community engagement
Civic Engagement 15/06/2022
How to leverage AI in community engagement
Civic Engagement 14/06/2022
6 tips for multilingual community engagement
Civic Engagement 14/06/2022
What is placemaking and what are examples of it?
Creative community engagement
Civic Engagement 14/06/2022
6 examples of creative ways to engage your community
Written By
Wietse Van Ransbeeck

Founder of CitizenLab. On a mission to shape local democracies for the digital age. Proud Forbes 30 Under 30 Europe & YTILI Fellow. Talk to me about #Brussels, #Europe, civic tech, philosophy, public policy on Twitter @WietseVR.

Solutions
  • All solutions
  • Strategy & Budgeting
  • Planning & Public Spaces
  • Environment & Sustainability
  • Mobility & Infrastructure
  • Neighborhoods & Community Development
Platform
  • Platform features
  • Services
  • Plans
  • Schedule a demo
Academy
  • Resources
  • Blog
Company
  • About
  • Jobs
  • Partners
Get in touch
  • +44 79 2676 4670
  • +1 530 270 3403
  • Contact form
Blog
Why a participatory budget helps you reach a more diverse audience
How Wichita used public participation to address affordable housing issues
Introducing CitizenLab’s Team USA
© 2022 CitizenLab: Community Engagement Platform. All rights reserved.
By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
_ga2 yearsThe _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_gid1 dayInstalled by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
cookielawinfo-checkbox-advertisement1 yearSet by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
CookieLawInfoConsent1 yearRecords the default button state of the corresponding category & the status of CCPA. It works only in coordination with the primary cookie.
geot_rocket_citysessionNo description available.
geot_rocket_countrysessionNo description available.
geot_rocket_statesessionNo description available.
PHPSESSIDsessionThis cookie is native to PHP applications. The cookie is used to store and identify a users' unique session ID for the purpose of managing user session on the website. The cookie is a session cookies and is deleted when all the browser windows are closed.
STYXKEY_geot_countrysessionNo description
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
CookieDurationDescription
bcookie2 yearsLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser ID.
langsessionLinkedIn sets this cookie to remember a user's language setting.
lidc1 dayLinkedIn sets the lidc cookie to facilitate data center selection.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
_gat_gtag_UA_65562281_11 minuteSet by Google to distinguish users.
_gcl_au3 monthsProvided by Google Tag Manager to experiment advertisement efficiency of websites using their services.
CONSENT2 yearsYouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
undefinedneverWistia sets this cookie to collect data on visitor interaction with the website's video-content, to make the website's video-content more relevant for the visitor.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
CookieDurationDescription
_fbp3 monthsThis cookie is set by Facebook to display advertisements when either on Facebook or on a digital platform powered by Facebook advertising, after visiting the website.
VISITOR_INFO1_LIVE5 months 27 daysA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSCsessionYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devicesneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-idneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt.innertube::nextIdneverThis cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.
yt.innertube::requestsneverThis cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
CookieDurationDescription
intercom-id-p4lr69wg8 months 26 days 1 hourNo description
intercom-session-p4lr69wg7 daysNo description
loglevelneverNo description available.
trx_addons_is_retina1 yearThis cookie is used for checking if the user has a retina display.
wp-wpml_current_languagesessionNo description available.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
CookieDurationDescription
AMP_TOKEN1 hourThis cookie is set by Google Analytics and contains a token that can be used to retrieve a Client ID from AMP Client ID service. Other possible values indicate opt-out, inflight request or an error retrieving a Client ID from AMP Client ID service.
SAVE & ACCEPT
Powered by CookieYes Logo

Please share your location to continue.

Check our help guide for more info.

share your location