Engaging citizens in policy-making on the internet is a relatively new practice, but one that usually comes with massive expectations. But what exactly should be measured when we speak of true success in an online participatory process? Here’s a concise guide on how to get started on it.
Start with why
Why should you measure the success of an online engagement process? An accurate measurement leads a correct judgement of the merits and flaws of the participation process. It helps you increase accountability inside the public administration, optimally manage finance and resources in the future, and assess whether the citizens’ input has made a difference in policy-making.
It is all-important to align everyone, from the technical solution provider to the members of the project team, on a handful of essential success metrics. These are often referred to as Key Performance Indicators or KPIs. Once KPIs are defined, it is time to track your progression -or regression- along trend curves that allow you to understand what works and what doesn’t. Eventually, you should be empowered to dip in at just the right time to ensure downturns don’t endure and the highs are lasting.
What to measure?
In short, one can call an online public participation trajectory a success once there is an engagement of a group of citizens representative of the broader population that has led to high-quality insights used for decision-making via a cost-efficient process.
1. Guaranteeing inclusiveness
The users you consider engaged -the crowd you actually « recruited »- must be inclusive enough. Direct online participation is designed as an improvement of our current political system, where most of the decisions occur under the approval of representatives elected by all. It is therefore natural that online participation must aim for the maximal level of representativeness among its audience. This is an imperative not only for legitimacy but for the quality of the crowdsourcing. It means large enough numbers of participants. And before a majority of the population is actually involved in your offer of a more direct democracy (which might take some time!), it is essential that the batch of joiners be diverse.
A frequent critique against digital democracy is that it only manages to attract, in essence, 25-40 year olds from the urban upper middle class with above average IT skills and formal training. Although this group may be easily converted into active participants, online participation has the potential to reach everyone. It is your responsibility to make sure that not only the digital elite get involved but that people from all walks of life take part —through improved interface accessibility, better targeted communication efforts, and so on. Hence, the advice here is to not forget carefully monitor the numbers of participants as well as their demographics.
Key measure: Participation distribution amongst pre-defined citizen groups (based on age, gender, area of residence, etc.).
2. Generating qualitative content for policy-making
There is no real point in involving people if what they share isn’t of use to anyone. Hence proposals and opinions sourced from the public should be qualitative, which entails several things. First the representativeness of your audience is key : views expressed exclusively by one segment of the opinion spectrum are no collective intelligence but rather a takeover by vested interests. Then the contributions should be addressing current and actual issues faced by the community. That is, they must be relevant in order to deserve a share of public resources or of the citizens’ time and efforts. These are limited and are always channelled to the most pressing shortcomings. Lastly but importantly, digital participation must be a vector for innovation. On top of increased legitimacy, it is expected that doubling up the government official responsible for a decision with a crowd of citizen-experts paves the way to disruptive solutions -not to dull ideas sustaining the status quo.
Key measure: Number of proposals that have actually contributed to the actual decision(s). (Note that demonstrating causal links between participation and policy outcomes is often difficult because of the time lag between processes and policy or public action. That’s why you’d better add qualitative measures to come to a more insightful case study analysis.)
3. Ensuring the process is cost-efficient
Direct online participation is here for the long run ; this means it has to be sustainable. The time and efforts required to make your engagement a success shouldn’t be disproportionate. The easiest way to measure this is to make sure the cost per engagement action remains moderate. By measuring the efficiency, you want to see questions answered such as: “Does online participation save money by easing the implementation of policies, for instance by decreasing conflicts to the final decision?” and “How much time did it take for the staff to go from design of the process over moderation to conclusions?”.
Key measure: Cost per engaged citizen, ie. the sum of monetary resources spent on the whole participation process (amongst others staff salary, communication resources, and purchase of technologies) divided by the number of engaged citizens. To make sense of this KPI, compare it to offline alternatives.
Learn from your citizens for future success
A measure that is often overlooked but has proven to be extremely insightful towards future successful participation programs is the satisfaction of your citizens. Luckily, it is fairly simple to get a representative sense of how the participating citizens felt about the participation process; a satisfaction survey does the job. This survey should ask the participant amongst others about the satisfaction with actual outcomes and his/her feedback about the participatory process design. These insights will help set priorities in which changes to be made towards your next participation programme. Good luck with it!
