Youth for Climate used CitizenLab’s NLP technology to turn thousands of citizen contributions into concise and actionable insights. Here’s what we learned from analysing over 1,700 ideas.
On August 20, a 15 year old Swedish climate activist called Greta Thurnberg sat down in front of the Swedish government with a sign protesting inaction against climate change. The next day, she was joined by a handful of other protesters. A few weeks later, hundreds had joined. A few months later, the movement had gone global and Youth for Climate groups took to the streets all over the world to get politics to act against climate change.
The movement was quick to reach Belgium. In early 2019 Belgian youths started protesting every Thursday, marching through Brussels and other major Belgian cities. As press attention and momentum grew, the organisers of Youth for Climate Belgium quickly decided they needed a way to channel the energy and ideas being expressed every week in the street. In January 2019, a CitizenLab platform was set up for them: Youth4Climate, where users from Belgium and beyond were invited to submit ideas on how to tackle climate change. Discussions on the platform were as lively and passionate as they were on the streets: in just under 3 months, users posted over 1,700 ideas, 2,600 comments and voted over 32,000 times for the initiatives they wanted to support.
As with all participation projects, this is where the challenge truly started. Collecting input was just the easy part: in order to turn these ideas into meaningful actions and recommendations, Youth forClimate needed to process thousands of ideas in a short period of time. This was a perfect use case for the automated data analysis feature of the CitizenLab platform.
As we accompanied Youth forClimate through the process of collecting and analysing the input, we were able to test our NLP technology on a wide scale, and run it on thousands of contributions written in multiple languages. Here are a few of the learnings we made along the way.
Learning 1: human first, tech second
With an end goal of submitting a concise and actionable report to elected officials, the YouthForClimate organisers together with CitizenLab decided to organise the ideas on the platform into top priorities for climate. The topics which existed on the platform and had already been assigned to some of the ideas couldn’t be used: most themes were too vague to determine true priorities, and a large part of users had chosen not to classify their ideas, meaning that the bulk of ideas were untagged.
Automated classifying wasn’t an option either: trying to automatically tag all of the ideas would have been near impossible. Without giving the algorithms instructions on how to classify the ideas, they would have applied a classification model that was either out of context or too vast. We ran some tests, which confirmed our intuition: with such complex conversations and with such a high number of messages, the algorithms need some framing before they can get reliable results.
The solution was therefore to manually determine the main categories by reading through the ideas and using recurring topics. In order to avoid reading through 1,700 ideas, it was decided to first focus on the most “popular” ones. The top 15% of ideas (217 ideas) accounted for almost 74% of the votes, a strong sign that they were representative of ideas as a whole. The ideas which gathered over 100 votes (60 ideas in total, accounting for 43% of the total votes) gave us the top 10 priorities. The ideas which gathered between 50 and 100 votes (57 ideas, representing 15% of the votes) gave us a further 5 priorities. These 15 priorities were enough to cover the rest of the ideas: we manually read through an additional 139 ideas which had gathered between 20 and 50 votes, and all of these fit into at least one of the 15 priorities.
Once these priorities were determined, we turned to NLP to apply them to the remaining 1,400 ideas. These ideas were tagged using the similarity method: every idea from the “untagged” pot was compared with an idea from the “manual tag” pot – comparison was done by looking at similar words and checking the context. It’s not enough that two ideas both use the word “book”: we also want to make sure the word means the same thing in both comments. The opposite is also true: if an idea mentions “a book” and another idea mentions “reading”, the algorithm recognises these as similar. Once the algorithm found the closest tagged idea, it applied the same tag to the untagged idea. Manual checking of the results showed that even when the algorithms got the tagging wrong, the tag that had been applied was still somewhat relevant to the idea.
Learning 2: anticipate bias
In terms of number of ideas and comments, YouthForClimate has been one of the biggest platforms we’ve worked with. It has therefore been an opportunity for us to train our algorithm on datasets that are bigger than the ones we usually see on other platforms. In doing so, we have been able to learn and to fine-tune our approach.
We are aware of the limitations that come with manually assigning ideas, and we know there is always a form of human bias that comes with selecting or summarising main ideas. Being aware of this possible bias means we can try to overlook it and stay as neutral as possible.
We made the decision that the most popular ideas were also the most relevant ones, but it could of course be that some ideas which did not get voted for also suggested valid priorities. Moreover, the platform showed popular ideas first, therefore reinforcing dominant propositions. In order to avoid this on future platforms, we’ve now randomized the order of ideas on project pages.
Finally, a positive learning has been that multi-linguism is not necessarily a barrier to citizen participation projects. Machine translations ensured that users who spoke different languages could read, comment and vote for ideas in different languages, thus eliminating the language bias. During the analysis phase, we were able to work with the 3 languages on the platform (French, Dutch and English) by translating the ideas into a single language and working from there. Manual checks showed that idea grouping was then able to work and group together ideas which were translated from different languages.
Learning 3: transparency, transparency, transparency!
Analysing contributions isn’t everything: in order for the project to come full circle, it’s also necessary to share the results with the participants. In the spirit of co-construction and transparency, Youth4Climate organisers were intent on keeping the whole process as open as possible.
Once the 15 priorities were determined, the results were posted on the platform along with an explanation of how the ideas had been processed. The results and methodology were also shared by email to all the users who had created an account on the platform, regardless of whether they had submitted an idea. This approach helped increase trust in the process, and kept users interested long after they’d interacted with the platform. It also helped get users back on the platform for the next phase of the project.
The 15 citizen priorities have been translated as new ‘ideas’ on the platform. Every one of these priorities comes with a short description, and features links to the most popular related ideas posted by citizens in the first phase. Citizens have been invited to vote for the topics they think matter the most. Once the prioritisation is finalised and votes are counted, the Youth4Climate organisers will share the proposals with elected officials and bring them to an expert panel they are in close contact with. These upcoming phases have been explained on the platform, making it clear to users where the project is going and why their vote matters.
Being part of this project has been a great learning experience for CitizenLab. It has allowed us to work alongside a civil society group for a topic we’re passionate about. Throughout the process, we’ve also been able to gather valuable learnings which we will be able to implement on future platforms. If you want to know more about how we use artificial intelligence at CitizenLab, head this way!
So, what’s next for climate? Head to the Youth4Climate platform and check out the status of the ideas!