Welcome to the third section of our Lost Voices report: Are metrics working? This section explores some of the key drivers that have led to the gradual erosion of trust between some of the key players in the digital campaigning world: charities, MPs and the public. We established in the first section of the report here that there is a cycle of mistrust between these key players, all of whom are feeling the impact of a deep lack of communication. The next section of the report expounded the impact that this has on the voices of lived experience, many of whom feel cheated by a system of digital campaigning that does not adequately listen to their concerns.
Now we’re analysing some of the drivers that have contributed to this breakdown of trust, starting with our metrics of success.
If you haven’t already, we highly recommend you read the first two sections of the report before embarking on this one to gain a full understanding of the issue we’re tackling. You can read them here and here.
Our research has revealed that the metrics of success of many charities are geared towards scalability rather than impact. This focus on numbers is sparking the breakdown of trust between charities, MPs and public. Charities can fall into the metrics trap of viewing members of the public as numbers rather than change-makers. Just one look at MPs email responses show that they are not on the same wavelength of believing that numbers equal impact. (There have been a shocking number of MP email responses that flat out refuse to interact with people if they send a campaign email.) And the result is that the public is given a terrible journey to create change.
Almost all charities interviewed cited numbers as the key factor that determined the success of their campaign. And yet, many also acknowledged the limitations of using numbers. One campaigner admitted that many of their metrics are “plucked out of thin air”. Along the same lines, other campaigners acknowledged their metrics as “a bit numbery” and “a numbers thing”.
While numbers are a powerful tool of showing mass support to decision makers, this heightened focus on numbers results in vague, untailored campaigns. As one campaigner discusses: “it’s important not to just focus on the number of actions taken, but also look at their impact. However, few organisations can track the impact their online actions had in creating change.”
In our interviews with those with the lived experience, many cited the lack of response from the decision maker as a reason they haven’t felt listened to in a campaign. This is a result of a number of intersecting factors, such as MP email accounts not functioning well with mass email, the implicit bias that comes with an MP reading a campaign email, and the necessity for MPs to prioritise their tasks. Yet there is also an erosion of trust between charities and the public which factors into this.
When charities focus all their time and effort on gaining numbers, the genuine and honest relationship with their supporters and, importantly, those with lived experience becomes lost. Those with lived experience are left wondering if they’ll ever hear a response, not only from the decision maker but also from the charity. Many members of the public feel disillusioned with the types of campaigns large campaigning organisations run. Rather than doing in-depth research, many felt campaigns were “chasing headlines” for the quickest success.
Last year at our conference Losing Control, which explored the practical realities of building social movements, Paul Steedman from Friends of the Earth candidly discussed the realities of what a drive for numbers looks like in an organisation. He discussed the controlling tendencies many organisations have, which are amplified by people recruiting people who look like them, rather than investing in the vast network of change makers.
Steedman also highlighted an issue that across the sector is highly apparent – even if we’re not quite ready to accept it. That is that, the people within our networks, the people whose power actually make things happen, are not nurtured within the top levels of an organisation. Staff find them frustrating, annoying, and difficult. As Paul states: ‘Our power to make change happen resides in millions globally who want to make that change happen’.
Thinking about this in the context of digital campaigning, charities need to be introspective about their motivations and intentions behind their KPIs. Transparency is key to building strong relationships, and a lack of transparency by some charities has resulted in this breakdown of trust, in which people feel cheated into signing a petition or emailing an MP because they’re treated like just another number to reach a target KPI.
There are a number of organisations who are already thinking about how to redefine success, here are a few we thought were especially interesting:
Unlock Democracy have been testing a new method of creating KPIs for the past year through the ‘Transformational Index’. This model allows organisations to track not only their impact in numbers, but also their social impact. Unlock Democracy understand that their work is about building systematic change, and they wanted their measurements of success to reflect that. Starting with key transformation indicators, Unlock Democracy have created an impact measurement model based based on the strategic factors they identified as required to create change. These indicators ranged from integrity to engagement to inspiration to proliferation and systematic change. Following from this, measures were decided based on how to best track each transformational indicator. This allowed space each measure to be linked strategically to Unlock Democracy’s overall goal.
I spoke to Director of Unlock Democracy Alexandra Runswick about Unlock Democracy’s motivations behind this shift in measuring success: “While it’s easy to report on the number of people who have opened an email, this doesn’t tell you if you’re achieving change. We wanted to see how we could measure systemic change, and what steps we’re taking in that direction. The transformational index helps you to think about, not the campaign techniques you use, but about what factors on the strategic level are necessary to achieve change.”
Unlock Democracy recognised that this is a difficult model to implement, not least because this type of approach to success is not fully recognised by all funders: “We know it’s easier to get funding for defined outcomes, rather than for building out a movement. It’s difficult to demonstrate to funders what success looks like without the more conventional indicators. The responsibility doesn’t just fall on the charities to change their approach to measuring success, but also on funders to recognise alternative measures of success.”
The shift in measuring success requires a collective agreement, from funders and charities alike, in the benefits of alternative measures. This, coupled with pioneers in this area, such as Unlock Democracy who are able to help provide best practice, will spark a much needed change in the way that many organisations assess change. What’s more, it’s fun! Alexandra’s voice at the end of our conversation was laced with excitement: “Measuring success can be so much more interesting when you experiment and do it differently. It’s genuinely much more interesting when what you measure is supporting your strategic thinking. It helps keep your campaigns focused on making change!”
Tech-for-good agency Reason Digital have developed a Social Value Reporting tool, IMPACT, which allows organisations across all sectors to track their Social Return on Investment (SROI). This tool enables organisations easily organise large quantities of data to produce comprehensive reports on social impact.
I chatted with Bethan, Development Manager of IMPACT, about the motivations behind this tool: “We found that in organisations across all sectors, data on social impact was being collected disparately across the organisation, and then cobbled together in an annual report, with organisations spending months putting together these stats. This tool allows people from across the organisation to input data and their desired outputs all in one place. You can input your metrics, and the system will automatically convert input to output.”
In simple terms, the tool allows everyone in the organisation to use the same database, centrally set multiple goals, and track how you are achieving these goals. What’s particularly interesting about this tool is its ability to segment the data in accordance to who is a beneficiary, or who is directly impacted by a campaign or activity an organisation is running.
Bethan talks in detail about this: “This feature allows you to gather real feedback from people who are impacted by your activity. There is a whole realm of impact that you can discover as a result of diving into the results of one activity. For example, you may find that X percent of people benefitted from an initiative for greater financial inclusion, not only because their financial situation has changed, but also their mental health has changed. The linkages of impact from just one activity can be far reaching. This system allows you to track all of these varying impacts all in one place, across multiple goals.”
I spoke to Chief Exec of Reason Digital, Jo Wolfe, about alternative measurements of success: “This is less about digital and more about planning. Understanding social impact will change everything, from the theory of change of a campaign, to the way a campaign is structured, to the way that it is followed through. There are so many opportunities with digital, but we need to stay open minded and never forget that this is about people”
The Children’s Society have pulled together an impressive evaluation framework for their campaigns. Speaking to National Campaigns Manager Anastasia French it became clear that The Children’s Society do not silo their digital campaigning into one team. Rather than looking at the success of their digital first, they analyse the success of each campaign as a whole. Interestingly, a part of this evaluation includes interviewing decision makers on their experience of being the target of a campaign. This allows The Children’s Society to tailor their tactics in accordance to what’s most impactful for their decision makers.
The success of a campaign needs to be about more than numbers. While numbers can be important, and often the easiest way to track the success of a campaign, what’s missing is the voices of people affected. Measuring success responsibly requires a comprehensive framework which takes into account the overall goals of the organisations, and key indicators that help pinpoint what success looks like. It includes things like feedback from key targets, building relationships with those most directly affected, and facilitating supporters to become change-makers. Gearing success metrics towards social impact will help to rebuild relationships between charities and the public, and in turn offer a better journey to creating change for those with the lived experience.