Measuring Digital impact: Moving beyond Net Promoter Score (NPS

And thank you for joining me this afternoon. For the next fundamental webinar. This time on strategic impact evaluation, measuring the return on digital. In this session, I’ll be expanding on a framework for Digital Impact Assessment and showing how organisations from micro to enterprise size can use this process for projects services products and programmes, of work. The session will be about 45 minutes, although we will have some time earmarked at the end for questions. So feel free to send questions in as we go along, or fire them over on Twitter. I’ll deal with them at the end. So just firstly to introduce myself. I’m Tim Dixon I’m currently a senior UX consultant at Nomensa, and this talk and this project that I’ve been working on kind of combines my current role and previous background and previous to this I was a principal consultant in socio economic research. And I also studied, Human Factors psychology previously, and I just generally have a love of data. As I say this talk combined several of those threads. So this talk will focus on four different areas. What is impact. Why impact matters. The digital impact framework, and the process of strategic impact evaluation. So to begin, what is impact impact isn’t always understood in the same way by everyone. It’s the kind of word that can be used in quite a specific sense by some people, and quite a general sense by others. So I’m going to give just a few definitions that hone in on where we’re coming from. So business directory, says the impact is the tangible change that research provides a measure of tangible and intangible effects of one things or entities action or influence upon another. The Oxford University Research Centre says impact is the tangible change that research provides be in policy business industry or society. And as we look at it. In this context, impact is the measurable, long term consequence of digital activity of lasting value. So, we can see that impacts can be both positive or negative, or indeed, it could be both in different ways. We need to be able to break him back down and understand different elements of what it is. And maybe at some point we can see both positive and negative aspects to the impact impacted is also challenging to measure accurately. I love this example, this is from a research paper that’s surveyed the C suite across different companies, asking how much impact they considered their cx initiatives had had this survey, though, forgot to inquire, if these initiatives could actually have a negative impact. In my opinion this is likely to be a biassed response that we see in this. So, again, in identifying how it can be quite tricky to really get a decent robust measure of impact. Following on from that, we have the traditional metrics that are around, such as Net Promoter Score customer satisfaction apps, App Store ratings and cost per click to name but a few there are many of these. I mean, these can all be sent out fundamental flaws, for example Net Promoter Score simply has no real place in quantitative analysis, as it isn’t. It has no better predictive value than many other measures and the premise that it’s a net score is actually in coherence. The cset can easily be ranked within a digital journey to give a positive response. App Store ratings,

well who’s received a request to rate an app at the exact point you’ve had a positive feedback response from it. This is an easy biassing of that response process and cost per click. It’s not really as irrelevant as, as you want to be thinking really in terms of profit per click, and there are more examples of this kind of thinking. So to overcome this. We’re going to look at a different model, then the model we use, and it can be impeccably measured more robustly using a logic model approach. We look for in the logic model, the golden thread that causally connects the inputs to the impacts. So you can see the inputs, start on the left hand side, they logically lead to activities that can be carried out. And they have logical outputs, like literally the things that are done. But then there are knock on effects, which we call the outcomes, and the knock on effects, kind of longer term bigger scale, are the impacts that we’re referring to in this case. And this is sometimes referred to as the theory of change, which we should return to later as the theory of change, allows us to work back from what we plan to be impact to understand what needs to come before, one other aspect is we need to consider the counterfactual. That is what would have happened anyway. So a nice, a nice quote here to sum up some of this. Initially, from Avinash Kaushik and his great Occam’s razor blog. So one way of removing silos and focusing on the entire business is to leverage acquisition behaviour and outcome metrics. This will allow may force, our senior business leaders to see the complete picture, see more of cause and effect and create incentives for the disparate teams to work together. So moving on. Why impact matters. So we’ve got a few different elements to this, but I like to conceive of impact as purpose at scale. As we’ll see the impact models scales from small projects to large programmes. But as we begin to think about the individual, what are their needs, what’s their purpose. The Japanese philosophy of Iki guy presents four facets, or drivers for an individual to have purpose in life, what you love doing what the world needs, what you’re good at doing and what you can be good at what you can get paid for doing. And that sweet spot that intersects all four in the middle. That’s your Iki guy or purpose in life. So if you have an individual for, who has their purpose. They then we can see organisations, as a collective, a collection of individuals, each with their own ego or purpose. So organisational value can be derived from the shared purpose of the individuals within the organisation. We can start to think about Simon Sinek start with wine model, which posits that while most successful organisational leaders will share with that stuff. What they do well. Only some most will clearly instil, how to do it, and few will get across, why the organisation is doing it and why people should care. This why this organisational value. I see is the individual purpose scaling and. But the question is how do we kind of measure that value measure that why. Well, this is where the impact comes in measuring impact provides a proxy for the value provided by the organisation can see, we have an individual purpose leads to the grouping of the value for an organisation, and then at bigger scale impact is the measurement which then feeds back to the individual to tell them that purpose of the organisation. So, evaluating impact. Therefore, provides a tool for communicating to society, about those how well the values of the organisation are being represented. It provides a tool for speaking to the C suite, about what needs to be done, and why it’s why it should be funded. It provides a tool for empowering staff to care about what’s being done for inspiring new ideas. And also for showing the tangible and less tangible returns on investment in this case in digital. Here’s another great quote the culture of a workplace, an organization’s values, norms and practices, has a huge impact on our happiness and success.

That’s from Adam Grant, Professor of occupational psychology in the States. So moving on once more. We’ll get into some more of the details now this is the digital impact framework, upon which the evaluation process is based. So therefore, value drivers of digital impact, as we’ve defined it. So this, the social societal driver. The economic driver operational driver and the innovation driver. And we can start to overlay, these four drivers into the logic model approach that we saw before, so we see the inputs activities, etc. And then at the end, we see the impacts are splitting down into these four different drivers. Now the four value drivers of impact can be split again into those which are external to the organisation. So, social drivers and economic they’re both outside the organisation you’re looking to make changes to society make changes to regional economic profits, as it were. And then the internal to the organisation, our operational drivers, said, changing processes making efficiencies for example, and also innovation within the organisation. So, coming up with new ideas and new ways to spin off from current ideas. So again, we’ll start to split this, and the table, and the model. Further, so we can see the different elements more clearly. At this point, it’s useful to think about the different metrics that might be useful for the, for picking apart, some of these different issues. So, in each of these cases we’ve split apart, what different types of metric we might want to be considering, and there are met. The, the key to understanding metrics for using the diff framework, so that we really want to be understanding, an individual project and individual programmes nuances. First, before selecting the most appropriate metrics, but these are just some exemplars. And also, they, we might talk about more off the shelf kind of easier routes through the metric decision process, but in terms of the, the examples we have here for, for social or societal and drivers, we might look to measure improved experiences in one form or another. And we might look at attitude and behaviour changes within a group, a target group or a population at large might think about increased well being further down the line through usage of some product or service. And we might look at monitored sentiment changes. And these could certainly be through thinking about things like social media monitoring. At the economic level, we can think about increased sales or customer attention stopping sales going to other businesses organisations. And for more charitable organisations, it might be a metric might be funding applications or successful funding bits that go through and then we start to see the more kind of stronger economic factors such as direct and indirect full time employment within an organisation have turnover increase. And then we can think about the real end impacts which are net additional gross value added at a regional or national level. So, this would mean that going beyond the organisation itself, because when an organisation makes money. It has the money spills into the local, local region. Therefore, allowing more people to spend money within that region operationally speaking, we might look at improve staff experience. Staff knowledge and increase skills efficiencies generated within the organisation and monetary savings. And then on the innovation side, we’ve got the novel product or service development itself, we could, we could consider patent applications spin off projects or companies. And also, we could think about the widespread adoption of new processes.

So, the other factor. I’ve also mentioned is the counterfactual. So, impact measurement on this robust scale needs to also consider the flip side of what’s actually being measured. That is what would have happened. Anyway, so really key question. Okay we’ve made some changes, but without this intervention without this product without the service, what would have happened anyway. Maybe those those changes still would have happened naturally within the situation. Maybe they wouldn’t. Whereas we can consider this. Is there a control group. If we’re early in a project, we might want to consider another group that isn’t going to be using the product or service. And we might be able to tap into collecting data from them as well. We can then compare those two groups. We would really want to statistically model the net difference between what would have or, as far as we know what might have happened. And what did happen. And then on the really kind of stronger, more economic side we can think about additionality, the net additionality is considered things like. Did we okay we’ve got some positive effect in the area that we’re interested in, but did it come at the cost of another area another region of the country, actually having a bit of a dip, did we just steal some of that impact that did not generate genuine new impact for example, or various different ways we can consider that, and those who have come down to some quite specific statistical modelling processes. Okay. So at this point, we’ve kind of gone through the basis for the framework. And now we’re going to break down into a bit more detail into the strategic impact evaluation process itself. So we really want to be understanding how we can use the framework to take it further, and actually go out there. So the strategic impact evaluation considers these internal impacts of drivers, the external drivers as well, but we can also then think about it in terms of a planning and implementation phase, and also the final impact assessment phase and different organisations different at different times, or or indeed different projects might want to consider any, any one of these four quadrants that we’re going to consider. So for example if you’re thinking just internally to your organisation, and you’re at the planning and implementation stage, you might consider what can be done within the project to improve processes to generate new ideas for example, if you’re looking more externally, you might consider what might, how might we reach new audiences, or stakeholders. What might we do to generate money in the local and national economy. If you’re going further into the impact assessment itself. Internally you consider what will be the knock on effects of efficiencies and innovations. And on that final bigger scale external facing impacts, what will be the increased well being, the positive sentiment, what will be the gross value added and additionality of this project. And so we can see that measuring it any or all of these will have different facts, we will have different outcomes for the organisation itself. And we’ll be able to improve things, both within the processes that are taking place, as well as within the plan strategy that’s going to come next. So again, we can overlay this on to the different model that we’ve been looking at before. And we can see that the planning and implementation more relates to the, to the inputs activities and outputs kind of phase, and the impact assessment aspect, more relates from outputs through to outcomes and through to that impact value driver phase. So next, what do you think about this bit about UX strategy. So, this is the Nomensa triple diamond.

And just to just add a quick notes I hope you all caught my colleague, Charlie Martin’s webinar on UX strategy. The most recent one that we have done, and where he presented this in quite a lot of detail so if you didn’t catch that be good to see if you can catch up on it. And it’s. We like to think of UX strategy as having this third diamond. While most people in the design community will be familiar with the, the second diamond, the design and the third, the develop aspects, we consider the first diamond as being that very early stage strategics but perspective. Within that we think about what is the vision for this project or this offering what what will, what will be the roadmap, and certainly what is the value proposition that makes that sets this apart for for the organisation. So with that in mind, we can now see that the UX strategy fits very closely, and it aligns with the impact approach that we have here. So, the impact strategy in the UX strategy map onto each other one to one, really. So the vision in the UX strategy informs our impact, and outcome goals. And by having a vision, we can then more closely define what, what, what will we want to achieve what what impacts do we want to create our roadmap, then relates to what outputs activities we need to carry out so much more of that kind of granular detail of what we will be doing. And then the value proposition is what sets us aside. It allows us to create our causal connection between these different elements. It allows us to start thinking about what assumptions go alongside what we’re suggesting. And as you can see here. And I mentioned before about the theory of change. And this really sums up what is our theory of change, we how do we go about defining what the impacts are what comes before the outputs activities, and those causal connections and assumptions that underlie them. So once again, we can reflect back on to the, to the model, and see that we have the strategic ostrich strategic string across the top. So we’ve got the vision roadmap and value proposition, and work on. And then, this really reflects the theory of change. And as things are, this is the this is our full framework and different elements of it, maybe, again, more or less, person for different organisations and different projects at different points in their lifetime. So I want to talk a little bit now about, when will be the most appropriate time to carry out a strategic impact evaluation. So think about timing. It may be, as I mentioned, that you’re early on in a project, you’re thinking about that planning and implementation phase, and it might be in fact that in terms of thinking about the our triple diamond approach, you’re more conscious about designing, or delivering that those, Those second and third diamonds as we put it, it might be then that a planning and implementation approach, considering these kind of elements, these questions as we’ve already gone through might be the more appropriate point for you to be coming in and speaking. And considering these. These factors. Alternatively, you might be just at the start of a big strategic piece of thinking, really, any strategy strategic thinking needs to be defining metrics, and they should be both long term metrics and, you know, more near term metrics. So it’s the ideal point to be thinking about that kind of final phase Impact Assessment aspect, if you are kind of really taking your UX strategy by the teeth. But of course, ideally you’d be thinking more holistically and across all three planning through strategy through design development and delivery. So that, in that case, you would really want to consider across the board of the elements of what we’re going to be

talking about. So to put this another way we can also consider the timing in relation to what where you are with planning a project. So for example, if you’re it’s useful to consider what elements or activities you might want to think through. So, if you’re at the very start of planning a project or thinking about getting some funding estimating what impact you’ll generate will allow you to justify that funding, whether that be to the C suite, whether that be to some external funding body and etc. So that’s the very early phase number one in. Number two, we have planning what activities will actually create impact within your strategy so need to go through and think about that impact in relation to what can be done to generate it. Number three, whilst the project and in full flow. You want to be monitoring the impacts to see where improvements can be made. So it’s an iterative feedback process that can can help to boost the impact and the processes really that are going alongside them. And then finally, at the end you want to evaluate the impact to the jump demonstrate the value. Again, back, go either going back to your original funders, going back to society in general. And going back to you know your staff and getting people to really care. And as we said about that, that value that you deliver, and the individual purpose that people come to you with. So, we’ll run through a few steps that can be taken. And then we’ll also have a hypothetical example that will take you through each of the steps, and we’ll end with some an example of how this might look in practice using the, the framework itself again with some data that were very close to a some projects that we’ve looked at but it’s hypothetical. So, firstly, the first step would be thinking about are we looking at it at planning, implementation or full impact assessment, as we said, really depends on where, where you’re at, what your thinking is, and how big a scale, you want to really be going into next step would be to explore the strategic context and rationale for the digital offering, that’s, that’s that. This will help to establish the theory of change. Next we would establish the priority impacts are these internal or external. This is important because we’ve, we’ve said there for value drivers, but not every organisation, not every project will be prioritising all four of these different value drivers, some sort of charitable organisations will most likely see the social as being the top priority, and enterprise organisations might see economic drivers as being more important. For example, some might see all four as being at the heart of it. Some might see only a couple, and we’d always suggest, at least two, maybe three drivers as being important to consider. The next step will be to establish workable smart metrics that specific, measurable, achievable, realistic and timely and smart metrics that can be monitored and collected. Next year to ensure monitoring tools and processes are in place. So, are those monitoring tools and processes going to be within the organisation running the project or externally to the organisation and thinking about how to gather the data. The probably be some types of user research or larger scale data collection. As it stands, so we’ll go through and look at the various different ways that we can grab that data. And then we’ll analyse it. And then, importantly would report. But the key thing here is to understand it’s not just a single one off report this is a longer term, we’re looking at impacts across a year or two

years even. So while it doesn’t have to be a continuous piece of work, it’s useful to follow up inside, six months, or 12 months, and see what data has been collected by that point in from passive and more active methods, and then reanalyze the data see how the impacts are developing. See how improvements can be made.

And finally, at the end of the project, see what has been established, and where that value has really been derived. So we’re gonna go through this again with a, an example that I’ve come up with. It’s a hypothesis hypothetical scenario but as I’ve said, you might have, for example a mobile health spin out company who sees a gap in the market really relating to efficiencies in house hip surgery is allocated and how it might utilise the latest data trends to fill the gap. So, this is our, this is our basis of our scenario, we’ll go through each of these steps in turn as to what they might do. So given a large scale programme level nature of this offering, we’d consider this as as covering planning, implementation, and impact assessments, this will be a four

assessment process.

We can support the organisation with building their vision and roadmap and value proposition. So in terms of vision, you might talk about wanting to bring about widespread regional changes to process it within hospitals. At the same time as increasing patient confidence in the process and and life satisfaction in those undergoing surgery, the roadmap will be delivered through a mobile app that can intelligently analyse and prioritise surgery, while the value proposition will be based around cutting edge access to current data and machine learning to predict patterns of surgery needs. Again this is all hypothetical but it’s it’s so realistic. And certainly seeing more of these kind of mobile projects coming out our theory of change. Therefore, relevant to this would be. There is a need to improve operation operation waiting times within UK hospitals for patients and staff, looking to have these carried out by investing in intelligent mobile infrastructure that can learn patterns of demand and behaviour, we can bring about immediate awareness of this new process for public get people using the service faster, and with a more positive experience compared to currently and longer term, improve wellbeing outcomes for those using the service. At the same time, training for staff will improve job satisfaction and increase efficiencies within the hospitals it’s worth

saving the service money in the longer term. So next priority impacts are likely to be an internal impact of operational improvements for nurses or doctors in the hospital and externally, improved patient waiting times and outcomes. But we’ll consider all of these impacts as being important in this case,

in terms of smart metrics, we’ll come on to that. And in terms of the approaches in gathering data given the nature of service users, we’re likely to use a combination of social social media monitors. Some lean data collection internal process data analysis and stuff so I think really important to understand that while we can run a large scale survey, etc. We’ve also been exploring lean data collection, which is very much about pinpointing four or five just key questions that you can fire out to a large number of people that have used the service in order to extract just the most significant most important pieces of information from them in a robust and reliable way. So, we would then review across time and catch up

as necessary to analyse the data. So, we’ll move on and look at this in practice, how might this look. So, we would have on the inputs, we might consider

a government funding government grant of some amount and private sector funding to match or to bring it up to a total, those would be the inputs activities that can be carried out. There’s going to be a lot of planning deviation, there’s gonna be design and development, we’re going to be looking at training people, we’re going to look at the launch of the resource, and the public relations that go alongside that. So the discovery and development sprint, are going to be in their design iterations. This is the full package of work that’s actually going to be done. In this example, I’ve actually got broken down into all four of the value drivers, although we’ve prioritised two of them. So first of all, we look at the innovation value driver. So an output would be one novel service is deployed and one new app working is a basic pure output. But the knock on effects what might be the knock on effects of this. So we might end to have say 10 trusts using the service and app, and one spin off service developed within a period of time. And we might have one or the service mainstreamed and the apps rolled out as a standard over a certain amount of time. In terms of the operational process. Value drivers, we would we would hope, say 200 people would be trained and to process documents created to support this. The knock on effects of that are due to people being trained 50 surgeons can take up the resource and 60% feel an increase in confidence in fulfilling surgery in a timely manner. For example, and then we might see that the average waiting time is reduced by. For example, two days across the board. Within these hospitals longer term and this would mean that costs would would be reduced by an amount or, again, the clear point being that we can set defined measurable goals to these. And we can also imagine stuff, increasing in job satisfaction by an even amount. In terms of economic value drivers, we might see 50 jobs created for the app, and the service and 20 jobs in training. And this. These direct jobs mean that, for example, 150 in indirect and induced full time employed might be employed within the in the region. What this means is that, because we’ve supported the the direct creation of jobs. Those people will spend money and go out into the region, and therefore, they will create new jobs because they have money to spend within the region. So these indirect and induced jobs are those knock on effects again. And we might also think about new funding opportunities that spin that out, and then further down the line we might be looking to make a gross value added, which is a regional GDP figure of 20 million for example, a cost, cost benefit ratio of some amount. And then finally, our social value drivers, we might think about 5000 people being made aware of the new service. 2000 public being selected for earlier surgery, 5000 people, maybe sharing info about the service. And then, as we see the final line might want to get 1000 people, positively posting about hip surgery on social media, and five people, 500 people with increased life satisfaction scores, as we go through. So that’s, that is, in essence, as I say a hypothetical but a very realistic and as we’re going through. We’re seeing more and more examples of how we can look into the, the nuances of bigger and smaller programmes of work and going right back down to individual project projects and products being developed at different scales. As we, as we’ve come to a close here. Just want to think about. And one final quote from Chuck Robbins CEO of Cisco, it’s imperative for us to consider how all that we do, makes an impact on the world around us.

And then with that, let’s say thank you. And just to say whether you’re deeply involved in strategic experience design or in the early stages of planning a new project. Love to speak to you, more so we can support your work, and to ensure significant impact is delivered and demonstrated. And I think we do have a bit of time for questions if there are any.

Okay, so we’ve got a question from me. So changing attitudes, is one thing, but changing behaviour is a very separate challenge. And what are your opinions on measuring against these metrics and is always representative of societal change. It’s a it is a very good point. behaviour change, as we all know, is a big challenge for most organisations to consider. And within the framework, we can still see that there would be a number of factors that we want to get across, and for the, for behaviour change, we would still envisage it being within the social kind of band of value driver. And the one thing about the social, the social wider social band is that dependent on the project or the, what what kind of service we’re talking about that’s going to have very very different considerations. For example, coming from a kind of small scale, and UX project that would be the, the immediate outputs might be just changes in measure changes in a person’s usage of a website that is, that’s a basic behaviour change, but if you’re thinking about on a big much bigger scale service design piece, and there are, it’s a very, very different kind of scale that we can start to think about it. This is where it becomes much more important to think about starting this process this impact assessment process early, I started the project. Ideally, then we can understand what the behaviour of that group is going to be using the service. And, I mean, in an ideal world, we talk about finding a control group who are not going to end up using the service, and that that shows the, the clearest difference between the behaviours of the people who are, and behaviours of the people who aren’t. But it that’s not always viable, and we are actually looking at other ways to look back. In fact, we’ve got some social media monitoring tools which allow us to look back to different stages of, say, on Twitter and into the past as well as the future. And so while it’s not an ideal approach. We could also envisage, say for example we were working with one organisation, seeing what kind of feedback they are receiving on in relation to something, we might also consider what kinds of feedback are very equivalent organisation is looking at, so we think about this in a very, a wide range of different methods for tapping into this hope that that’s got to grips with your question somewhat. I’ve got another question. Just a comment from Ronnie thoroughly enjoyed that. So thanks very much. If there’s any more questions. Yes, and again from Ronnie, how’d you get around resistance while implementing the framework, from the wider business. This is, it’s an interesting pointer. I think the most important thing to think about when looking at the impact is that it can make positive change and it gained. It can provide ways to improve services as well. It’s never going to be, you know, get getting early understanding of what what the purpose is, is really important so running through slides like we have done today, it’s really useful. But, yeah, certainly getting across that the, the process can support those different. The different points within a project, and help to make improvements, as well as actually get some final measurements that. Basically, I think every, every manager is these days is driven by some kind of metrics and getting the best, most robust metrics, available, and often gives quite good justification for running this kind of process. Any more questions. Okay. Well, thanks very much. Again, and it’s been a real pleasure to go through this, this afternoon. And we’ll call it a day there.

Thanks. Bye.