As a researcher, I’m trained to look beneath the surface”to question, rather than take things at face value. Imagine my annoyance, then, when I find that I’ve accepted at face value something I should have questioned. Lulled into a false sense of security by big brand names like “Harvard” and “London School of Economics,” I, like many others, bought into the Net Promoter Score (NPS) as a leading indicator of business performance. So this is my chance to set the record straight.
Earlier this year, Millward Brown published a Point of View on the topic of Word of Mouth, written by yours truly, which cited the London School of Economics study:
The London School of Economics Advocacy Growth Study 2005 confirmed previous work by Frederick Reichheld in the United States, finding that Word of Mouth advocacy is linked to company growth in the UK; the more brand advocates you have, the higher your growth.¯
As I was preparing the POV, one of my colleagues suggested that the validation of NPS against business performance might not be bulletproof. Lacking the time to investigate further, I forged ahead. Later, when I came across this post, I found the time to carefully re-read the original article.
Reichheld’s original work was published in the Harvard Business Review in December, 2003 under the title The One Number You Need to Know. The article described the work done by Reichheld to develop and validate a measure of consumer loyalty. Assisted by colleagues at Bain & Company, along with a company called Satmetrix, which works with customer satisfaction data, Reichheld matched consumer responses to 20 questions from Bain’s “Loyalty Acid Test” with purchase and claimed referral data for the same respondents. With information on 4000 customers, they built 14 case studies with sufficient base size to measure the linkage between statements and behavior. The winning statement, “How likely are you to recommend (company X) to a friend or colleague?” ranked first or second in 11 of the 14 studies.
Subsequently, in the first quarter of 2001, Satmetrix started tracking the “would recommend” scores of thousands of people for hundreds of companies in more than a dozen industries. The brief survey asked people to rate one or two companies with which they were familiar. The results from these questions were correlated with growth rates for the companies asked about. Reichheld concluded: “Research shows that, in most industries, there is a strong correlation between a company’s growth rate and the percentage of its customers who are ‘promoters’ that is, those who say they are extremely likely to recommend the company to a friend or colleague.” (In fact, the score used was actually the now famous “Net Promoter Score,” which subtracts the proportion of negative promoters from positive promoters.)
That sounded very credible. A pilot test, lots of people interviewed and results validated against independent measures of business performance. Then I came across the one small fact that I had not previously realized. The dependent variable for the analysis was the company’s average growth rate over a three-year period, which not only included the survey period, but actually preceded it by at least one year (1999 to 2002).
Let me ask you, does that not sound just a little odd? The article does not misstate the facts. The NPS score does correlate to the growth rates used in the analysis. But that analysis does not prove that NPS will predict future growth rates. In fact, a legitimate argument would be that the NPS score was the outcome of business performance which preceded the time of the surveys. Sound business practices and a great product or service have always been requirements for growth. Maybe both the growth rates and the willingness to recommend reflect that fact? We cannot tell from the article.
Turning now to the work I actually quoted in the POV, The London School of Economics (LSE) Advocacy Growth Study 2005, I can now say with absolute conviction that that study does nothing to confirm that the NPS is predictive of business success. Again, a conclusive sounding claim is made: “For the brands in the survey, a one per cent increase in NPS equated to £8.82m additional sales, with a seven per cent increase yielding one per cent additional brand growth.” However, the NPS survey was conducted in 2005, with the growth rates derived by comparing 2004 revenues with 2003. Again, we cannot tell from this data whether the NPS score follows business performance or leads it.
Over the years I have been involved in several exercises to validate Millward Brown’s research measures against business performance, such as the validation of TVLinkTM pre-test results to short-term share change and the BrandDynamicsTM Voltage score to year-on-year share change. In both cases, however, the validations have been forward-looking, i.e., they were based on sales data from periods which followed that of our research. For Link, we used the share increase or decrease in the eight weeks after the tested ad went on air. For Voltage, we used the annual brand share in the year following the survey. In our business there is no point in predicting history; it is future performance that matters.
Upon reflection, I think that the annoyance I feel as a result of taking the Harvard and LSE data at face value stems from something more than just irritation at my own lack of diligence. I am also frustrated to observe the different standards of proof required of big academic brands and big market research brands. Both the academic brands seem to find the correlation with previous business performance to be convincing proof of the value of NPS. I do not. Nor, I suspect, would many of our clients if they realized that the validation was actually backward-looking.
In closing this post let me be clear. I am not saying that NPS is useless or invalid. People’s willingness to recommend your company or brand is a good measure of their satisfaction and predisposition. But is it really the one predictive number that you need? Sorry, but from my point of view, the jury is still out. We need to see more compelling proof that NPS actually does lead business performance before we adopt it as the sole benchmark of success.
Is the nature of the NPS validation news to you? Please let me know.
(15 votes, average: 4.2 out of 5)
Loading ...
Email This Post
September 6th, 2006 at 11:23 pm
Nigel,
I’ve read his Reichheld’s book: “The Ultimate Question”…it’s pretty compelling. I think we also want to believe in the simplicity and elegance of the one-number framework. But one thing that’s clear — even from the book — is that the model is not applicable to all industries and situations, let alone the lack of proof you note above. But I think the question he stumbled upon is a fascinating one, because it really delves into deep, powerful human motivations and characteristics — psychological and social. “Recommendations to friends” infers trust, expression, knowledge and status, among many other things.
Have you tried using the question on Millward Brown’s own clients?
September 7th, 2006 at 7:12 am
Hi Max, thanks for the comment. Who could argue that a willingness to recommend a brand is not a good thing, for all the reasons you suggest?
In answer to your question we have used the question on some of our client’s tracking studies and last year added a simpler version to BRANDZ (would recommend/would not recommend rather than a 10 point scale).
My recollection is that an impromptu review of the tracking suggested mixed success as a leading indicator, just as you suggest might be the case. The BRANDZ data collected across 3000+ brands worldwide demonstrated a relationship with brand strength but does not prove it to be a leading indicator.
I will post more if I find some data worth sharing.
September 7th, 2006 at 10:13 am
Dear Nigel,
pls find here two brief tips which support some doubts about the validation method (which does not mean that the WoM doesn’t count).
from a methodological point of view “correlation is not explanation”, which stat controlls have been applied ? indeed if we make a correlation with number of pigeon passed on the london sky we may find a good result…is that enough to consider that a relationship have been discovered ?
from another point of view there may be a kind of ecological fallacy issue using data at different aggregation level (or logic typs in the russel language).
but here i stop leaving the thing to the most expert on the topic.
September 7th, 2006 at 10:58 am
Nigel,
Thanks for shining light on these validation claims.
I’d call myself an NPS skeptic in an organization that has wholly integrated Net Promoter as both a consumer and B2B satisfaction metric. I fully accept the very inituitive hypothesis that growing an adovacy base/reducing your “detractor” base can only yield positive results for your brand/organization. The strength of NPS is that it provides an elegantly simple way (as Max suggests) to measure and trend the size of these groups over time.
The issues I’m faced with as essentially the lone researcher in my organization is that NPS is held up as both the measurement tool AND the means for understanding why (via the open end follow-up question Reichheld recommends). Even if we could say for certain NPS did predict future growth, my fear is that many organizations are not surrounding NPS with the additional insights required to make the NPS results actionable.
I’m hoping that the work MB (and others in the industry) is doing helps add depth to the Net Promoter dialogue.
September 8th, 2006 at 8:35 am
Thanks for the comments everyone.
Tim, I would share your concern. The question to my mind is not “Is NPS a useful metric?” but “Is NPS the only metric you need?” When it comes to diagnosing why people recommend a company or not I am sure you are only hearing from the vocal minority and they may not give you the complete picture on an unaided basis. Problems and issues that may have compounded a specific problem may go unmentioned in favor of focusing on the issue that tipped people over the edge to become detractors. I say “may” because I do not have any data to go on here but good research practice suggests that you should confirm the scope and depth of issues identified this way with an aided approach (if only to see whether issues are lurking among the silent majority as well).
I do have some additional feedback on the value of NPS as a predictive measure. Based on one category in the UK we can say that NPS does have a positive relationship with future share growth, but that NPS in that category seems related to rational and emotional affinity and is less related to some of the other brand drivers, e.g. perceived popularity and price. We do not have a like for like comparison but the relationship identified was similar but weaker to the relationship between MB’s measure of brand potential - Voltage - and future share growth in the same category (but in a different year). Even if they were the same the advantage offered by Voltage is that it is derived from a brand equity framework that allows you to identify the drivers of brand strength or weakness on an aided basis. A colleague of mine has also pointed out that the customer satisfaction firm identifies three behavioral outcomes of experience: willingness to recommend, likelihood to continue using and likelihood to increase usage.
Time series modeling of sales data in the US suggests that in another product category willingness to recommend was related to sales in one channel but not in others, which were more related to brand saliency. This suggests that the value of the NPS measure may not just differ by product or service category but may also differ by sales channel. So online, when someone conducts a search for instance, the degree of trust inspired by the name at the time could outweigh what others have previously told you.
We are continuing to dig into this issue and I will post more information when we have it.
September 14th, 2006 at 6:11 am
Nigel,
As you, I’m not the world’s biggest fan of the NPS, and not just becuase my wife works for Enterprise!
The great thing about NPS is that it does provide a powerful and easily grasped metric of customer commitment - “jeez, x% of our customers like us enough that they’d recommend us to other people.” But much as this is intuitively powerful, what our clients really need from us (and I think we really do try to deliver) are results that help them look into the future to understand the likely future health of their brands and to decide what actions they should take.
My main criticism of the NPS is usually the same one that you raised in your original post in this thread - that the validation is against historical rather than future growth.
A more interesting, and probably more powerful, criticism that’s emerged in the discussion (thanks to Tim I think) is that we need to give more detailed diagnostics of customers’ views in order to guide business decisions. This in turn opens up the challenge of communication - once Reichheld and co have persuaded senior decision-makers that research can help them to drive growth (and I’m all for that), we need to help them understand the need for detail as well. They may not need to see the detail, but the people who are making decisions at the coal face to need it if those decisions are to be rooted in the consumer reality.
September 19th, 2006 at 6:55 am
Thanks Trevor, I agree that Tim’s point is critical. You would not be very happy with a doctor who stated, “You have a fever and it’s likely to get worse, but I have no idea what the cause is or how to treat it.”
September 25th, 2006 at 7:52 am
Nigel - have you thought about going back to the LSE and getting them to repeat the analysis now based on the latest growth rates for those companies vs the NPS score obtained in their original study? Then surely you’d have your answer, no?
September 27th, 2006 at 1:47 pm
Hi Nigel,
I sat in on a WebEx yesterday titled, “The One Number You Need: Does One Size Fit All?” The Speaker was D. Randall Brandt from Maritz Research. While Brandt agrees that the NPS is attractive because it is easy to calculate, people understand promoters and detractors and it correlates with sales (in some sectors), he feels strongly that the NPS is not the best predictor of customer behavior.
Brandt categorized his problems/issues with the NPS into four general areas:
1. There are many differences between industries, markets and customer types and it should not be expected that one size could fit all. There is evidence that customer loyalty varies by industry, market and customer type. Further, in some industries, loyalty may not even be the best metric to predict customer behavior and using the NPS would not be relevant.
2. Although the NPS may perform well in predicting customer behavior, other measures might work better. Brandt correlated the following metrics with customer behavior:
A. Willing to recommend (NPS)
B. A hybrid measure that included overall satisfaction, willing to recommend and intent to continue business
C. A probablity allocation measure (respondents allocate share of their next 10 visits)
Brandt found that the probability allocation measure had the highest correlation with customer behavior followed by the hybrid measure. He also discovered that the probability allocation measure had the best discriminatory power.
3. Using NPS as a core index with key driver analysis may cause managers to focus on the wrong issues. Brandt used different loyalty measures as dependent variables and ran a true driver analysis, a type of stepwise regression. In comparing the NPS and the probablility allocation measure, the probability allocation measure matched many more attributes with actual behavior than willing to recommend (NPS).
4. The NPS has limited interpretability and usage in data analysis. NPS takes a precise 10 point scale and then reduces it to a 3 point scale. If more points on the scale were used, correlations with actual behavior would likely be better. Further, because the NPS is an aggregate level number, you can’t run analyses at the customer level. Finally, because we’re subtracting detractors from promoters to get to the NPS, the NPS score could be the same in very different scenarios. Brand A, a brand with no detractors at all could receive the same score as Brand B, a brand that had many detractors, but more promoters - not quite apples to apples.
Brandt concluded by saying that first it is important to define the critical measure that you need to predict customer behavior. If loyalty is that measure, then do your homework and evaluate what metric would work best, it may be NPS, but there might be other metrics out there that would work better.
September 28th, 2006 at 3:34 am
Cathy, thank you so much for this comprehensive review of Randall Brandt’s presentation. It really does seem to confirm the concerns that some of us have had about this measure. However, do you remember what the dependent behavioral measure for the analysis was and how many brands were covered?
September 28th, 2006 at 7:34 am
Nigel,
Brandt tried to cover a lot of information in our one hour session and, unfortunately, he did not get into specific details of the statistics. We have asked for a copy of his presentation and we’re hoping that the finer details will be footnoted. I’ll forward the presentation on to you when we receive it.
September 28th, 2006 at 8:25 am
Cathy, thank you for forwarding on Brandt’s presentation it makes interesting reading (copies can be requested from http://www.maritzresearch.com/onenumber
The behavioral data used was share of restaurant visits (judging from the charts). NPS had a correlation of 0.36 with behavior and Probability Allocation 0.71.
We have been able to compare NPS data collected in a couple of Millward Brown studies with market share data in the year following the survey. Both NPS and our forward-looking measure Voltage did a reasonable job of predicting which brands would gain or lose share and the difference was not as marked as that reported by Brandt. Voltage seems to have a higher correlation with share change than NPS but the results are not conclusive. It is not clear what time frame Brandt used and it may be that a year allows peoples’ recommendations more chance to have effect. Equally, it could just be that different categories have different relationships as Brandt suggests.
December 12th, 2006 at 8:40 am
My colleague Travyn Rhall mentioned this December 2006 article by Ken Roberts, managing principal at Forethought, in Australia. Ken also doubts the value of the NPS and uses data from a financial services customer to prove that NPS fails to capture any relationship with market share in the corporate market and is not a significant predictor in the consumer market. Instead he finds that “performance” is the strongest predictor in the corporate market. Ken concludes that multiple measures are required to understand the drivers of business sucess.