How many responses do you need in order to have an accurate measure of the Net Promoter Score? What is the confidence interval on your score? Do you really know if it has changed since last measure? If you are going to use the score for anything, you need to know the answers to these questions.
For the very impatient, a good rule of thumb is that with 1000 responses you (only) know your Net Promoter Score to within 10 points (±5) and it takes fully 100,000 responses to know the score to one point (±0.5).
The Net Promoter Score (NPS) is a metric for measuring the loyalty and engagement of a firm’s customers that is extremely popular in across many industries. Originally introduced by Fred Reichheld in a 2003 article in the Harvard Business Review (link); the current reference is the book The Ultimate Question 2.0 by Reichheld and Markey (Amazon paid link UK | US).
The score is calculated by asking the firm’s customers a single question, How likely is it that you would recommend our company/product/service to a friend or colleague?
, on a scale from 0 (very unlikely) to 10 (very likely). Those customers who respond 9 or 10 are considered Promoters, those who score 7 or 8 are Passives, and the rest scoring 0-6 are Detractors. The Net Promoter Score is then simply the percentage Promoters minus the percentage Detractors, which gives a number between -100 and +100.
We are not going to go into the merits or criticisms of the Net Promoter Score here. Many companies have adopted the score as a key performance indicator; all we want to do here is to help the managers in those companies understand the uncertainty in the measured score inherent from the sample size (ie number of responses).
What does it mean?
For now let us just focus on the rule of thumb: with 1000 responses the sampling uncertainty on your Net Promoter Score is 10 points, with 100,000 responses it is 1 point, and so on, scaling by the square root of responses. The details are in the section below and a later post will have all the maths, stats, and R code, so stay tuned if you need the details.
Some organizations become obsessed with the score itself and use it as a key performance indicator and perhaps even bonus target for the whole company and by region or product line. Clearly, unless you have hundred thousand responses or more, or you are really looking for a transformational change to the score (say, +20 points), that is unrealistic at the company level and probably always unrealistic at the line of business level. You just don’t have enough data.
So if you are, for example, a large telecommunications provider in the US (or China or other large country) with about 100 million customers you can probably get of the order of 1 million responses per month from a NPS survey and you are in good shape even if you cut it 10 ways (say, by Corporate, SME, and Consumer, and by Prepaid, Post-paid, Fixed-line, and data-only, which is already ~12 segments). By all means use the score as a business KPI. But don’t just use the score.
And don’t give up if you have less than hundred thousand responses. Certainly do not walk away from NPS. Reichheld tries, with varied success, to talk about the Net Promoter System (as opposed to –Score) of which measuring the score is just one component. By all means measure the score (as best you can) but more importantly do something with the feedback you get from your customers. Close the loop by (1) sharing the feedback across your organization, (2) contacting customers to understand the root causes behind their feelings of their experience with you, (3) analyzing the root causes to recognize what really matters to your customers, and (4) taking actions that change the customer experience you deliver.
And repeat the process. All the time. The outcome is to become truly customer centric, to really listen to your customers.
There is only one boss. The customer. And he can fire everybody in the company from the chairman on down, simply by spending his money somewhere else.
— Sam Walton (founder of Wal-Mart Stores)
Fred Reichheld argues the point in a recent post:
Smart executives know that there’s no such thing as an “average” customer. Chasing the average customer really means trying to serve the broadest potential market. In the end, everyone’s underserved because investments lack focus and impact. For some companies, this is a death spiral.
The idea behind the Net Promoter System is a powerful one. How likely are you to recommend us?
Recommending someone takes social capital which is precious for a social species like ours. It represents a real commitment to you and your brand; it is a significant signal. It is real.
You can argue if it is the best question and you can certainly argue with how the score is calculated. But that is really not the point. The point is to get real with your customers, and NPS is a good tool for that. Just don’t get too obsessed with the score.
NPS uncertainty from sample size
Consider as an example what we will call the balanced case where the company has exactly one third of each of Promoters, Passives, and Detractors in its customer base, for a (true) Net Promoter Score of zero. However, if we only ask, say, 10 people, we are unlikely to exactly zero as the measured NPS. The graph shows the 95% confidence interval for the score in this balanced case where we have 10, 30, 100, 300, …, 100000, and 300000 responses.
With 1000 responses you only know the score to ±5 and you need 100,000 responses to be confident you know your Net Promoter Score to the whole number (±0.5).
That’s a lot of responses required to know anything with any certainty. It may be easier to see as the width of the 95% confidence interval by the number of responses received as in the figure below. This is the special balanced case, but the slope is (of course) always such that to get 10 times better estimate you need 100 times more responses.
The width of the confidence interval varies with the proportion of Promoters and Detractors; the table below shows the (half-) width of the 95% confidence interval for n=1000 responses and proportions 0.1, 0.2, …, 0.9.
| d | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| 0.1 | 2.8 | 3.3 | 3.7 | 4.0 | 4.1 | 4.2 | 4.1 | 4.0 | 3.7 |
| 0.2 | 3.3 | 3.9 | 4.3 | 4.6 | 4.8 | 5.0 | 5.0 | 5.0 | |
| 0.3 | 3.7 | 4.3 | 4.8 | 5.1 | 5.4 | 5.6 | 5.7 | ||
| 0.4 | 4.0 | 4.6 | 5.1 | 5.5 | 5.8 | 6.1 | |||
| 0.5 | 4.1 | 4.8 | 5.4 | 5.8 | 6.2 | ||||
| 0.6 | 4.2 | 5.0 | 5.6 | 6.1 | |||||
| 0.7 | 4.1 | 5.0 | 5.7 | ||||||
| 0.8 | 4.0 | 5.0 | |||||||
| 0.9 | 3.7 |
Footnotes
In this table we employ the usual approximation and take 1.96 times the standard deviation as the confidence interval so take care at the extremes. We will show the correct way to create confidence intervals from your data in a later post.↩︎