Satisfaction Guaranteed (Or Is It?)

Many factors affect the reliability of customer satisfaction scores

Customer loyalty is never more important than in turbulent times. High levels of customer satisfaction are a must to win customer loyalty. That's one reason measuring customer satisfaction is a great idea--if the numbers are real! Unfortunately, many customer satisfaction scores are unreliable.

For example, Service Management Group's research from 60,000 retail and restaurant locations shows that customer satisfaction scores vary greatly depending on something as simple as how the data is collected. For example, customer satisfaction surveys taken by phone are noticeably higher than the same surveys taken on the Internet. In fact, the difference between these two scores can range from 3 to 17 percent when measuring customer satisfaction for the very same customer experience. There are a couple of good reasons for this, something to keep in mind when you conduct your own customer satisfaction surveys.

First, the people who respond to phone surveys often are a different group from those who take Internet surveys. Internet respondents skew toward younger, male, and the more affluent; women, the elderly, and the less affluent are overrepresented in phone surveys.

Who are your customers that you want to satisfy?

Second, people will process information differently when they hear it (phone, auditory) compared with information they see (Internet, visual). Neither method is inherently better or worse, more or less accurate, but they are different--which raises the question of which method you should be using. The answer depends on your customer demographics. The better choice is the measurement method that most closely aligns with those of your "ideal" customer.

For example, if you buy at Starbucks often enough, you'll eventually get a special receipt. It will invite you to take their customer survey on the Internet--a good choice for Starbucks customers, who generally are more sophisticated with computers than say, Burger King's customer base.

Burger King's customer demographics are a better fit for phone surveys. You can see that for yourself: just look on the back of your Burger King receipt. You'll find a customer satisfaction survey invitation with a toll-free number to call. Use it and you'll be "talked through" an interactive voice response (IVR) phone survey and use the phone keypad to punch in your ratings.

What happens to the reliability of customer satisfaction measurement when a company gives customers the option to take the same survey by Internet or IVR?

Combining IVR and Internet survey scores is a little like measuring the temperature of a swimming pool with both Celsius and Fahrenheit thermometers. For the same pool, one will read lower, the other higher--and the combined, averaged figure will vary dramatically based on the number of Celsius and Fahrenheit thermometers you use.

When measuring customer satisfaction with both Internet surveys and IVR phone surveys at the same time, you'll get different scores: Internet surveys yield lower numbers than IVR phone surveys. Using both, your combined, averaged customer satisfaction score will go up or down based on the number of Internet or IVR phone survey measures included.
Now, get this! These differences in customer satisfaction scores are caused only by differences in survey methods. They have nothing to do with changes in the customer experience!

The problem with combining Internet and IVR phone survey scores is actually much more difficult than combining temperature scores. The comparability between Celsius and Fahrenheit is constant, and a little math will set you straight. The relationship between Internet and IVR phone survey scores is much more complex and unpredictable. Take a look at this graph.

Think of it this way: If Internet survey scores are lower than phone survey scores (and we know they are), then as the number of Internet surveys increases customer satisfaction scores must decrease. But this change in score is just an illusion.
It's still very important to measure customer satisfaction. But let's get real.

Jack Mackey is vice president of Service Management Group. Contact him at jmackey@servicemanagement.com or (816) 448-4556. For information about accurately measuring customer satisfaction, send him an email requesting "Five Things We Learned from Talking to 100 Million People."