Revealing Information From Customer Surveys

I currently manage a retail enterprise whose customers receive surveys from our product manufacturer in addition to the surveys we solicit from our customers for our own business management purposes. These surveys offer a revealing look into the mind and motivations of both our manufacturer partners and our customers.

First, our manufacturer. Our business is roughly divided into two key operating areas– sales, and service. The sales survey sent by the manufacturer has 6 numerical question categories, most of which are broken down into alphabetical sub-questions. In total the manufacturer is actually soliciting input from a sales customer on 23 (!) different questions, most of which are rated on a 1-10 scale while some are a binary “Yes/No”. The customer is of course invited to provide color commentary on these questions as they like, as well as on the survey overall. Similarly, the service survey has 8 numerical question categories but these are subdivided alphabetically so that the end result is 25 separate questions with 1-10 or “Yes/No” ratings.

The survey questions range across topics such as the timeliness and convenience of the business’s service, the friendliness and knowledgeability of staff, the subjective perception of the value given or fairness of charges, the perceived honesty of the process and people involved, etc., as well as the overall level of satisfaction and the willingness to recommend to others. Using a specific weighting formula (where some questions actually receive 0% weight, considered ancillary in nature, and others receive a relatively heavy weighting), the manufacturer arrives at a composite score on a 100 point scale (accurate within 1 decimal place) of the business’s overall “Customer Experience” index score. The bottom 2% of survey scores are thrown out at the end of each month and then the manufacturer provides bonus funds to the business if the composite score is above an arbitrary hurdle.

The national average for all related businesses in terms of both sales and service is ended up at four tenths of one percent above the hurdle for the year ended December 31st, 2015, and the hurdle is being moved up this year to one tenth of one percent above that!

The first interesting thing about all of this that I would note is the concept of false precision. The multiplicity of dimensions against which the business can be rated and the fractionality of the composite scoring system suggest an extremely precise, professionally-calculated measuring tool which itself suggests a customer experience that is almost scientifically specific in nature which, at end we would hope, reflects a consumer demographic that is nuanced, discerning and tasteful in character.

All three of these things are false. The measuring tool’s complexity is its own undoing in that customers rarely seem to understand what they’re rating or why (more on that below) and the surveys are sent out to the fraction of total customers who provide an e-mail at time of purchase, of which a still-smaller fraction actually bother to respond to the survey. Instead of measuring incremental behavior per thousand, for example, which might accurately capture meaningful changes in trend, the tool is instead measuring “fractions of a person’s experience” per tens in a given month in a given business… significantly meaningless specificity. The customer experience process is not as specific as the survey would suggest, many of the items being surveyed are accidents of history and essentially not controllable by the business without undue capital investment to change them. And finally, most of the customers are crude rubes who leave the business either gushing about how great it was, or pounding their keyboards in rage behind a Yelp review page trying to convince everyone that the business should be burnt to the ground and its employees mutilated on the public square in retribution for some minor slight or hiccup. There isn’t a middle ground and as far as the manufacturer’s scoring criteria is concerned, the middle ground isn’t valuable real estate anyway. As you will learn in a moment, there are entire categories of customers who don’t know or don’t care about many of the sub-questions on the survey which means the tool captures little more than their ignorance or angst.

The surveying system, both its conceptualization, construction and monetary reward system, betray a highly bureaucratic mind completely detached from both business reality and customer capability. The bureaucratic mind sees the world as a series of levers to be pulled, with no easy answers, simple solutions or “good enough” approaches. The bureaucratic mind seeks to measure everything, regardless of how valuable it is. The bureaucratic mind ignores the variability in quality and capability of human response (the customer) and tries to slice and dice a bunch of statistical averages rather than being merely curious about something resolute like “Were you completely satisfied? Why or why not?”

The fact that the survey system is tied to a monetary reward means there is a strong incentive for the business to find ways to game the system (coach customers — even if “illegal” — and input fake emails or remove them entirely when a bad survey is likely), especially as the manufacturer moves the hurdle ever closer to 100. The bar being set as high as it is (95) betrays both a kind of cluelessness concerning how simple it is for slight mishaps in the customer experience to bomb the score below that and an undue sense of ambition that a “truly great brand” would have nothing less than perfect scores. “If we just keep moving our standards up, our customers are bound to think more of us!” Meanwhile, setting a monetary reward above a hurdle turns the survey system into the equivalent of a binary “Were you/weren’t you satisfied overall?” despite the 20+ questions because anything less than the hurdle is essentially a penalty. And without a statistically significant sample size the manufacturer’s agents have no real place in advising the business’s management team about responses to perceived trends in the data.

So, what about the customers?

There is great confusion on the part of the customer about who he is responding to and what the consequence of his response will be. Many customers can’t differentiate in their mind between the manufacturer’s brand and the business’s brand, and a common lament when the latter of a pair of surveys sent from the couple is received is “I already filled out your survey!” Few customers who had a positive experience understand how important it is (for the economics of the business) that they register their complete satisfaction by completing the survey. And fewer still who had a negative experience understand that by completely bombing the survey they’re increasing the likelihood that their survey gets thrown out and therefore has no impact to the business whatsoever. These disgruntled customers also don’t understand that their individual complaints are read not by the manufacturer, who is only concerned with the statistical averages, but by the business they dealt with, as they are often filled with specific pleas to right some wrong or to put the business out of commission.

The way customers respond to the survey questions is also revealing.

Some customers reveal what angry, destructively vengeful people they are. They will rate the entire experience poorly (for example, rating a 0 for honesty of personnel) because one aspect of it wasn’t to their satisfaction (for example, the product wasn’t received in the condition expected, or they paid more than they would’ve liked, etc.) Or they will rate negatively and cite as their reason a small slight or problem they could’ve easily brought to the attention of the business and had resolved with little cost or inconvenience. This suggests a personality obsessed with power and control that is easily touched off and uses the “tattle” opportunity as a kind of political leverage to punish the perceived wrong-doer.

Other customers will rate the experience a 7 or 8 with comments about never rating 9 or 10 because “nobody is perfect.” These customers seek to use the survey to make grandiloquent philosophical statements about the state of metaphysical reality and can think of no better place to register their beliefs than on a business survey. Their comments are edifying, perhaps, but again completely useless from the point of view of the manufacturer and the business being held financially hostage.

Some customers are incompetent. They will rate the questions all 10s and then rate the final “overall satisfaction” question a 5. When contacted, they’ll express surprise or confusion and say that they “gave you a great survey”, not realizing that final 5 drops the overall score down to a 90% and thus a failing grade, if they can even explain why their “overall” score was inconsistent with the rest of the data they relayed about the specific parts of their experience (they usually can’t). Others will put negative color commentary and express unresolved problems but rate the sections of the survey highly. Others will write very positive comments, including a willingness to recommend to others, but then provide mediocre scores, especially on the willingness to recommend question.

Then you have the “deep thinkers.” They will get extremely granular on every question, providing a specific rationalization for each score given. Sometimes, when questions ask for similar information about a part of the experience, they will take the time to repeat themselves at length but using slightly different words. One gets the impression of a person who takes themselves and everything they do much, much too seriously. Undoubtedly hemming, hawwing and head-scratching were the prelude to the pages-long survey submission.

Everybody shows a bit of themselves and their values with a survey, both the survey maker and the survey taker. The particular survey world I inhabit leaves a lot to be desired in terms of making the survey a useful, honest tool for managing my business. At the very least, however, it provides a good chuckle now and then in reading an inane response or contemplating the unknowable mysteries of the workings of the manufacturer agent’s mind that thought a 20-some item questionnaire would provide invaluable insight into the customer experience. Ignoring the signal that profitability sends in a competitive market, I guess it’s still better than some I’ve heard about wherein the manufacturer’s scoring system revolves around customer responses to the prompt, “Can you imagine a world without [the manufacturer’s product]?”

That’s a real epistemological misfire right there!

What Drives Operating Metrics? Simple Things I Forget

A few weeks ago I was looking over the Morningstar.com tearsheets for about forty different mid-to-large-cap companies on a small assignment for an acquaintance. The project involved looking at the major financial metrics for a number of companies as provided on these tear sheets and trying to develop a quick, summary opinion of their business performance over the last decade.

On the one hand, it was just a bunch of series of numbers, going up, going down, sometimes not going much of anywhere at all. Depending on the metric being examined, this might be good, bad or ugly. But, if you looked a little closer, you might realize that even these simple numbers could tell a larger story about the business and what was happening to it or with it over time. You see, every financial metric I looked at was merely the result of the interplay of two, three, four or more other numbers “beneath” it. As one or some of these numbers changed, so, too, did the “top level” metric I was looking at on the tearsheet.

I spent a considerable amount of time, specifically, looking at the income and cash flow metrics, the reason being that most of these were large, stable and often growing enterprises– the balance sheet is always useful but with these kinds of companies what happens to income and cash flows over time can be an even more exciting and compelling story.

When analyzing an income statement, there are three key profit measures that investors like to look at: gross, operating and net profits. Additionally, when examining the balance sheet for the cash generating abilities of the business, there are two key measures investors focus on: operating and free cash flows.

If you’re like me, you sometimes forget the simple relationships these metrics have with other financial statement figures in terms of how one number drives another. Or maybe you’re new to studying financial statements and are looking for a handy reference. Either way, the following information may be helpful for you.

Let’s start with the income statement metrics:

  • Gross Profit, equal to total revenues minus the costs of goods sold; because of this, higher gross profits are arrived at by either increasing the price at which goods are sold (making revenues larger) or decreasing the costs of the goods sold (making COGS smaller)
  • Operating Profit, equal to gross profit minus the expenses incurred in running the business, such as paying sales costs (marketing, advertising, commissions), employees salaries and bonuses and corporate administration (SG&A), using PP&E and incurring depreciation or amortization expense and any costs related to the development of new products and services (R&D); because of this, higher operating profits can be achieved by lowering R&D expenses, lowering SG&A expenses, lowering D&A expense or raising gross profits by one of the methods discussed previously
  • Net Profit, equal to operating profit minus interest expense and tax liability; because of this, higher net profits can be achieved by minimizing tax expenses, reducing financial leverage and the interest burden that comes with it (or by refinancing existing debt at a lower rate), or raising operating profit by one of the methods discussed previously

If you are examining a series of income results for a company over a period of years and notice a variance in any of these primary profit metrics, look to the component drivers of those metrics to help explain the reason for the variance.

Now let’s take a look at the two cash flow items:

  • Operating Cash Flow, equal to net income plus the addition of all non-cash expenses (such as depreciation and amortization) and the net change in working capital; increases in operating liabilities increase operating cash flow, increases in operating assets decrease operating cash flow, higher D&A charges increase operating cash flow and of course, a higher starting net income increases ultimate operating cash flow
  • Free Cash Flow, equal to operating cash flow minus total capital expenditures; the less the company invests in maintenance and growth expenditures of capital now, the higher free cash flow will be, but if not enough is spent to protect the business’s earnings power and market position, long-term free cash flow generation abilities may become impaired

The operating cash flow metric is important to look at because it tells you whether the company actually generates a positive cash return on its investments from its main business activities. Companies that can’t generate cash from their operations over time are destined to financial and economic failure. Changes in operating cash flow when compared to changes in net earnings can give a window into how the company is generating profit– through business and market management, or through accounting manipulation and trickery.

The free cash flow metric is valuable because it shows the resources the company generates, beyond those needed to grow and maintain its current capital investments, which can be used to reduce indebtedness or reward shareholders with buybacks and dividends. Companies with relatively high and sustainable free cash flow-generating characteristics can be rewarding investments over long periods of time if they can be bought at low multiples to their normalized free cash flow.

There is a lot more to financial analysis than this. There are hundreds of ratios, metrics and other financial data you can use to measure the operating efficiency and management talent of companies you are interested in. This is not meant to be a comprehensive review. It’s possible I even missed a few items of note with regards to the metrics I singled out.

But sometimes I forget these simple truths when trying to think deeply about businesses I am analyzing, so I wanted to leave this little note for myself in case I ever get stuck and forget the obvious again. With any luck, this information will help someone else out in a similar fix, or else it will prove to be a stepping stone for a beginner making their first inquiry into the world of business analysis.