Is ROI the analyst’s greatest frienemy?

In the world of marketing and business, it’s all about ROI, right?

Well, not if you want to understand what is happening and what you can do about it.

Extremes are our enemy. Typically, we get tangled in the extreme of over-complication. We want to run statistical analyses, regressions, econometric models, and the like on our data, even sometimes on relatively simple data sets. This is when we alienate our audience by trying to look super smart rather than keeping the story simple and digestible. When we involve all of this mathematical alchemy, we can lose touch with the underlying business questions and get lost in the numbers, muddying the message.

On the other hand, overly-simple is equally a pitfall. That overly simple view typically boils down to the viewpoint that everything has to have a demonstrable ROI. And that, as they say, is poppycock.

I can’t prove this, but I think that on any given day, the vast majority of your “customers” (visitors on your site, people on your Facebook page, etc.) are in the high-funnel. Maybe not even in the funnel at all. They are just beginning their search for a washing machine. Just researching what replacing their old digital camera will get them in terms of picture quality.

In our media mix, we have keywords, emails, social engagement, display creative and more targeted to the high-funnel. Remarketing campaigns have the sole purpose of bringing customers back after an “unsuccessful” visit. Keywords like “best digital cameras” drive consumers to commerce sites, when the consumer is almost certainly looking for content to help them narrow the field and is not yet a buyer.

And what do businesses do? They measure it all to ROI. Spend vs. return. They see everything through the lens of our business, rather than through the lens of the consumer. And that’s really just not the best approach when you want to understand what will lead to greater financial success in the future. This view just doesn’t give you enough detail into why things are working or not working or what you can do about it.

A consumer’s interaction with our business is a chain of micro events that hopefully leads to macro outcome in terms of a sale, a subscription, a lead, etc. It’s in these micro events that our business “leaks” customers. And the only way to see this leak clearly is to zoom in to each element and measure the business’s success at that level.

I think that luxury hotels offer a great metaphor for how this works. Luxury hotels want your loyalty. And the way they earn it is not a single coup de grace, it’s a sequence of tiny little events that all go perfectly.

When you arrive at the Four Seasons, for example, the valet will open your door and ask you if you are checking in (unless they recognize you, in which case they will welcome you back). When you give him your name, the valet will wait for you to turn around, immediately radioing to the front desk where they will look you up to see if you are a repeat customer. The desk then immediately radios back to the valet and doorman. By this time, you’re at the doorman who is swinging the door open. He says, “Welcome back, Mr. Smith.”

Wow.

Throughout your stay, you notice everything. The flowers in the lobby. They politeness of the front desk clerk. The professionalism of the bellman as he asks if you’d like your hanging clothes to be put into the closet for you. You notice every aspect of the room: its cleanliness, the smell, whether the clock on the bedside table is correct.

When you come back out into the hallway to head out for the afternoon, employees you see in the hallway step aside to let you walk by. They push carts out of your way and everyone greets you and wishes you a good day. On the way out the door, they will hand you an umbrella if rain is in the forecast and may offer you a bottle of water and directions, if you need them.

And this goes on and on throughout your stay.

Now, here is the thing. Each of these actions has an ROI, it’s true. But each of these things is partially or completely dependent on all of the others. The ROI of the doorman can’t be localized to the doorman. If the room smells funny, your entire experience could be marred. Your opinion of everything else changes because of one “leak” in their system.

With your web site, your social media, your marketing channels, etc., it is no different. The ROI of your marketing channel is completely dependent on your landing pages, your product detail pages, your cart, your checkout process, everything. Yet we measure this channel to an ROI. And going even a step farther, we measure even the high-funnel segment of this channel to ROI as well. These being people who are not there to convert; they are there for a different micro case you could measure and optimize against.

The only way to be an effective analyst is to intimately understand the role of each of your micro-events and find opportunities to improve. Almost zero A/B or multivariate tests should have sales, revenue, or leads as the goal or measure. There is simply too much in-between — this is not to say don’t look at it, just realize that any single interface is rarely intrinsically tied to the end-game. You need to look at the pieces themselves. Make each piece serve its function perfectly. As you eliminate leaks on the micro level, the macro level will take care of itself. Faith-based initiative? Sure. What isn’t?

I know that the culture of ROI exists in most companies and your colleagues aren’t going to want to talk about sissy pants micro conversions. But that’s where the magic is. Your customer reacts to every little thing presented in your marketing, your architecture, your interfaces, etc. Consciously or not, your customers judge you and interact with you by the micro. So if you want to keep more customers and make your existing customers more valuable, judge yourself like they judge you.


No Trackbacks

7 Comments

  1. Evan, get on script. It’s not the metric, it’s how it’s calculated. If it excludes long-term and brand-equity type assessments, then it is misleading for sure. When those are included and when put in the hands of benevolent dictators, it can be very useful.

    Posted February 29, 2012 at 4:29 pm | Permalink
  2. There is no script! It’s the wild west out here.

    Would like to hear more of your thoughts on this. I’m essentially saying that ROI (most any calculation of it), is not how you enhance your marketing, your site, or your business. At best, it allows for “fat trimming” or reallocation, but gives you very little insight into the subcomponents that lead to different levels of performance. Reallocation assumes the components themselves have maximized intrinsic value (or that the cost/benefit of reallocation outweighs that of fixing the touch points). The model I’m throwing out there assumes the opposite: the components themselves have a potential intrinsic value that is much higher than what we see today, and that’s because the intermediate “micro events” are handled poorly by most businesses.

    Even if a marketing tactic can be scored against brand equity or long-term “net effect” ROI measures, the touch points that live in between the marketing you’re measuring and the endpoint of value generation are what need to be individually assessed to maximize the potential to generate value in the first place. Like in the metaphor, when measuring the ROI of the doorman, whether it be direct revenue, brand equity, or long-term value, there is no metric I’m aware of that can point directly to a different component in the ecosystem that is causing a problem. If ROI of doorman has been depressed because of a customer’s interaction with housekeeping, I’m not aware of a metric that can show that. Perhaps a series of metrics all beside each other?

    Only when there are zero leaks in your system can you get an accurate picture of any ones component’s contribution to the whole. And it’s my feeling that we will never reach the day of zero leaks. But that doesn’t mean that we won’t reach the day (or aren’t already there) that reallocation has the clearer cost/benefit.

    I’m certain people would appreciate your perspective if you have a few minutes to share. Can you go into a little more detail on your view?

    Posted February 29, 2012 at 5:00 pm | Permalink
  3. The “art and science” phrase comes to mind here, doesn’t it? As does “cognitive dissonance.”

    Starting with the latter, marketers have known/said/preached for years that marketing messages should be integrated and congruent across channels. The sum of the parts is greater than the whole. If you have a polite doorman AND the room smells nice AND there are fresh flowers in the lobby, then your positive impression is likely greater than if you added up the incremental lift in your impression for each of three experiences where only one of these factors was great while the others were only okay. We know that. But…then we want to measure the specific contribution to the overall experience of *just* the doorman’s behavior and *just* the smell of the room. Cognitive dissonance.

    (BTW…you have a lot of detail about the Four Seasons. Are you a 1%-er in disguise?)

    So…art and science. As the hotel manager (the “marketer”), you’ve got to be clear on what your goal is, right — to provide an absolutely exceptional experience (…without breaking the bank — complimentary diamond-encrusted toothbrushes probably aren’t feasible)? So, you’ve got to evaluate everything you can control — the training (and compensation/rewards model) for every member of the staff, the flower delivery service and schedule, the room-cleaning process, etc. — and determine *how* it can contribute to an exceptional experience, and then identify appropriate measures that can be used to measure and optimize that “contribution” at a micro level as efficiently as possible.

    Am I restating your point effectively?

    I took a crack at this from a slightly different angle a while back — looking at it more as a “complex B2B process with subprocesses” than as a whole series of potentially non-serial micro conversions. The micro conversions mentality is more robust and, I think, a better representation of reality. But, if you’re interested (it’s not a long post): http://www.gilliganondata.com/index.php/2008/04/10/complex-processes-and-analyses-therein/

    Great stuff (as always!).

    Posted March 1, 2012 at 10:20 pm | Permalink
  4. Slightly different take / related idea – there are far too many claims of ROI being made where ROI cannot be measured properly.

    Web analysts spend way too much time declaring something to be “measured” when actually they are only “tracking” it. There is nothing wrong with telling someone “I cannot measure the ROI of that because…” then suggest the proper way to do it, including any new tools or techniques that would have to be used. Long run, this is a far better approach than providing “faux confidence” to those using the output.

    Take the latest shiny object in WA – attribution. Just because someone was exposed to a sequence of events does not mean each had a positive impact on the end game, never mind the fallacy of assigning “weights” to these events. You need control groups and probably media mix models to get even close. Nothing wrong with telling people you can track, but can’t really measure, without more resources.

    I’d like to see a “rude doorman” test, for example – what is the incremental lift on customer value at the doorman level, for the stay itself and over the next 6 months, of a “nice” versus “rude” doorman?

    If the company does not have the stones to do a rude doorman test, then ROI can’t be measured. Nothing wrong with that, it just is what it is. Going forward, the company simply “believes” a nice doorman to create value – but they can’t get to “ROI”.

    Posted March 2, 2012 at 10:21 am | Permalink
  5. Evan, I would mostly agree with you (yikes!) except for one thing. Marketing is all about the message. That means you focus on coding your data into essential marketing messages (e.g. one might be a price or promotional message, another a branded message and still another focuses on some product features). Comparing the performance of one message v,. another is important and is a forward looking application.

    Also, I agree with Jim Novo. Tracking is not ROI. In fact, metrics are not ROI either. Marketing ROI really is the most abused term in business today. I find it humorous that a large volume of mentions of the subject are by people who do not have a clue what it means.

    Posted March 14, 2012 at 1:00 pm | Permalink
  6. Yikes! :)

    Agreed. How do you feel about this: marketing (in the broader sense) is about the message AND the brand’s ability to make good on the message’s promises and expectations. The second part is what this post is really about. I believe that brands who analyze their ability to actually meet consumers’ goals (both micro and macro) and live up to the marketing hype will realize a “same store sales” effect. It is always more powerful, and more scalable, to improve the profitability and productivity of all efforts by optimizing the funnel itself, not just the action of dumping into the funnel.

    Posted March 14, 2012 at 2:07 pm | Permalink
  7. Geez, Evan, I agree with you on two things today. Double Yikes!

    Posted March 14, 2012 at 3:41 pm | Permalink