How to measure web engagement, for real.

Ever wonder why “engagement” and “enragement” are spelled so similarly? Measuring engagement is tricky business, made even trickier by the use of esoteric indices and web-centric (as opposed to economics-centric) metrics to describe the concept.

So, here are some thoughts on how to measure, evangelize, and create delta in engagement.

First things first

We have two key issues in the area of measuring and expressing engagement to our businesses:

  1. Most of the makeup of “engagement” consists of metrics that have not been translated to economic value. This requires extra explanation as the measures are passed around the organization, and may alienate certain very important audiences.
  2. Largely, the types of “engagement” indices and calculations I’ve heard of are typically non-predictive in nature. They do not fold into or constitute a model that can be used to calculate the return of incremental investment or a shift in budget allocation* across media or resources. Typically, these engagement models are used to show delta in an index when delta in revenue or ROI is hard to create or isn’t enough for a pushy organization.

* Shifting your budget allocation based on distribution of historical ROI values is a highly unsophisticated long term strategy (fine in the short term) because it fails to consider the impact you can have on ROI by delivering better messaging and a better experience (read more).

It’s the economy, stupid!

The first thing to deal with is a nugget of wisdom made popular by James Carville. Fortunately, it’s a concept that the measure (argg! hate that word) community is transitioning toward as we speak: making measures economic in nature. Tweets, Likes, Pageviews, subscriptions, shares, etc. are the language of web metrics, and the good news is that they are all worth something. We just have to figure out what.

So let’s get started. The first thing to lay groundwork on is that there are two great, pre-existing models we can use to estimate the worth of something.

The first one is obvious: real return, which describes the actual dollars we get in a transaction. In a paid search campaign, we can see the money that the campaign drove in real terms. In truth, we only see a certain percentage of the money, because real return models don’t intrinsically consider ripple effects: that a single click could lead to a positive experience, a sale, lifetime value that can’t all be captured in our data systems, the potential for word to spread to their family and friends, etc., etc.

Real return is both a blessing and a curse: it gives us cold, hard data, but it also impregnates our businesses with the idea that everything is suddenly measurable in cold, hard terms. That is completely inaccurate.

The second model is probably the one that will be more powerful for us in translating engagement into value, and doing so in a predictive way: willingness to pay (WTP). WTP describes the investment a business is comfortable making to get a desired outcome, often an outcome that is non-financial in nature, or difficult to translate to financial terms. Investments in radio advertising, for example, are based largely on willingness to pay: “We can deliver your message to over 300,000 listeners in our lunch time block.” The advertiser does some back of the napkin math to figure out what the financial impact of 300,000 listeners may be (the business owner obviously wants some sort of a return), and they decide on whether the ad is worth the cost.

Truthfully, WTP models can and should be derived from real return data, but getting too obsessed about turning WTP into real return will bring your decision making to a halt. You must fight the urge to get stuck in this trap; keep moving.

Getting your Financial Figures

Now, it’s time to figure out how to put dollar signs on your tweets, likes, etc. One of the best places to start (if your business is large and does traditional advertising) is by finding proxies in the traditional advertising side. If a single post on Coke’s facebook wall reaches an audience of 3mm viewers (measurable), and is liked/commented by 10% of viewers (measurable), and that is seen by their friends (measurable), you can estimate the total reach. How much does it cost you to get that same reach with a billboard?

Now add a little premium to that figure because a) facebook is an interactive medium and builds affinity and b) you may have learned something about your creative or brand from the comments you received, which has value far above and beyond what non-interactive, traditional media can provide (you have to pay extra for the focus groups, in fact).

If you aren’t a big place with huge advertising budgets, you can just investigate what it would cost you if you did. Make a few phone calls to the newspaper, TV station, radio, or an agency, and see what you will get for what price in terms of reach for an audience that fits your target market.

You’ll have to go through your whole organization and look at each action, trying to understand the value of each effort (you will probably be using a mix of real return and WTP/proxies in each scenario):

  • What am I willing to pay for a paid search click, email open, or display impression? (ROI + extra lifetime value + word of mouth potential)?
  • What am I willing to pay for an additional developer? (cost or incremental revenue of releasing a feature or product earlier)?
  • What am I willing to pay for a usability person or designer? (potential upside of a range of basis-point deltas in conversion rate [a fancy way of saying build a matrix] for applicable traffic sources).

ROI is NEVER the answer to any of these questions because ROI fails to consider anything but a moment in time. On the WTP side, don’t forget to factor in other costs like salary, resources, efficiency loss, etc. in addition to the hard cost of the media or whatever it is you’re looking at.

No, this is not going to be something you do in a week. But it’s also not something you have to do for your whole company before you can start making predictions. Take off a few bite-sized pieces and see where you get with it.

Making it predictive

Now that you have your models for valuation and a method of getting some numbers to work with, it’s time to make sure that your model is a predictive one, not a backwards-looking one.

Creating engagement data for the past is easy. Creating it for the future is much harder. The good news about a WTP model is that it is pretty forwards-looking, though: you are focused on what an incremental set of eyes, an incremental sale, etc. is worth to you. If you are stuck in the ROI trenches, you will never be able to understand what a sale is worth, and you will never be able to state what you are wiling to pay to create a new customer or rekindle the flame with an old one. As long as you get things back to WTP, you will have a predictive model on your hands (it would take me 20x the length of this post to actually start discussing a viable model, so if you’d like a post that dives into real models, please leave a comment!)

Delivering the message

Now here’s the critical part: when you deliver the message, deliver a prediction, not a predictive model. Use your model; don’t ask others to use it. Here are a few examples:

  • “Mr. CEO, our work has turned up an opportunity to bring in an incremental 5% market share with a $1.75mm investment, all in.”
  • “Mr. CMO, we have found a way to reallocate 10% of our radio budget to social media and improve that budget’s reach by 50%.”
  • “Mr. COO, we have found that there is a significant tradeoff between process and productivity in the IT department that we estimate has cost us $7.5mm in revenue in the last 2 quarters. Some changes in our workflow can close that gap in the next 6 months.”

Also, deliver the right message to the right people. CEOs care about stock price and company valuation, which means cash flow, market share, and innovative things to talk to the world about. COOs care about efficiency and good ideas that will impact the business directly. CMOs care about using their budget effectively and being able to show the value of their existence in terms of changes since the last CMO was fired 3 months ago.

Your mileage with your executives may vary, but keep in mind that different executives come with different goals and perspectives. Try to guess what motivates them and reposition your idea in a way that suits their personal or professional goals.

What happened to engagement?

Nobody gives a crap about engagement. Engagement is just a stepping stone on the way to “worth,” and it’s a non-predictive stone. Once you’ve moved on, you’ll never look back.


No Trackbacks

7 Comments

  1. Jason Paulsen

    Hey Evan,

    Very thought-provoking. I would love to see a post based on the models you have created, even if it’s 20x the length of this one. 🙂

    Posted August 3, 2011 at 10:51 am | Permalink
  2. Good article, been there and done that.

    We have a metric called SEI for Social Engagement Index.

    1. We have proof that it correlates to sales and its impact/ROI can be measured.
    2. The key insight from this is it’s all about the message(e.g language/linguistics) and not the number of friends or followers.

    For those interested go to my LinkedIn profile and download paper “Social Media Metrics & the Holy Grail”

    Also, Evan, don’t be down on measurement. Marketing effects are both short time and long-term. Both can be modeled or measured

    Posted August 3, 2011 at 11:16 am | Permalink
  3. I’m with Jason on this one. I would certainly like to see more on this subject, please. Maybe a series of posts breaking down different examples.

    Anything which gets us closer to some credible financial value for social marketing is extremely interesting.

    Posted August 3, 2011 at 5:40 pm | Permalink
  4. Great post! “WTP” is officially now part of my lexicon. On that front, though, am I right in interpreting that WTP is, at least in part, a means of illustrating that it’s not necessarily reasonable to expect the link from investment to revenue to be wildly more perfect in the digital world than it is in the offline world? It’s a way to say, “You’re paying X to get Y through another channel, so let’s look at delivering Y at a lower cost or delivering more Y for the same X.” Right? WTP as the underlying concept is a more elegant way to frame the discussion I regularly have with clients regarding social media ROI (not that I will start using “willingness to pay” as a phrase, but it certainly helps focus the structure of the discussion).

    Posted August 4, 2011 at 8:59 am | Permalink
  5. I think you are right on, Tim. WTP as a concept is that in the absence of real return (or even better yet, in the presence of real return, but also realizing that there is value BEYOND the real return), you can still value an outcome appropriately.

    The best WTP and distribution models are based on some things we can measure, a la econometric modeling. But if you don’t want to get that complex, traditional advertising (among many other things) provides a great example where we can prove an equal or superior return. I think the facebook example in the post may be one worth pursuing. I saw a ridiculous article about how Coke is blowing money on facebook, but the author never considered what that same reach would cost in “the real world,” nevermind that the reach in the real world comes with zero proof of impression levels, zero feedback in terms of effectiveness, and zero sharing potential (in fact, most great billboards are probably shared via a photo share on a social network, rather than telling your friends to drive downtown to see a sign!). Dollar for dollar, facebook is better for Coke in every way, and a social media practitioner has every right to ask for and demonstrate the upside of budget redistribution.

    Keep in mind that these are just the simple leading metrics, though (impressions, likes, etc. — it keeps the digital to traditional comparison simple); the important ones are the financial outcomes. WTP should always have its roots in financial outcomes, and it always can. That is the business that Michael Wolfe (also a commenter) is in, and it’s something we can learn a lot from. But the truth is that the financial outcomes are often guesses or reverse-engineered correlations, meaning that oftentimes the econometric models aren’t created from scratch, but are derived from real world behavior and tests by running different media mixes or tactics in separate, but similar DMAs or other segmentation methods.

    OK, don’t want the comment to get longer than the post, but hope it all helps!

    Posted August 4, 2011 at 9:17 am | Permalink
  6. Thanks, Evan. We actually have an “Equivalent Impressions” calculation that we do for Facebook for this every reason. But, we only use that when it ties back to WTP. A client that lives and dies by impression measurement and is comfortable with that as a measure is one where we can roll it out and use it. It always makes us a little nervous, though, because, as you note: 1) it’s upstream, and 2) it’s pretty distant from the “real value.” We lead with, “What are the objectives, and what sort of logical model — if we can’t build a true, hard numbers model — can we construct to map measures of the initiative to business value.” For brands that are bought into the fact that they should be engaging with their customers and prospects in the channels where the customers and prospects are spending their time…this is a positive and fruitful exercise. Hah! I guess this is a one-off take on your subsequent post — it’s situational for us based on the client and how they view their business and the problems they’re looking to us to help them solve!

    Posted August 4, 2011 at 9:56 am | Permalink
  7. I’ll get started on a follow-up post, or a series. I have a few others in the hopper, but I’ll get to some real-world examples, and if there are any specific scenarios (like social) that you’re interested in discussing (Jason, Tim, or anyone else please chime in), I’ll focus on those.

    Posted August 4, 2011 at 5:02 pm | Permalink