The role of statistics in web analytics

In analytics, I think there are some big misunderstandings out there in the world, to put it lightly. I think that one of the biggest misunderstandings is about the role of statistics in the world of web analytics.

What role should statistics play? Can we call this stuff we do “web statistics?”  Are statistics people good hires for web analytics roles? Is Stouffer’s better than Kraft?

Definitely.

I mean definitely about the mac ‘n cheese. If you leave it in the microwave for an extra 20 or 30 seconds, the cheese around the edges gets burned and turns into one of the 3 tastiest things on the planet.

When it comes to web analytics, though, I really don’t get all that excited about statistics. I should say that I’ve taken a bunch of courses in statistics and feel pretty comfortable with it, but in the last 10 years, I can’t tell you a single time that statistics was the deciding factor in making any of my big decisions about web sites and the companies that run them (minus multivariate tests, etc., where the stats are automatic, not done by me).

With that said, I do think that a working knowledge of stats can be critical at times. A lot of what we do revolves around working with an information set that may or may not be enough to turn gut feel into proof, and it’s important for us to know the difference between (and document) when we’re going on hunches vs. data. But here’s the reality: you’re going to have to go on educated hunches a lot of the time (in addition to using many qualitative and “non-data” sources of information); otherwise, you’ll die waiting for 10,000 clicks on “decorative wool seahorse sweaters” to get a statistically-significant conversion rate.

I just feel like having a stats guy around for web analytics is like having your accountant follow you around 24 hours a day, documenting your donut, grocery, movie ticket and magazine purchases. Yeah, he’s helpful when you’re ready to buy a car or get a mortgage, but there’s no need for him to tell you about the tax advantages of Caramello vs. Rollos, and frankly, it would just be annoying as hell.

Analytics people are self-sufficient businesspeople, at their best, and it’s a fluid blend of skills that makes them talented; not one more than another, and definitely not statistics as a highlight. Would you say that financial analysts’ key skill is statistics? No. Their ability to combine incredible amounts of qualitative and quantitative data into their brains and come out with simple next steps (buy/sell/hold + how much) is what makes them valuable, and dare I say our output isn’t all that different from theirs. We are just prioritizing efforts, identifying problems, and asking the organization to invest or divest in specialized thinking and tactics to solve problems. We may be more directly involved in that thinking and the tactics/execution, but it’s still essentially the same idea.

When it all comes down to it, you can be an excellent statistics person simply by reading books. Not so with analytics. Even the mighty analytics trident, Web Analytics Demystified, Web Analytics: an Hour a Day, and Web Analytics 2.0 fail to create great analysts. It seems that great analysts are born, not made. Or, I should really say that they are born and then made, and these books do help a lot with the making part. But these books are completely powerless to teach people how to care about cash flow, operational excellence, and real business. They certainly train you to wield tools and information with great power, and they may be able to get you to create a dashboard that shows business vitals, this doesn’t mean that the “analyst” has the communication skills, sales skills, appreciation of multiple viewpoints, and hunger to do this job well. Perhaps I’m wrong and we can re-shape people, but in my experience, it’s pretty darn difficult.

OK. I know I’m doing statistics a huge disservice. I know there are brilliant statisticians who can correlate capital markets to the temperature of Senate seat cushions, but let’s be realistic. Those people won’t even like web analytics, and we probably won’t like them either. I don’t want to downplay the trade. I just don’t think it’s the right one.

I think the real risk with statistics is when stats-oriented people are placed into positions high-up in the analytics or web operations ladder, the thinking being that their skill set will strengthen the gut to data ratio. I don’t think this assumption could be farther than the truth. That ratio is not going to change. What will change is the mindset of the organization. There will be an obsession with correlations and significance. There will be a paralysis of creative solutions. There will be a gaping void where intuition used to be. OK, maybe that’s dramatic. But it’s likely.

You can also weave data into the story you want with statistics, and with surprising ease. If you have any doubt about that, read Freakonomics, where some pretty mind-blowing alternatives to “statistically-proven” explanations are offered. Statisticians are often brought in to illustrate a point with data, as if data were crayons with which we draw the pictures we want to draw. I’ve been there. And it’s very possible.

Good people are hard to hire.

Here’s an exercise to illustrate a point.

Get on Monster or Yahoo Jobs or something like that with the goal of hiring two people. First, a statistics ninja. Second, an analysis ninja (click it — I’m guilty of phrasing this the wrong way in the past). In 30 minutes, see how many people you can find who you’d actually be willing to hire for those roles.

I’m willing to bet that you’ll find about 100 statisticians, and between 0 and 2 web analytics people.

Yeah, yeah, go ahead and tell me all about how much longer statistics has been around, and how it’s a degree, and all that crap. ZERO to TWO. 98% fewer qualified people. It’s not because of the education or this history. It’s because one is a tool, and the other is a trade. You can go to Sears and buy a hammer. You can’t go to Sears and buy a carpenter.

Indiana Jones - This swap didn't work, either

Indiana Jones - This swap didn't work, either

So, my point is that these people are not exchangeable; not suitable substitutes. And my greater point is that they should be used effectively as incredibly powerful tools to support a craft. I really dispute putting statistics people in web analytics leadership roles, again, because I think it promotes the wrong mindset and tilts the scale out of balance toward one of the sides of web analytics that is ultimately not the most effective, productive, or change-inducing. Statisticians are great in support roles, and can be used to great effect on projects, to make things more certain, or to help solve parts of complex problems. But even then, what is their career path if they do this well for a year or two? Is that a fair shake for these guys? Will they really transition from tool to trade?

And now it’s your turn: burn it up! What do you think?

Oh yeah…one more thing. No, it can’t be called web statistics.


No Trackbacks

3 Comments

  1. Tim

    Very well put. I couldn’t agree more.

    Posted September 15, 2010 at 3:32 pm | Permalink
  2. Wow, are you trying to create an awesome lead-in for my eMetrics DC presentation? 🙂

    Ok, here is my view, and it most agrees with your own. A statistician is NOT a web analyst, and in my experience, wouldn’t be a good substitute. However, as a part of an integrated team, I think there’s value.

    My current team has both web analysts and statisticians on it. The statisticians help to validate findings (statistically), create forecasting models, and do incredibly deep analysis. They’re also pretty much data people – throw anything at them, they can model it. But they normally don’t know what it means or why it matters. I’m sure there are more business-minded statisticians out there, but the approach that we take is to let statisticians do what they’re good at, which is to dive deeply into data, and let the web analysts be the conduit between the stats and the business users.

    Our web analysts are our subject matter experts. They know the site, they know how it’s tracked, they know why it matters if page x drops in PVs/Visit, and why that is important for revenue. They analyse the site, recommend optimisation ideas, assess the success of site changes, forecast the impact of things that haven’t even been tested yet. They tie together multiple data sources because they understand them all, and analyse our site holistically. They help to guide statisticians about the business when engaging them – they’re the conduit between “this number went up” and “but what does it MEAN?”

    Our VP is more of the statistician/economist variety, and the reason why our team works is that he speaks both languages. He IS a business-minded person, who also has the stats skills. So he can speak both the language of the business and the language of the statisticians. I’m not sure that, without that person, it would necessarily work.

    As for web analysts, I think we need to understand how statisticians can help, and understand what they do and what it means. For example, I’ll be called upon to present an analysis, to speak to the business impacts, but I have to be able to understand what the statistician did to do that! The difference is, I don’t need to be able to pick it up and run with it in their absence. I don’t have to do the math, but I have to understand it.

    A few years ago, I didn’t have this same experience. But now, having had it, I argue there is great value in incorporating both on a team. But I still agree with the central point – they are NOT the same, and should not be expected to be.

    Posted September 15, 2010 at 3:52 pm | Permalink
  3. Ned Kumar

    Hi Evan,
    Good post – and I agree that number crunching and statistical significance are not the only things needed to be a good analyst. However, I am going to play devil’s advocate to your reasoning and defend the statisticians a bit 🙂

    1. First off, I feel it is not fair to generalize statisticians the same way it is not fair to generalize analysts. True, there are statisticians who are deeply entrenched in mathematical theory and buried in the world of distributions and parameter estimations with absolutely no sense for business. These are definitely not good candidates to cross-over to the business or analytical world. However, there are statisticians who have excellent grounding/background in statistical concepts and also is very well-versed about the business marketplace and needs. This set can make excellent web analysts (imho) given the right opportunity and even take on a leadership role.

    2. I like your example of “You can go to Sears and buy a hammer. You can’t go to Sears and buy a carpenter”. However, the problem I see many a times is that folks hire carpenters with myopic skill set. Meaning, there might be situations that call for the use of a “rubber mallet” but their carpenter will still go ahead and use a “metal hammer”. Point being that contextual factors play a role in the decisions and knowing the choice of tools/techniques available and how to use them given a context is very critical.

    3. I maybe in the minority, but I do think statistics plays a very important role within our field of web-analytics. This is especially true in the world of “qualitative” web analytics & campaigns. Now, I agree with you that “you’re going to have to go on educated hunches a lot of the time “. Very true, but there are a lot of statistical methodologies that can be used to ensure that these educated hunches are really based on “education” and not just “hunches”.

    4. And lastly, I am going to twist it around and make a statement that there are emerging areas of web analytics where you cannot do without statistics. Two examples here without going into detail are predictive web analytics and social analytics.

    Anyway, to me the label is of no consequence when it comes to hiring – it really comes down to an individual and his/her acumen in various analytical, quantitative, and behavioral areas.

    Best,
    Ned

    Posted September 15, 2010 at 5:15 pm | Permalink