Pegs & Holes – What “big data” is really about.

There’s a lot of talk about “big data” floating around these days. The next magic bullet, so they say. But what is big data? That’s what everyone’s trying to answer.

First of all, there are two axes of what it seems people are collectively calling “big data” today. Probably more, in fact. But the two simple axes are breadth and depth. There are big data systems like Amazon, where they have outrageously detailed information about individual users’ purchase behavior, product correlations, multi-year buying trends, etc., all in a single source with all data already joined. This is DEEP data. Then, you have big data systems where behavior from multiple systems is being brought together under one roof and joined with common keys of some sort. This is WIDE data. Wide data may have one or more data sources that are also deep, but the depth rarely has anything to do with how well wide data can integrate (you either have usable common keys or you don’t).

There is a lot of hope placed on big data. To what end, though? What is this all about? Justin Cutroni posed this question on Google+ this morning, and it’s sort of amazing that this industry is investing (mentally and monetarily) so heavily in an idea where few people really have discussed outcomes. There was a short discussion on twitter that said something to the effect of:

Duh…when you have more/better information, you can market better.

Academically, sure. In real life, I don’t see it.

This is exactly the hope placed on big data. We hope that there are answers on the other side of big data. This isn’t unlike me buying a pair of running shoes hoping that there will be more running on the other side of my purchase. It doesn’t happen. We invest based on correlation, and my fat ass is still on the couch a month later.

There are many, many examples of businesses who do extraordinarily well, whether they have big data or not. I think this is because they are in sync with the consumer. Big data is certainly one way to get information that can be used to get in sync, and will likely prove to be one of the best, but the business still has to get in sync. Some businesses will spend millions consolidating information and do nothing (a circle peg through concrete). Some will merely learn how to more effectively message to their audience with creative and timing (circle peg through a square hole, in many cases). The best will use big data to drive new and improved products and offerings (circle peg, circle hole).

There are so, so many cases today where consumer needs are simply not met by the present array of products and services. Marketers are so focused on selling their current portfolio, they seem to have completely forgotten that it is possible to make new stuff.¬†Startups do this every day — how many entrepreneurs leap from larger brands to solve for an unmet need? Great brands let these offerings grow. They seek information to improve the portfolio itself, making the sales and marketing process frictionless. And there are all sorts of brands that are in-between.

If all we do with this “360 view” of a customer is figure out when to poke them with a stick, that is pitiful. If, instead, we figure out what their unmet needs are, we are apt to start all sorts of interesting things. I like interesting things. Whether your data is deep or wide or both, you probably already have what you need. If you have the motivation to get in sync, you have all the ingredients you need. But if the motivation isn’t there, your shoes are going to sit and collect dust.