Feb

8

SiteTuner’s AttentionWizard Opportunities

Filed Under Web Analytics Tools 

The folks over at SiteTuners have been working on a great piece of technology for simulating eye tracking.  But be warned: it’s in beta and does come with a little bit of baggage so far.  But Tim Ash is one smart dude, and he’s out there gathering feedback on forums and blogs to make the tool a winner when it’s out for good.

People like tools like this because they are intended to solve arguments quickly.  We can use data from users or computer simulations to tell the bonehead CEO that his ideas of putting jumping pink kittens and 24 calls to actions are bad ones. And this is great, so long as the data is inherently valuable.

The trouble with AttentionWizard so far is that it evaluates web sites like you would evaluate photography or art, using design concepts like contrast, weight, etc., with an obvious dose of “top-leftedness,” meaning that the tool anticipates where headers and text would start and shows greater focus there. And it is also seemingly weighted heavily toward identifying outlier colors.

But that’s not exactly how people evaluate web sites. The classic example is banner ads and how conditioned people are to avoid them, regardless of how bright and obnoxious they are.  In AttentionWizard’s demo video, you can see this error in their demo of 1-800-flowers.com, where the house ad gets a lot more attention than it would in the real world.

But to be fair, this demo does reach good conclusions, and you can see how much more straightforward the end result is.  But the point is it reaches good conclusions with the wrong data.

Unfortunately, we live in a phase of web analytics where most of the tool data is taken at face value.  So, people see a heat map and believe that’s reality.  Or they get a report about keyword performance and assume that keyword A is better than keyword B, just because it has a higher ROI (but without seeing if the landing pages make sense, if the prices are competitive, and on and on).

Tim makes great points on his site about what this tool is and isn’t, but right now that’s like John Stewart making points about how his political opinions are just his own: it’s irresponsible because it ignores how powerful credible sources are, and to both Tim and John’s credit, they are credible sources because of their history (and charm, of course!).

Below, I’ve shown a few demos of where AttentionWizard deviates from human behavior so radically that it can actually drive the wrong decisions.  And I’m not trying to show pictures of half-naked women or anything that the tool shouldn’t or couldn’t technically be capable of, because I do think that’s an unfair test.

Test 1: Pattern recognition

Here’s one that humans will get right away.  Just try to take your eye off of the star.  AttentionWizard didn’t get it at all.

In the world of the web, this is important for low-contrast sites that might use shapes and graphical weight to point things out.  We probably won’t see an example this simple online, but it is possible. I’m envisioning package design – you see 11 identical-shaped shampoo bottles on Amazon.com and one crazy-shaped one.  Where does the eye go?

Screen shot 2010-02-05 at 2.07.57 PM

Screen shot 2010-02-05 at 2.08.14 PM

Test 2: Position-sensitivity

So here’s one where I’ve placed calls to action at 1150 pixels.  This one tests to see whether the tool will hold up to the boardroom when someone says, “See, people will see this button at the bottom of the page!” Unfortunately, it seems that the tool isn’t yet sensitive to where things are and what screen sizes are most popular, which is definitely something that needs work before AttentionWizard is ready for its close-up.

I think this is an easy win for the tool – something that shouldn’t be too tough to change.

Screen shot 2010-02-05 at 2.12.35 PM

Screen shot 2010-02-05 at 2.13.57 PM

So Tim is to be applauded on creating a great technology.  But I would strongly urge people to not circulate the data it renders quite yet, not because the tool is bad, but because data is notoriously misinterpreted in the average company – taken for face value.

People can caveat their tools all day long, just like Charles Barkley could say he was not a role model, but we have to be responsible to the real world and produce things that don’t send mixed messages with the expectation that people are smart enough to filter the data.  They aren’t.  Not yet.




Comments

6 Responses to “SiteTuner’s AttentionWizard Opportunities”

  1. Fabian Stelzer on February 15th, 2010 1:07 pm

    Full disclosure: I’m co-founder of a German neuroscience startup that offers a similar product.

    We’ve been researching human visual attention on websites for over a decade now – your tests actually mirror two of the most challenging mechanisms of neuroscientific attention prediction.

    AttentionWizard’s failure on test 1 is in line with what our collaborators Ahissar and Hochstein have proposed as the “reverse hierarchy theory” of visual attention: complex features like shapes are often processed and perceived faster than simpler features like color. The original paper can be found here: http://atar.mscc.huji.ac.il/~ahissar/files/Merav&Shaul_TCS2004.pdf

    Your second test might be even more important: position is a very powerful predictor of a user’s eye movements. Modeling effects like this is hard, but not impossible. However, you have to go beyond classical “saliency map” algorithms.

    I’d like to point you to our EyeQuant technology, which does precisely that: http://www.eyequant.com

    EyeQuant also predicts eye movements on websites, but achieves more than 90 % of an eye-tracking study’s predictive performance. This is only possible since EyeQuants attention model is based on actual eye-tracking studies with over 400 human subjects. I’d be glad to elaborate on any details of our evaluation and method.

    We just launched our english web service and are currently offering free test accounts – if you’d like to give EyeQuant a try, feel free to get in touch.

  2. evanlapointe on February 15th, 2010 1:42 pm

    Thanks for chiming in. I’d love to try out the tool and will be in touch.

  3. San Diego short sale on April 23rd, 2010 12:28 am

    Hey mate, thanks 4 writing but this post is hard to read when using Netscape it is is overlapping.

  4. evanlapointe on April 23rd, 2010 10:59 am

    Hmm, that sucks! Although it’s probably time to upgrade your browser…

    http://www.getfirefox.com

  5. Dennis on July 22nd, 2010 7:58 pm

    I saw this tool today and thought it would be a great addition as a tab in our software. Its like a reminder of what to test.

    But seeing these results… auch not ready for the real world.

  6. evanlapointe on July 22nd, 2010 8:27 pm

    Dennis,

    Don’t be too gun-shy. It is a powerful tool, but you just need to know how to interpret the results, and where it is different than human behavior. Give it a shot, and talk to Tim Ash, he is a great guy.

Leave a Reply