Overview Of Eye Tracking At Yahoo


Published on

I presented this deck at the Bay-CHI Birds of feather meeting at Yahoo on June 22, 2010.

The deck provides a brief history of eye-tracking and the the kinds of decisions we make using the method.

1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • We did a number of different Experiments including changing the way certain terms are bolded -- showed significant double digit % increase in clicks, and user engagement – which helped us quantify the impact, and understand more about why’s
  • People make a number of subconscious decisions. In this case this abstract would have significantly eroded perception of relevance, and increased user frustration.
  • Even though users said that they liked the design with the image thumbnails – the actual behavior indicated that users tended to avoid the images because they thought of them as ads or irrelevant content. Additionally there is a certain amount of cognitive overhead associated with switching contexts from scanning text to scanning images – as a result we decided not to try to change habits -- because in this case the behavior is hardwired into users.
  • We were able to quantify the effectiveness of a user interface using eye-tracking. Senior leadership was not interested in launching search assistance, until we were able to prove a 14% increase in page effectiveness – at which point the Search organization quickly agreed to support a launch. Yahoo launched Search assistance in 2007 to rave reviews. This significantly improved user experience and market share. Google followed a year later. Search assistance is now a standard feature for anybody searching
  • Eye tracking predicted a 6-8 week learning curve for search assistance. Once launched click logs confirmed the findings. The Click log findings on learnability were presented at SIGIR 2008 ( A longitudinal study of real-time search assistance adoption, Peter Anick, Raj Gopal Kantamneni ) Planning for learnability is critical for the web because the cost for switching is minimal. If something takes 20 weeks to learn users are likely to migrate to a different product rather than learn the new experience even if it will eventually be a better experience. At Yahoo! I usually shoot for learnability to be under 6 -10 weeks for search. Other products have different thresholds depending on the nature of user interaction.
  • The rich advertisement program increased ad conversions (not clicks!) significantly. Currently this program commands a premium in the Yahoo! ad marketplace.
  • The same heat map will mean different things depending on the user intent. This heatmap on the yahoo shopping experience is bad. As a result the page was redesigned to better match user inent.
  • Overview Of Eye Tracking At Yahoo

    1. 1. Prasad Kantamneni Eye-Tracking At Yahoo! Overview of the Method Bay-CHI: Eye-Tracking Birds of feather meeting. June 22, 2010
    2. 2. About Me <ul><li>Research Scientist at Honeywell Labs </li></ul><ul><li>Research Engineer at WSU </li></ul><ul><li>Principal architect of the Human Perception Center of Excellence at Yahoo!. </li></ul>Yahoo! Presentation
    3. 3. The Big Picture <ul><li>“ When you do something on Yahoo! Search 54M people will see it by the end of the week” </li></ul><ul><li>2.6 Billion visits on the Yahoo! home page alone </li></ul><ul><li># 1 or #2 in in Mail, Sports, Finance, Search… </li></ul><ul><li>More than 2x the users of any of the other mail providers </li></ul><ul><li>~5000 customers come through the user research labs in any given year (US only). </li></ul><ul><li>Use a range of techniques from ethnographic studies to click metrics to understand customer needs, and define UX. </li></ul>Yahoo! Presentation
    4. 4. The Big Picture – Eye Tracking <ul><li>Running ET for about 4 years at Yahoo! </li></ul><ul><li>Deployed on multiple products such as Search, Advertising, Mail, Finance, Sports, Shopping, Intl etc. </li></ul><ul><li>Usually run around ~ 15 - 20 ET studies a year </li></ul><ul><li>With significant increase in UX and Revenue. </li></ul><ul><li>New ET facilities designed for large scale collection of data </li></ul><ul><li>12 eye-trackers, with custom built software. </li></ul>Yahoo! Presentation
    5. 5. Why are we doing what we are doing? <ul><li>In 2005 Traditional measurement methods were falling short of measuring and understanding user behaviors. </li></ul><ul><li>Unable to get to the Why </li></ul>Yahoo! Presentation
    6. 6. History <ul><li>Round 1 of Eye-tracking studies in early 2006 with the standard Tobii software – to understand the value of the method </li></ul><ul><ul><ul><li>Pros: </li></ul></ul></ul><ul><ul><ul><ul><li>Cool factor </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Get engineers to come in and attend the studies </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Easier to build a case </li></ul></ul></ul></ul><ul><ul><ul><li>Cons: </li></ul></ul></ul><ul><ul><ul><ul><li>Too much data to make sense of </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Easy to bias: more sensitive instrument means more susceptible to biases </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Quality of information not significantly different from what you would get from traditional studies </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Starting to understand that certain decisions are made subconsciously – yet unable to quantify or measure them </li></ul></ul></ul></ul>Yahoo! Presentation
    7. 7. History <ul><li>Round 2 of Eye-tracking studies in late 2006 - 2008 </li></ul><ul><ul><ul><li>Changed Software to Eyetools </li></ul></ul></ul><ul><ul><ul><li>Fine tuned protocols to minimize Biases </li></ul></ul></ul><ul><ul><ul><li>Pros: </li></ul></ul></ul><ul><ul><ul><ul><li>Strong Quantitative analysis capabilities </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Reduced biases </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Allowed for solid experimental design </li></ul></ul></ul></ul>Yahoo! Presentation
    8. 8. High resolution image seen by the Fovea Reduced visual acuity experienced by the parafovea Progressively reducing visual acuity from the periphery of the retina
    9. 9. Users use parafoveal preview to identify the parts most likely to have relevant information based on the location of boldfaced terms
    10. 10. Familiar summary patterns draw user attention and clicks Users are relatively blind to unfamiliar summary patterns.
    11. 11. Learned how to model and extrapolate <ul><li>Users use bolding in titles to rapidly scan the SRP. </li></ul><ul><li>Bolding in scan path is critical to making users notice a result. </li></ul><ul><li>If a result is not bolded here, it is not noticed, and hence cannot be judged as relevant . </li></ul>
    12. 12. Learned to test Models with click logs Design A Conversational title style Design B To the point title (query term – property)
    13. 13. Learned when and how to introduce changes into a UI <video> Yahoo! Presentation
    14. 14. Learned when not to introduce change Rapidly Evaluating new ideas, and predicting user behavior  Launched with Keywords instead of image thumbnails -- despite more positive response to thumbnails – because of cognitive overhead.
    15. 15. Learned how to quantifying the effectiveness of a UI and optimize for learning Old Yahoo! Y! with Search Assistance
    16. 16. Learned how to predict learning and better interpret logs <ul><li>Once you optimize for learning, give time for people to learn the new experience. </li></ul>Yahoo! Presentation
    17. 17. Learned how to make money <ul><li>Rich advertisements help advertisers convert better, and command a premium in the marketplace </li></ul>Yahoo! Presentation
    18. 18. History <ul><li>Round 2 of Eye-tracking studies in late 2006 - 2008 </li></ul><ul><ul><li>By the end of the program we had </li></ul></ul><ul><ul><ul><li>Redefined the Search box experience with the Launch of the Search Assistance in 2007 </li></ul></ul></ul><ul><ul><ul><ul><li>Is now the default expectation with search boxes </li></ul></ul></ul></ul><ul><ul><ul><li>Supported significant Market Share and revenue increases </li></ul></ul></ul><ul><ul><ul><li>Enabled product teams to teams launch new features in 11 weeks compared to 1+ years for other teams </li></ul></ul></ul><ul><ul><ul><li>Demand outstripped our ability to deliver </li></ul></ul></ul><ul><ul><li>Cons: </li></ul></ul><ul><ul><ul><li>No cool factor – was too mechanical </li></ul></ul></ul><ul><ul><ul><li>Not scalable </li></ul></ul></ul><ul><ul><ul><ul><li>Time to manage panels </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Time to collect data </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Time to analyze </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Availability of trained researchers </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Time to roll out to other domains. </li></ul></ul></ul></ul>Yahoo! Presentation
    19. 19. History <ul><li>Round 3: 2009 - date </li></ul><ul><ul><li>Bring the cool back </li></ul></ul><ul><ul><li>Wanted to scale up our eye-tracking capacity </li></ul></ul><ul><ul><li>Support innovation </li></ul></ul><ul><ul><li>Reduce skills gap </li></ul></ul><ul><ul><ul><li>Make it easier </li></ul></ul></ul><ul><ul><ul><li>Phased training </li></ul></ul></ul><ul><ul><li>Automated metrics to further speed up analysis </li></ul></ul><ul><ul><li>Expand to other domains </li></ul></ul><ul><ul><li>Cons: </li></ul></ul><ul><ul><ul><li>There is no software to support the needs </li></ul></ul></ul>Yahoo! Presentation
    20. 20. History <ul><li>Round 3 - 2009 - date </li></ul><ul><ul><li>Redesigned Labs to support concurrent data collection </li></ul></ul><ul><ul><ul><li>Can collect data from 130 people in 3 days </li></ul></ul></ul><ul><ul><li>Custom Software built by Eye-Square </li></ul></ul><ul><ul><li>Decomposition of tasks to allow for graceful learning </li></ul></ul><ul><ul><ul><li>Researchers not thrown into the deep end in the first round </li></ul></ul></ul><ul><ul><li>Open access to data </li></ul></ul><ul><ul><ul><li>A large repository of data to mine and build on </li></ul></ul></ul><ul><ul><li>Automated process to correlate ET data with click logs </li></ul></ul>Yahoo! Presentation
    21. 21. The future <ul><li>Eye Tracking is a stepping stone to generating finer behavioral models. </li></ul><ul><li>These models will significantly enrich the way we design experiences </li></ul><ul><ul><ul><li>Reduced guess work </li></ul></ul></ul><ul><ul><ul><li>Faster design cycles – with the team buy-in </li></ul></ul></ul><ul><ul><ul><li>Deeper understanding of user behaviors </li></ul></ul></ul><ul><ul><ul><li>Quantification and Prediction of UI performance </li></ul></ul></ul>Yahoo! Presentation,
    22. 22. Thought Exercise Which is better?
    23. 23. Why? Which is better?
    24. 24. Intent is Key
    25. 25. Prasad <ul><li>Kantamneni </li></ul><ul><li>Principal Architect – Human Perception Center of Excellence </li></ul><ul><li>[email_address] </li></ul>