Your SlideShare is downloading. ×

1M.10M.100M. Data! - @mrogati's talk at Strata 2011

3,436

Published on

Follow along w/ the video: http://www.youtube.com/watch?v=2SQ0O_oPpe4 …

Follow along w/ the video: http://www.youtube.com/watch?v=2SQ0O_oPpe4

How do data infrastructure, insights and products change when your user base grows by orders of magnitude? When should you move your user-facing data product off your laptop? (hint: now!) Does your data offer insights about the world at large, or is it just mirroring your early adopters? In this talk, I will share some of the data scaling lessons we've learned at LinkedIn, recount war stories (and close calls!) and document the evolution of the data scientist.

Published in: Technology
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,436
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
1
Comments
1
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • 1M.10M.100M. Data! : How do data infrastructure, insights and products change when your user base grows by orders of magnitude.
  • Scale changes your problems, your solutions and your mindset.
  • Scale changes your problems, your solutions and your mindset.
  • Hardware, software/technology, people and everywhere
  • You can truly chase the long tail.
  • Transcript

    • 1. @mrogati
    • 2. Scale changes what’s possible.
      Scale changes what’s possible.
      Scal
    • 3.
    • 4. 2004-2006
    • 5. Possible :
      High risk,
      rapid innovation
      Chasing the long tail
      (by hand)
      Not possible:
      Long tail recommendations
      Network effects
      Insights into the world at large
    • 6.
    • 7. The
      Data
      Scientist
      is born
      -- LinkedIn job ad, April 2008
    • 8. Data products
      infrastructure innovation
      and adoption
    • 9.
    • 10. Data products
      infrastructure innovation
      and adoption
      Voldemort
      Azkaban
    • 11. Data products
      infrastructure innovation
      and adoption
    • 12. 1999
      software engineer
      web developer
      2001
      research assistant
      PhD student
      2008
      game artists
      Insights: beyond the early adopters
    • 13. Scale changes what’s possible.
      Scale changes what’s possible.
      Sca
    • 14. Possible :
      Insights into the
      world at large
      Network effects
      Infrastructure innovation
      Not possible:
      Long tail recommendations
      Segmented insights and products
    • 15.
    • 16.
    • 17. Data infrastructure team!
      ~1900
      machines
      Kafka
      real time data streams
      Reporting
      there’s a (mobile) app for that!
      … and servers, and dedicated teams
      Infrastructure – evolved.
    • 18. Insights: The world – sliced & diced
    • 19. Insights: The world – sliced & diced
    • 20. Insights:
      The world
      – sliced & diced
    • 21. Data Products: Chasing the long tail
    • 22. Data Products: Personalized insights
    • 23. Data Products:
      Crowdsourced Insights
    • 24. Scale changes what’s possible.
      Scale changes what’s possible.
      Scal
    • 25. Possible :
      Sliced-and-diced
      insights and
      products
      Network effects
      Economies of scale
      Fast A/B tests
      Not possible:
      Casual, hour-long outages
      Testing in production on 100% of users
    • 26.
    • 27. The
      Data
      Scientist
      is born
      -- LinkedIn job ad, April 2008
    • 28. The
      Data
      Scientist
      & the teenage years
      +
      -- LinkedIn job ad,
      September 2011
    • 29. Scale changes what’s possible.
      Scale changes what’s possible.
      Scal
      @mrogati

    ×