Successfully reported this slideshow.
@mrogati<br />
Scale changes what’s possible.<br />Scale changes what’s possible.<br />Scal<br />
2004-2006<br />
Possible :<br />High risk, <br />	rapid innovation<br />Chasing the long tail <br />										  (by hand)<br />Not possibl...
The<br />Data<br />Scientist<br />is born<br />-- LinkedIn job ad, April 2008<br />
Data products    <br />		infrastructure innovation <br />                         and adoption<br />
Data products    <br />		infrastructure innovation <br />                         and adoption<br />Voldemort<br />Azkaban...
Data products    <br />		infrastructure innovation <br />                         and adoption<br />
1999<br />software engineer <br />web developer <br />2001<br />research assistant<br />PhD student<br />2008<br />game ar...
Scale changes what’s possible.<br />Scale changes what’s possible.<br />Sca<br />
Possible :<br />Insights into the <br />	world at large<br />Network effects<br />	Infrastructure innovation <br />Not pos...
Data infrastructure team!<br />~1900 <br />machines<br />Kafka <br />real time data streams<br />Reporting <br />there’s a...
Insights: The world – sliced & diced<br />
Insights: The world – sliced & diced<br />
Insights: <br />	The world <br />	– sliced & diced<br />
Data Products: Chasing the long tail<br />
Data Products: Personalized insights<br />
Data Products: <br />Crowdsourced Insights<br />
Scale changes what’s possible.<br />Scale changes what’s possible.<br />Scal<br />
Possible :<br />Sliced-and-diced <br />	insights and <br />	products<br />		Network effects<br />	Economies of scale<br />...
The<br />Data<br />Scientist<br />is born<br />-- LinkedIn job ad, April 2008<br />
The<br />Data<br />Scientist<br />& the teenage years<br />+<br />-- LinkedIn job ad,<br /> September 2011<br />
Scale changes what’s possible.<br />Scale changes what’s possible.<br />Scal<br />@mrogati<br />
Upcoming SlideShare
Loading in …5
×

1M.10M.100M. Data! - @mrogati's talk at Strata 2011

4,292 views

Published on

Follow along w/ the video: http://www.youtube.com/watch?v=2SQ0O_oPpe4

How do data infrastructure, insights and products change when your user base grows by orders of magnitude? When should you move your user-facing data product off your laptop? (hint: now!) Does your data offer insights about the world at large, or is it just mirroring your early adopters? In this talk, I will share some of the data scaling lessons we've learned at LinkedIn, recount war stories (and close calls!) and document the evolution of the data scientist.

Published in: Technology

1M.10M.100M. Data! - @mrogati's talk at Strata 2011

  1. 1. @mrogati<br />
  2. 2. Scale changes what’s possible.<br />Scale changes what’s possible.<br />Scal<br />
  3. 3.
  4. 4. 2004-2006<br />
  5. 5. Possible :<br />High risk, <br /> rapid innovation<br />Chasing the long tail <br /> (by hand)<br />Not possible:<br /> Long tail recommendations<br />Network effects <br /> Insights into the world at large<br />
  6. 6.
  7. 7. The<br />Data<br />Scientist<br />is born<br />-- LinkedIn job ad, April 2008<br />
  8. 8. Data products <br /> infrastructure innovation <br /> and adoption<br />
  9. 9.
  10. 10. Data products <br /> infrastructure innovation <br /> and adoption<br />Voldemort<br />Azkaban<br />
  11. 11. Data products <br /> infrastructure innovation <br /> and adoption<br />
  12. 12. 1999<br />software engineer <br />web developer <br />2001<br />research assistant<br />PhD student<br />2008<br />game artists<br />Insights: beyond the early adopters <br />
  13. 13. Scale changes what’s possible.<br />Scale changes what’s possible.<br />Sca<br />
  14. 14. Possible :<br />Insights into the <br /> world at large<br />Network effects<br /> Infrastructure innovation <br />Not possible:<br /> Long tail recommendations<br /> Segmented insights and products<br />
  15. 15.
  16. 16.
  17. 17. Data infrastructure team!<br />~1900 <br />machines<br />Kafka <br />real time data streams<br />Reporting <br />there’s a (mobile) app for that!<br /> … and servers, and dedicated teams<br />Infrastructure – evolved.<br />
  18. 18. Insights: The world – sliced & diced<br />
  19. 19. Insights: The world – sliced & diced<br />
  20. 20. Insights: <br /> The world <br /> – sliced & diced<br />
  21. 21. Data Products: Chasing the long tail<br />
  22. 22. Data Products: Personalized insights<br />
  23. 23. Data Products: <br />Crowdsourced Insights<br />
  24. 24. Scale changes what’s possible.<br />Scale changes what’s possible.<br />Scal<br />
  25. 25. Possible :<br />Sliced-and-diced <br /> insights and <br /> products<br /> Network effects<br /> Economies of scale<br /> Fast A/B tests<br />Not possible:<br /> Casual, hour-long outages<br /> Testing in production on 100% of users<br />
  26. 26.
  27. 27. The<br />Data<br />Scientist<br />is born<br />-- LinkedIn job ad, April 2008<br />
  28. 28. The<br />Data<br />Scientist<br />& the teenage years<br />+<br />-- LinkedIn job ad,<br /> September 2011<br />
  29. 29. Scale changes what’s possible.<br />Scale changes what’s possible.<br />Scal<br />@mrogati<br />

×