0
Data Insights in Netflix                      Danny Yuan (@g9yuayon)                      Jae BaeFriday, March 1, 13       ...
Who Am I?Friday, March 1, 13               2
Who Am I?    Member of Netflix’s Platform    Engineering team, working on    very large scale data    infrastructure (@g9yu...
Who Am I?    Member of Netflix’s Platform    Engineering team, working on    very large scale data    infrastructure (@g9yu...
Who Am I?    Member of Netflix’s Platform    Engineering team, working on    very large scale data    infrastructure (@g9yu...
Friday, March 1, 13                                                                  3Developers usually think about monit...
No Monitoring Metrics TodayFriday, March 1, 13                                                                  3Developer...
photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/Friday, March 1, 13                ...
1,500,000Friday, March 1, 13                                                                    5During peak hours, our da...
70,000,000,000Friday, March 1, 13                6Or 70 billions a day on average.
Server Farm                                                    Log Filter          Sink Plugin          Hadoop      Server...
Highly Reliable Data Pipeline      Server Farm                                                    Log Filter          Sink...
A Humble BeginningFriday, March 1, 13                                                                            8We didn’...
A Humble BeginningFriday, March 1, 13                                                                            8We didn’...
A Humble BeginningFriday, March 1, 13                                                                            8We didn’...
A Humble BeginningFriday, March 1, 13                                                                            8We didn’...
Friday, March 1, 13                                                                          9Something happened. Our traf...
Friday, March 1, 13                                                                          9Something happened. Our traf...
Application                                                                 Application                                App...
So We EvolvedFriday, March 1, 13                                                                      10So we evolved. One...
So We EvolvedFriday, March 1, 13                                                                      10So we evolved. One...
So We Evolvedhgrep -C 10 -k 5,2,3 users.*[1-9]{3} *catalina.out s3//bucketFriday, March 1, 13                             ...
So We Evolvedhgrep -C 10 -k 5,2,3 users.*[1-9]{3} *catalina.out s3//bucketFriday, March 1, 13                             ...
Friday, March 1, 13                                                   11A search tool that searches live instances’ logs i...
Friday, March 1, 13                                                   11A search tool that searches live instances’ logs i...
Friday, March 1, 13                                                   11A search tool that searches live instances’ logs i...
Friday, March 1, 13                                                   11A search tool that searches live instances’ logs i...
Friday, March 1, 13                                                   11A search tool that searches live instances’ logs i...
Friday, March 1, 13                                                   11A search tool that searches live instances’ logs i...
Field Name      Field Value                      Client     “API”                      Server   “Cryptex”               St...
Friday, March 1, 13     13DSE Sting is a bless.
Friday, March 1, 13     13DSE Sting is a bless.
Friday, March 1, 13     13DSE Sting is a bless.
Friday, March 1, 13                                                  14So we built yet another tool to scratch it with the...
Still, We Have a Real-Time ItchFriday, March 1, 13                                                  14So we built yet anot...
Friday, March 1, 13                                                                     15Error summary in the past 10 sec...
Friday, March 1, 13                                                                     15Error summary in the past 10 sec...
Friday, March 1, 13                                                                     15Error summary in the past 10 sec...
Friday, March 1, 13                                                                          16A query of all the users wh...
Friday, March 1, 13                                                                          16A query of all the users wh...
Friday, March 1, 13                                                                          16A query of all the users wh...
Interested?Friday, March 1, 13                 17
See You                      TomorrowFriday, March 1, 13                                                                  ...
Upcoming SlideShare
Loading in...5
×

Strata lightening-talk

572

Published on

This is a lightening talk for Metamarkets' real-time rave party. It's a teaser talk to start the conversation

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
572
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
24
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Strata lightening-talk"

  1. 1. Data Insights in Netflix Danny Yuan (@g9yuayon) Jae BaeFriday, March 1, 13 1
  2. 2. Who Am I?Friday, March 1, 13 2
  3. 3. Who Am I? Member of Netflix’s Platform Engineering team, working on very large scale data infrastructure (@g9yuayon)Friday, March 1, 13 2
  4. 4. Who Am I? Member of Netflix’s Platform Engineering team, working on very large scale data infrastructure (@g9yuayon) Built and operated Netflix’s cloud crypto serviceFriday, March 1, 13 2
  5. 5. Who Am I? Member of Netflix’s Platform Engineering team, working on very large scale data infrastructure (@g9yuayon) Built and operated Netflix’s cloud crypto service Worked with Jae Bae on querying multi-dimensional data in real timeFriday, March 1, 13 2
  6. 6. Friday, March 1, 13 3Developers usually think about monitoring metrics when “real-time” data ismentioned. We have powerful monitoring systems that track millions of metricsper second. But I’m not going to talk about it today. Monitoring metric is crucialdata. That itself would warrant another multi-hour talk by our monitoringteam. :-)
  7. 7. No Monitoring Metrics TodayFriday, March 1, 13 3Developers usually think about monitoring metrics when “real-time” data ismentioned. We have powerful monitoring systems that track millions of metricsper second. But I’m not going to talk about it today. Monitoring metric is crucialdata. That itself would warrant another multi-hour talk by our monitoringteam. :-)
  8. 8. photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/Friday, March 1, 13 4Instead, I’m going to talk about logs. Why is it interesting at all?
  9. 9. 1,500,000Friday, March 1, 13 5During peak hours, our data pipeline collects over 1.5 million log events per second
  10. 10. 70,000,000,000Friday, March 1, 13 6Or 70 billions a day on average.
  11. 11. Server Farm Log Filter Sink Plugin Hadoop Server Farm Kafka Log Filter Sink Plugin Druid Log Collectors Server Farm Log Filter Sink Plugin ElasticSearchphoto credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/Friday, March 1, 13 7We have this tens of thousands of machines, all of which send log data over a robust datapipeline to highly reliable data collectors. The collectors then filter the data, transform thedata, and dispatch the data to to different destinations for further processing.Photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/
  12. 12. Highly Reliable Data Pipeline Server Farm Log Filter Sink Plugin Hadoop Server Farm Kafka Log Filter Sink Plugin Druid Log Collectors Server Farm Log Filter Sink Plugin ElasticSearchphoto credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/Friday, March 1, 13 7We have this tens of thousands of machines, all of which send log data over a robust datapipeline to highly reliable data collectors. The collectors then filter the data, transform thedata, and dispatch the data to to different destinations for further processing.Photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/
  13. 13. A Humble BeginningFriday, March 1, 13 8We didn’t build everything in one night. Actually, we had a humble start. I did a lot of logscraping like these. I also used R to analyze logs. But these are specific tasks, and at somepoint
  14. 14. A Humble BeginningFriday, March 1, 13 8We didn’t build everything in one night. Actually, we had a humble start. I did a lot of logscraping like these. I also used R to analyze logs. But these are specific tasks, and at somepoint
  15. 15. A Humble BeginningFriday, March 1, 13 8We didn’t build everything in one night. Actually, we had a humble start. I did a lot of logscraping like these. I also used R to analyze logs. But these are specific tasks, and at somepoint
  16. 16. A Humble BeginningFriday, March 1, 13 8We didn’t build everything in one night. Actually, we had a humble start. I did a lot of logscraping like these. I also used R to analyze logs. But these are specific tasks, and at somepoint
  17. 17. Friday, March 1, 13 9Something happened. Our traffic turned into a hockey stick, and the number of applicationsexploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.
  18. 18. Friday, March 1, 13 9Something happened. Our traffic turned into a hockey stick, and the number of applicationsexploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.
  19. 19. Application Application Application Application Application Application Application Application Application ApplicationFriday, March 1, 13 9Something happened. Our traffic turned into a hockey stick, and the number of applicationsexploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.
  20. 20. So We EvolvedFriday, March 1, 13 10So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It ismuch more useful that the one provided by Apache Hadoop Distribution, because it supportsmany more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.
  21. 21. So We EvolvedFriday, March 1, 13 10So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It ismuch more useful that the one provided by Apache Hadoop Distribution, because it supportsmany more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.
  22. 22. So We Evolvedhgrep -C 10 -k 5,2,3 users.*[1-9]{3} *catalina.out s3//bucketFriday, March 1, 13 10So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It ismuch more useful that the one provided by Apache Hadoop Distribution, because it supportsmany more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.
  23. 23. So We Evolvedhgrep -C 10 -k 5,2,3 users.*[1-9]{3} *catalina.out s3//bucketFriday, March 1, 13 10So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It ismuch more useful that the one provided by Apache Hadoop Distribution, because it supportsmany more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.
  24. 24. Friday, March 1, 13 11A search tool that searches live instances’ logs is also developed.
  25. 25. Friday, March 1, 13 11A search tool that searches live instances’ logs is also developed.
  26. 26. Friday, March 1, 13 11A search tool that searches live instances’ logs is also developed.
  27. 27. Friday, March 1, 13 11A search tool that searches live instances’ logs is also developed.
  28. 28. Friday, March 1, 13 11A search tool that searches live instances’ logs is also developed.
  29. 29. Friday, March 1, 13 11A search tool that searches live instances’ logs is also developed.
  30. 30. Field Name Field Value Client “API” Server “Cryptex” StatusCode 200 ResponseTime 73Friday, March 1, 13 12Hive becomes indispensable.
  31. 31. Friday, March 1, 13 13DSE Sting is a bless.
  32. 32. Friday, March 1, 13 13DSE Sting is a bless.
  33. 33. Friday, March 1, 13 13DSE Sting is a bless.
  34. 34. Friday, March 1, 13 14So we built yet another tool to scratch it with the help of Druid.
  35. 35. Still, We Have a Real-Time ItchFriday, March 1, 13 14So we built yet another tool to scratch it with the help of Druid.
  36. 36. Friday, March 1, 13 15Error summary in the past 10 seconds. You get to slice and dice through arbitrarycombination of different dimension across multiple time series.Trends over search query of “90210” by CanadiansHow many people started streaming any episode of House of Cards in the past hour, grouped
  37. 37. Friday, March 1, 13 15Error summary in the past 10 seconds. You get to slice and dice through arbitrarycombination of different dimension across multiple time series.Trends over search query of “90210” by CanadiansHow many people started streaming any episode of House of Cards in the past hour, grouped
  38. 38. Friday, March 1, 13 15Error summary in the past 10 seconds. You get to slice and dice through arbitrarycombination of different dimension across multiple time series.Trends over search query of “90210” by CanadiansHow many people started streaming any episode of House of Cards in the past hour, grouped
  39. 39. Friday, March 1, 13 16A query of all the users who started streaming House of Cards in the past three hours, andresults came back in seconds.
  40. 40. Friday, March 1, 13 16A query of all the users who started streaming House of Cards in the past three hours, andresults came back in seconds.
  41. 41. Friday, March 1, 13 16A query of all the users who started streaming House of Cards in the past three hours, andresults came back in seconds.
  42. 42. Interested?Friday, March 1, 13 17
  43. 43. See You TomorrowFriday, March 1, 13 18If you’re interested in how we did the real-time interactive queries with the help of Druid, docome to our talk. See you tomorrow
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×