Download as PDF, PPTX
![hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket](https://image.slidesharecdn.com/netflix-suro-140817103132-phpapp01/85/DevOps-in-the-Amazon-Cloud-Learn-from-the-pioneersNetflix-suro-36-320.jpg)
Uploaded byGaurav "GP" Pal
DevOps in the Amazon Cloud – Learn from the pioneersNetflix suro
The document outlines Netflix's data pipeline, emphasizing the company's focus on data as a critical asset and the complexity of managing vast amounts of event data generated daily. It discusses various tools and techniques used for data processing and storage, including Hadoop, Cassandra, and Druid, and highlights the challenges faced by app owners and data scientists in validating and analyzing this data. Additionally, it covers the need for efficient querying, real-time data handling, and fault tolerance in their systems.
More Related Content
Similar to DevOps in the Amazon Cloud – Learn from the pioneersNetflix suro
DevOps in the Amazon Cloud – Learn from the pioneersNetflix suro
- 1. Netflix Data Pipeline SudhirTonse (@stonse) Danny Yuan (@g9yuayon)
- 2. photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/! Netflixis a log generating company that also happens to stream movies - Adrian Cockroft
- 3. Data Is themost important asset at Netflix
- 4. If all thedata is easily available to all teams, it can be leveraged in new and exciting ways
- 5.
- 6.
- 7. ~1000 Device Types ~500Apps/Web Services Dashboard
- 8. ~1000 Device Types ~500Apps/Web Services ~100 Billion Events/Day ! 3.2M messages per second at peak time ! 3GB per second at peak time Dashboard
- 9. Type of Events •User Interface Events • Search Event (‘Matrix’ using PS3 …) • Star Ra>ng Event (HoC : 5 stars, Xbox, US, …) ! • Infrastructural Events • RPC Call (API -‐> Billing Service, ‘/bill/..’, 200, …) • Log Errors (NPE, “Movie is null”, …, …) ! • Other Events … ! !
- 10. Making Sense ofBillions of Events
- 11.
- 13.
- 17.
- 20.
- 21. We Want toProcess App Data in Hadoop
- 27.
- 28. @NetflixOSS Big DataTools
- 29. Hadoop as aService
- 30. Pig Scripting onSteroids
- 31. Pig Married toClojure
- 32. S3MPER S3mper is alibrary that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index. S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.
- 33. Efficient ETL withCassandra Cassandra
- 34.
- 35.
- 36. hgrep -C 10-k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
- 42. We Want toAggregate, Index, and Query Data in Real Time
- 43.
- 44. Let’s walk throughsome use cases
- 45. client activity event * /name= “movieStarts”
- 46.
- 47. Pipeline Challenges • Appowners: send and forget
- 48. Pipeline Challenges • Appowners: send and forget • Data scientists: validation, ETL, batch processing
- 49. Pipeline Challenges • Appowners: send and forget • Data scientists: validation, ETL, batch processing • DevOps: stream processing, targeted search
- 50.
- 53. We Want toConsume Data Selectively in Different Ways
- 55. • Message broker! •High-throughput! • Persistent and replicated
- 56.
- 57.
- 58.
- 59. Guided Debugging inthe Right Context
- 60. Guided Debugging inthe Right Context
- 61. Guided Debugging inthe Right Context
- 62. Guided Debugging inthe Right Context
- 63. Guided Debugging inthe Right Context
- 64. Guided Debugging inthe Right Context
- 65. Guided Debugging inthe Right Context
- 66. Guided Debugging inthe Right Context
- 67.
- 68. •Ad-hoc query withdifferent dimensions What We Need
- 69. •Ad-hoc query withdifferent dimensions •Quick aggregations and Top-N queries What We Need
- 70. •Ad-hoc query withdifferent dimensions •Quick aggregations and Top-N queries •Time series with flexible filters What We Need
- 71. •Ad-hoc query withdifferent dimensions •Quick aggregations and Top-N queries •Time series with flexible filters •Quick access to raw data using boolean queries What We Need
- 72. Druid • Rapid explorationof high dimensional data! • Fast ingestion and querying! • Time series
- 73. • Real-time indexingof event streams! • Killer feature: boolean search! • Great UI: Kibana
- 74.
- 75.
- 77.
- 78. It’s Not AllAbout Counters and Time Series
- 80. RequestId Parent IdNode Id Service Name Status 4965-4a74 0 123 Edge Service 200 4965-4a74 123 456 Gateway 200 4965-4a74 456 789 Service A 200 4965-4a74e 456 abc Service B 200 Status:200
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87. A System thatSupports All These
- 88. A Data PipelineTo Glue Them All
- 89.
- 90.
- 91. Message Producing • Simpleand Uniform API • messageBus.publish(event)
- 92. Consumption Is SimpleToo consumer.observe().subscribe(new Subscriber<>() { @Override public void onNext(Ackable<IncomingMessage> ackable) { process(ackable.getEntity(MyEventType.class)); ackable.ack(); } }); ! consumer.pause(); consumer.resume()
- 93. RxJava • Functional reactiveprogramming model! • Powerful streaming API! • Separation of logic and threading model
- 94.
- 95. Design Decisions • TopPriority: app stability and throughput
- 96. Design Decisions • TopPriority: app stability and throughput • Asynchronous operations
- 97. Design Decisions • TopPriority: app stability and throughput • Asynchronous operations • Aggressive buffering
- 98. Design Decisions • TopPriority: app stability and throughput • Asynchronous operations • Aggressive buffering • Drops messages if necessary
- 99.
- 100.
- 101.
- 102. Fault Tolerance Features •Write and forward with auto-reattached EBS (Amazon’s Elastic Block Storage)
- 103. Fault Tolerance Features •Write and forward with auto-reattached EBS (Amazon’s Elastic Block Storage) • disk-backed queue: big-queue
- 104. Fault Tolerance Features •Write and forward with auto-reattached EBS (Amazon’s Elastic Block Storage) • disk-backed queue: big-queue • Customized scaling down
- 106. There’s More toDo • Contribute to @NetflixOSS ! • Join us :-)
- 107.
- 108. You can buildyour own web-scale data pipeline using open source components
- 109. Thank You! Sudhir Tonse http://www.linkedin.com/in/sudhirtonse Twitter:@stonse Danny Yuan http://www.linkedin.com/pub/danny- yuan/4/374/862 Twitter: @g9yuayon