DevOps helps accelerate the delivery of software applications through automation and by removing Development & Operations silos. The Netflix Platform Engineering team has developed a robust data pipeline solution called SURO that has been open sourced. Come learn from the experiences of pioneers like Netflix how they are leveraging the data pipeline for new and innovative use cases. This is the presentation by Danny Yuan, Netflix Platform Engineering Team on operational and monitoring aspects of applications on cloud platforms.

1. Netflix Data Pipeline
Sudhir Tonse (@stonse)
Danny Yuan (@g9yuayon)

2. photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/!
Netflix is a log generating company
that also happens to stream movies
- Adrian Cockroft

3. Data Is the most important asset
at Netflix

4. If all the data is easily available to all
teams, it can be leveraged in new and
exciting ways

5. Dashboard

6. ~1000 Device Types
Dashboard

7. ~1000 Device Types
~500 Apps/Web Services
Dashboard

8. ~1000 Device Types
~500 Apps/Web Services
~100 Billion Events/Day
!
3.2M messages per
second at peak time
!
3GB per second at peak
time
Dashboard

9. Type of Events
• User
Interface
Events
• Search
Event
(‘Matrix’
using
PS3
…)
• Star
Ra>ng
Event
(HoC
:
5
stars,
Xbox,
US,
…)
!
• Infrastructural
Events
• RPC
Call
(API
-­‐>
Billing
Service,
‘/bill/..’,
200,
…)
• Log
Errors
(NPE,
“Movie
is
null”,
…,
…)
!
• Other
Events
…
!
!

10. Making Sense of Billions of Events

11. http://netflix.github.io
+
ElasticSearch
Druid

13. A Humble Beginning

17. Evolution …Scale!

20. Application
Application
Application Application
Application
Application
Application
Application
ApplicationApplication

21. We Want to Process
App Data in Hadoop

27. Our Hadoop Ecosystem

28. @NetflixOSS Big Data Tools

29. Hadoop as a Service

30. Pig Scripting on Steroids

31. Pig Married to Clojure

32. S3MPER
S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.
S3mper is a library that provides an
additional layer of consistency
checking on top of Amazon's S3 index through
use of a consistent, secondary index.

33. Efficient ETL with Cassandra
Cassandra

34. Offline Analysis

35. Evolution … Speed!

36. hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

42. We Want to Aggregate, Index, and
Query Data in Real Time

43. Interactive Exploration

44. Let’s walk through some use cases

45. client activity event
*
/name = “movieStarts”

46. Pipeline Challenges

47. Pipeline Challenges
• App owners: send and forget

48. Pipeline Challenges
• App owners: send and forget
• Data scientists: validation, ETL, batch
processing

49. Pipeline Challenges
• App owners: send and forget
• Data scientists: validation, ETL, batch
processing
• DevOps: stream processing, targeted search

50. Message Routing

53. We Want to Consume Data
Selectively in Different Ways

55. • Message broker!
• High-throughput!
• Persistent and replicated

56. There Is More

57. Intelligent Alerts

58. Intelligent Alerts

59. Guided Debugging in the Right Context

60. Guided Debugging in the Right Context

61. Guided Debugging in the Right Context

62. Guided Debugging in the Right Context

63. Guided Debugging in the Right Context

64. Guided Debugging in the Right Context

65. Guided Debugging in the Right Context

66. Guided Debugging in the Right Context

67. What We Need

68. •Ad-hoc query with different dimensions
What We Need

69. •Ad-hoc query with different dimensions
•Quick aggregations and Top-N queries
What We Need

70. •Ad-hoc query with different dimensions
•Quick aggregations and Top-N queries
•Time series with flexible filters
What We Need

71. •Ad-hoc query with different dimensions
•Quick aggregations and Top-N queries
•Time series with flexible filters
•Quick access to raw data using boolean
queries
What We Need

72. Druid
• Rapid exploration of high dimensional data!
• Fast ingestion and querying!
• Time series

73. • Real-time indexing of event streams!
• Killer feature: boolean search!
• Great UI: Kibana

74. The Old Pipeline

75. The New Pipeline

77. There Is More

78. It’s Not All About Counters and Time Series

80. RequestId Parent Id Node Id Service Name Status
4965-4a74 0 123 Edge Service 200
4965-4a74 123 456 Gateway 200
4965-4a74 456 789 Service A 200
4965-4a74e 456 abc Service B 200
Status:200

81. Distributed Tracing

82. Distributed Tracing

83. Distributed Tracing

84. Distributed Tracing

85. Distributed Tracing

86. Distributed Tracing

87. A System that Supports All These

88. A Data Pipeline To Glue
Them All

89. Make It Simple

90. Message Producing

91. Message Producing
• Simple and Uniform API
• messageBus.publish(event)

92. Consumption Is Simple Too
consumer.observe().subscribe(new
Subscriber<>()
{
@Override
public
void
onNext(Ackable<IncomingMessage>
ackable)
{
process(ackable.getEntity(MyEventType.class));
ackable.ack();
}
});
!
consumer.pause();
consumer.resume()

93. RxJava
• Functional reactive programming model!
• Powerful streaming API!
• Separation of logic and threading model

94. Design Decisions

95. Design Decisions
• Top Priority: app stability and throughput

96. Design Decisions
• Top Priority: app stability and throughput
• Asynchronous operations

97. Design Decisions
• Top Priority: app stability and throughput
• Asynchronous operations
• Aggressive buffering

98. Design Decisions
• Top Priority: app stability and throughput
• Asynchronous operations
• Aggressive buffering
• Drops messages if necessary

99. Anything Can Fail

100. Cloud Resiliency

101. Fault Tolerance Features

102. Fault Tolerance Features
• Write and forward with auto-reattached EBS
(Amazon’s Elastic Block Storage)

103. Fault Tolerance Features
• Write and forward with auto-reattached EBS
(Amazon’s Elastic Block Storage)
• disk-backed queue: big-queue

104. Fault Tolerance Features
• Write and forward with auto-reattached EBS
(Amazon’s Elastic Block Storage)
• disk-backed queue: big-queue
• Customized scaling down

106. There’s More to Do
• Contribute to @NetflixOSS !
• Join us :-)

107. Summary
http://netflix.github.io
+
ElasticSearch
Druid

108. You can build your own web-scale data
pipeline using open source components

109. Thank You!
Sudhir Tonse
http://www.linkedin.com/in/sudhirtonse
Twitter: @stonse
Danny Yuan
http://www.linkedin.com/pub/danny-
yuan/4/374/862
Twitter: @g9yuayon