Video: http://www.heavybit.com/library/blog/streamlining-distributed-stream-processing-with-superchief/ Distributed stream processing is all about gaining actionable insight from high volumes of data in near real-time. Stream processing is complex so many people look to frameworks for help. Last October Librato went in the other direction, moving from Apache Storm to an In-House distributed stream processing system named SuperChief. Ray discusses the experience of moving from Apache Storm to SuperChief. He talks about some of the challenges Librato faced with Storm and the motivation to move away from frameworks. He covers the design and implementation of SuperChief, the benefits of this new approach and the direction SuperChief is going in the future.

1. Streamlining Distributed Stream
Processing
with SuperChief
April '16 SF Metrics Meetup
Ray Jenkins

2. Hi, I’m Ray
@_rayjenkins
github.com/rjenkins

4. Metrics as a Service

6. Metrics and Monitoring
as a Service

7. Thresholds, Windowed
and Absent Detection
Alerting

11. Metrics and Monitoring
as a Service

12. Metrics and Cloud
Monitoring
(end of sales pitch)

13. Distributed Stream
Processing

14. Stream Processing at
Librato
• Librato API ~15TB/day
• 1 - 2Gb/sec Ingress Streaming Tier
• Millions Messages/sec
• 20 Kafka Topics / 15 Kafka nodes
• 9 SuperChief clusters/workloads
• 17 m3.2xls - 24GB Heap / G1
• 12 c3.2xls - 8GB Heap / G1
• Avg CPU Util 15% / Max 30% utilization

15. Philosophy

16. We Favor Composability
and Small Components

17. Well Understood
Technologies

18. End To End Visibility

19. Start Simple

20. API
C*

21. Raw Data 1 Second
Granularity

22. Time Series Aggregation
1minute, 15 minute, 1hr
Min, Max, Sum, Average, Count

23. Do the easiest thing
possible

24. API
C*
Batch

25. This Works Great!

26. Until It Doesn’t
Processing Time > Aggregation
Window

28. API
API
API
C*

29. 3 Storm Topologies
For Time Series Aggregations

30. ƒ(ƒ(x)) ≡ ƒ(x)
Discrete Measurements Per Metric/Source
“Rollup Topologies”

31. 60 Discrete One Second Measurements
15 Discrete One Minute Aggregations
4 Discrete 15 Minute Aggregations

32. r60 r3600r900

33. 3 5 Storm Topologies

34. r60
r_out_60
r900
r_out_900
rwriter60
r3600
C* r60
rwriter
r_out_900
r_out_3600
r_out_3600
C* r900
r_out_1
C* r3600
API
API
API
C* raw

35. New Features

36. Service Side
Aggregation
Non-Discrete Variable Window Aggregations

37. Alerting, Historical
Reporting

38. 5 12 Storm Topologies
I’m not even going to try to draw it

43. Non-Idiomatic Code

44. Inability to jstack or
Heap Dump

45. Unhandled Exception
Kills Entire Worker

47. Inability To Isolate,
Debug and Reason about
Performance Issues

48. Overprovisioning

55. This Works Great!
Until It Doesn’t

56. Disparate Tasks In Executors
Tweak # of tasks to alter hashing, reship, and hope
for a better allocation

57. Rewrote Spout
Allocation Logic

58. 1 Worker Per
Instance
Separate Cluster for each Workload

59. Multiple JVM Hops
Serialization and Netty Calls Add Up

60. Queue Contention
4 Threads From Entry to Exit

61. CPU Util Tuple Acking
Enabled vs. Disabled

62. Engineering Involves
Trade-Offs

63. When You Use A
Framework
You Trade-Off The Ability To Make Trade-Offs

64. YAGNI
Introduce Abstractions When Needed

65. SuperChief
Library for Sane Distributed Stream
Processing

66. SuperChief > Storm
Storm
~45 instances
SuperChief
~20 instances
~3x Reduction in Resource Utilization

67. superchief-core

68. Dropwizard
Configuration, Metrics, Logging and
Managed

71. ZooKeeper Ensemble
SC SCSC

72. Work Partitioning
and Allocation
I Want To Read From Kafka

73. Avoid Coordinating
Wherever Possible
The real trick of horizontal scalability is
independence

74. Leader Partitions
and Writes Work
No Other Coordination

77. ZooKeeper Ensemble
SCSC SCL

78. Semantic Partitioning
Write To Kafka Partitions
By How You Will Consume Them

80. Kafka Partition x
Broker = Work Unit

84. ZooKeeper Ensemble
SCSC SCL
kafka-reader-config
znode
“{ partitions: { sc1:
[ work … n ] sc2:
[ work … n] }}

87. ZooKeeper Ensemble
SCSC SCL
kafka-reader-config
znode
“{ partitions: { sc1:
[ work … n ] sc2:
[ work … n] }}

88. ZooKeeper Ensemble
SC SCSC L

90. KafkaReader
Thread Per Broker/Topic/Partition

91. KafkaCheckpointer
Per Reader, Offsets Pending, Watermark
and Checkpoint to Zookeeper

92. Consumer Threads
Default: availableProcessors()-1

93. Back Pressure

94. More Back Pressure

95. Offset Tracking / Ack

96. Fail Gracefully
Be Partially Available

97. superchief-timeseries

98. C* r60
SCSC
L
r_out_1
sc-r60
SCSC
r_out_60
sc-rwriter60
L

99. New Stuff

102. Thank You &
Questions