Downloaded 11 times
Uploaded byTianjian Chen
Play With Streams
The document discusses stream processing architecture, including the definition of streams and their characteristics, reliability solutions, and handling workload fluctuations. It emphasizes the importance of message relay, uptime, and offers a high-level overview of programming complexities and network management challenges. The conclusion highlights future trends in stream processing, including the trade-off between reliability and performance, with a focus on high-level programming and scale-up strategies.
More Related Content
![Complex Er[jl]ang Processing with StreamBase](https://cdn.slidesharecdn.com/ss_thumbnails/complexerjlangprocessing-pptx-110611095407-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Play With Streams
- 1.
- 2. About Me • PrincipalArchitect @Baidu.com • Contract Programmer – C/C++/Python • Post Engineering Disorder Therapist
- 3.
- 4. Murphy’s Law • Everythingthat Can Go Wrong, Goes Wrong – Unstable Servers – Unstable Network – Unstable Data Source – Unstable Managers – Unstable…
- 6. Agenda • Section I:A Scratch – Brief Intro to Streams • Section II: Build From Ground Up – Modern Stream Processing Architecture • Section III: Could Be Much Sexier – Stream Evolution in Progress
- 7. Section I: AScratch Brief Intro to Streams
- 8. Highlights • Natural Beautyof Streams • Stream Processing Design Language(SDL) • Scratching Stream Applications
- 9.
- 10. Definition of Stream •A Series Of Data Packs • Data is Structured or Semi-Structured • With Internal Topologies – DAG in most cases
- 11.
- 12.
- 13.
- 14. Stream Description Language •Normalize • Accumulate • Join • Partition • Filtering
- 15.
- 16.
- 17. Common Accumulator Types •Time Window • Volume • Condition Trigger
- 18. Join JOIN Indexed Info. Message With akey Joined Message
- 19. Common Join Rules •Internal Mode – Indexing Only Input Messages – Limited Time Window • External Mode – Indexing External Data – Extreme Useful In Integrating Asynchronous Components
- 20.
- 21. Common Partition Rules •Feature Based – Application Related • Random – Balancing Load • Hash – Aggregating Data by User Defined Key • Replication – For Improving Availability
- 22.
- 23. Pit Stop • Normalize •Accumulate • Join • Partition • Filtering
- 24.
- 25. A Simple HelloWorld: Word Count Text Parser ACCU
- 26. An Improved Version TextParser PART ACCU ACCU ACCU NORM
- 27. Sum Up! • Streamsare everywhere • We have a modeling tool to describe stream processing flows(SDL) • Proceed to build real systems
- 28. See U InNext Hour
- 29. Section II: BuildFrom Ground Up Modern Stream Processing Architecture
- 30. Highlights • Flow MeansNon-Stopping • Reliability Matters • Workload Fluctuation Handling
- 31. Message Relay SystemBasics • Message Queuing • Message Passing • Hybrid Queuing
- 32. With Message Queuing Operator1 Queue A Queue B Queue C Operator 2
- 33. Message Passing Operator 1 Operator2 Operator 3
- 34. Hybrid Queuing Operator 1 QueueA Queue B Operator 2 Operator 3
- 35. Pit Stop • Messagerelay is the foundation of stream processing • 3 basic methods of message relay
- 36. Why Reliability Matters? •We got mission critical applications – Real time stock exchange analysis – Ads network click monitoring & billing
- 37. Reliability Solutions • UpstreamBackup & Replay • Source Backup & Replay • Processing Status Backup & Replay
- 38. Upstream Backup &Replay Upstream Downstream Upstream Downstream ACK ACK
- 39. Source Backup &Replay Upstream Downstream Upstream Downstream ACK ACK
- 40. Processing Status Backup& Replay Upstream Downstream Upstream Downstream Stream Operator Downstream Upstream Stream Operator Shadow Status Synchronize Messages Redirect Redirect
- 41. While Reliability HurtsPerformance • All Reliability Solutions are Based on – Indexing – Snapshot – Replay • A Club Of Performance Penalties
- 42. Tuning Strategies • StateOperators VS. Stateless Operators – Independent State Storage – White Board Programming Model – Lazy State Synchronization • Micro Batching Snapshot
- 43. Micro Batching Batch SizeTime Window Throughput Snapshot Cost Restore Cost 1 1ms 1x very high very low 10 10ms 10x high low 100 100ms 100x medium low 1000 100ms 1000x medium low 10000 1s 5000x low high Most Systems Are Here May Constrained By Network Configuration
- 44. Pit Stop • Reliabilityis based on – Message Backup & Replay – Status Snapshot • When tuning, think of – How to handle operator status – Micro Batching
- 45.
- 46. How Does FluctuationHappen? • Data Source Fluctuation • Fault Tolerance Operations
- 47. Fluctuation Handling • TechnologiesTo Obtain – High Performance RPC Framework – Auto Partitioning – Dynamic Resource Allocation – Global Flow Control
- 48. High Performance RPCFramework • Indication of High Performance – Over 20k QPS/sec, with 1byte payload – On commodity server with 2 6-core CPU and Giga Ethernet • See Also – SOFA Framework from Baidu.com – https://github.com/BaiduPS/sofa-pbrpc
- 49. Auto Partitioning • AdjustSub-Stream Num. • Tricks: – Consistent Hashing – Exponential Allocation
- 50. Dynamic Resource Allocation •Tree Model • Evaluation Function • Network Constrains IDC A IDC B PHY 1 PHY 2 PHY 3 PHY 4 VM 1 VM 2 VM 3
- 51. Global Flow Control Op1 Op2 Op3 Op4 FlowControl Service
- 52. Pit Stop • WorkloadFluctuation Handling – Very Fast RPC Framework – Auto Partitioning – Dynamic Resource Allocation – Global Flow Control
- 53. Sum Up! • BasicArchitecture of Stream Processing System • Reliability of Stream Processing • Ways to Handle Workload Fluctuations
- 54. See U inthe Afternoon
- 55. Section III: Could BeMuch Sexier Stream Evolution in Progress
- 56. Highlights • Challenges inReal World Applications • High Level Stream Programming • Optimization Inside Hardware
- 57. Applications • Stream WebCrawler • Co Serving with Hadoop M/R
- 58. Stream Web Crawling OP1 OP2 RedisCluster Web Page Cache OP3 OP4 Log Filter Data Join Feature Extraction Logging API Crawling Job Updater OP5 User-Model Updater OP6HBase Cluster Web DataBase Crawling Job Generator OP7 Crawling Bot OP8 Image Crawling Job Generator OP9 Cache Synchronizer OP 10 Image Crawling Cluster Status DB
- 59. Co-Serving • Distribute RPC •Dynamic Routing OP1 OP2 OP5 OP6 Online Web Services Query Preprocess Query Transform Result Merge Intent Extraction Online Query Log OP3 OP4 User Intent Mining Hadoop Initiating DRPC from Mapper Mapper Receiving DRPC result
- 60. Pit Stop • ProgrammingComplexity • Network Management Complexity • Scale-out Difficulty
- 61. High Level Programming •Stream DataBase & StreamSQL • Stream Computing Description Language • Stream Programming Framework
- 62.
- 63.
- 64.
- 65. Pit Stop Programming InterfaceComplexity Flexibility Stream SQL Low Low Stream DML Medium High Stream Framework High Very High
- 66. Network Management Challenges •Things Are App. Dependent – QoS Request – Relay Priority – Security Strategy – Resource Allocation
- 67. Network Management Solution SDN Integration StreamApp Flow Control Network Configuration
- 68. SDN Integration Stream Flow ControlService Monitors on Streams Open Flow Supported Network Devices
- 69. Pit Stop • NetworkManagement is Crucial – Stream Processing is Bandwidth Consuming – Stream Processing may be Latency Sensitive • Solution is Simple – Software Defined Network Integration (SDN)
- 70. Scale-Out Difficulty • ApplicationConstrains • Algorithm Constrains • Network Hardware Constrains
- 71. So Scale-Up ! •Co-Process Device • Re-Programmable Co-Processors
- 72. FPGA Computing Card •Cheaper than GPGPU • Flexible than GPGPU • Higher POE than GPGPU
- 73.
- 74. See Also • AReconfigurable Fabric for Accelerating Large- Scale Datacenter Services – 20x Performance – ISCA 2014 by Microsoft Research – http://research.microsoft.com/pubs/212001/Cata pult_ISCA_2014.pdf
- 75. Pit Stop • Scale-outis difficult, think of scale-up • Reconfigurable CPU has got significant performance improvements
- 76. Conclusion • Stream ProcessingSystem can be Well Modeled by SDL • Trade Off between Reliability & Performance • High level programming & Scale-Up are Future Trends
- 77. References • Stonebraker, Michael,Uǧur Çetintemel, and Stan Zdonik. "The 8 requirements of real-time stream processing." ACM SIGMOD Record 34, no. 4 (2005): 42-47. • Zaharia, Matei, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. "Discretized streams: Fault-tolerant streaming computation at scale." In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 423-438. ACM, 2013. • Murray, Derek G., Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martin Abadi. "Naiad: a timely dataflow system." In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 439-455. ACM, 2013. • Castro Fernandez, Raul, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. "Integrating scale out and fault tolerance in stream processing using operator state management." In Proceedings of the 2013 international conference on Management of data, pp. 725-736. ACM, 2013.
- 78. Thank U! • G+:chentianjian@gmail.com • Skype: tianjian_chen • Linkedin: http://lnkd.in/bRN6xsh