Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Paper6745 presentation tianjian

441 views

Published on

Paper presentation for IEEE BigData 2014 Paper 6745

Building A Massive Stream Computing Platform For Flexible Applications

Published in: Technology

Paper6745 presentation tianjian

  1. 1. Building A Massive Stream Computing Platform For Flexible Applications Tianjian Chen Zhengrui Man Hao Li Xin Sun Raymond K. Wong Zhiwei Yu Jun, 2014 IEEE BigData Congress
  2. 2. Unveil Something behind the Paper
  3. 3. Highlights • Applications Design the System Themselves • Complete Modularization Strategy • Extreme Simple Stream Model
  4. 4. A More Accurate Version of Development History
  5. 5. “Hey guys, there is a new stream computing technology coming!”
  6. 6. Great! Can it make every process go faster?
  7. 7. “OK , that may depend on …”
  8. 8. OP1 OP2 OP3 OP4 Mobile Devices User Filter Ads Recalling Ads Ranking Push Controller User Preference Database Location Logging API Ads POI Database LBS Ads Service DRPC DRPC LBS Ads Query Can I do this? • Location Based Ads Push System • Co-Serving With Online Services
  9. 9. OP1 OP2 Redis Cluster Web Page Cache OP3 OP4 OP5 User Model Log Filter Data Join Feature Extraction Model Update Web Crawling Logging APICan I do this? • User Preference Tracking System
  10. 10. Stream Application Vortex Platform Hadoop Platform Map-Reduce Application OnlineWebServices Can I co-operate it with m/r?
  11. 11. Private Cluster Leased from Public Clusters Can I scale-out it on leased containers?
  12. 12. Handle Fluctuating Workloads?
  13. 13. Can it be 100% reliable?
  14. 14. Complete Modularization
  15. 15. Nukua Automation System Universal Resource Manger Spinal DMQ Stream Computing Core DRPC Service Interface Layer 1 : Computing Resources Layer 2 : Deployment Automation Layer 3 : Data Transmission Layer 4 : Topology Representation Layer 5 : Stream Application 5 Independent Sub-Systems
  16. 16. Everything is Configurable Even for The Message Relay Model
  17. 17. Message Queuing Model Operator 1 Queue A Queue B Queue C Operator 2
  18. 18. Message Passing Model Operator 1 Operator 2 Operator 3
  19. 19. uplink uplink uplink downlink downlink downlink Spinal DMQ Cluster SCC Operator Cluster SCC Master
  20. 20. downlink downlink downlink uplink uplink OP1 Sub-Links of OP1 Downstream Sub-Links of OP2 Upstream OP2 Operator Cluster Operator Cluster Spinal DMQ Cluster Message Queuing Configuration
  21. 21. downlink downlink downlink uplink uplink OP1 Sub-Links of OP1 Downstream Sub-Links of OP2 Upstream OP2 Message Passing Configuration
  22. 22. A Simple Stream Model
  23. 23. Traditional Stream Model 0 1 2 3 5 6 7 8 n • Independent Consumer Status • High Index Overhead • High Snapshot Overhead
  24. 24. Vortex Stream Model 0 n Head Tail • Unified Status • Minimal Index Overhead • Minimal Snapshot Overhead
  25. 25. Lessons Learned • Highly Configurable System For Flexible Applications • Big Data Requires Everything Simple & Reliable
  26. 26. Thank U! • G+: chentianjian@gmail.com • Skype: tianjian_chen • Linkedin: http://lnkd.in/bRN6xsh

×