Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Lessons learned fromTwitter Storm03/30/2013Robbie Cheng
Agenda•   Why Twitter Storm•   Real World Examples•   Performance Gain•   Lessons Learned•   Future Works                 ...
Why Twitter Storm                  Queue      Storm     Hadoopmore concurrentjobs            X            V          V    ...
Real World Examples• simple query through social network API• detailed query through social network API• csv import/export...
Performance Gain• simple query through social network API  •   daily throughput : 20MM  •   AWS instances : m1.large * 6• ...
Performance Gain• Csv Export (60k)  •   queue : 4 min 33 sec  •   storm : 1 min 44 sec• Csv Import (60k)  •   TBD         ...
Lessons Learned• simple task is better  •   http://en.wikipedia.org/wiki/File:AmdahlsLaw.svg• Batch processing is required...
Future Works• How do we scale out by adding more  instances?                                       8
ConclusionStorm is not the silver bullet, itdepends on your requirements                                     9
Q&AWe’re hiring!jobs@fliptop.com                   10
Upcoming SlideShare
Loading in …5
×

Lesson learned of twitter storm

932 views

Published on

  • Be the first to comment

Lesson learned of twitter storm

  1. 1. Lessons learned fromTwitter Storm03/30/2013Robbie Cheng
  2. 2. Agenda• Why Twitter Storm• Real World Examples• Performance Gain• Lessons Learned• Future Works 2
  3. 3. Why Twitter Storm Queue Storm Hadoopmore concurrentjobs X V V highthroughput low (sync) (parallel) Vprogressmonitoring V Vpriority queue V ViTune Import(Stop/Resume) X VFault tolerance X V V 3
  4. 4. Real World Examples• simple query through social network API• detailed query through social network API• csv import/export 4
  5. 5. Performance Gain• simple query through social network API • daily throughput : 20MM • AWS instances : m1.large * 6• detailed query through social network API • daily throughput : 2MM • AWS instances : m1.large * 26 5
  6. 6. Performance Gain• Csv Export (60k) • queue : 4 min 33 sec • storm : 1 min 44 sec• Csv Import (60k) • TBD 6
  7. 7. Lessons Learned• simple task is better • http://en.wikipedia.org/wiki/File:AmdahlsLaw.svg• Batch processing is required if possible performance bottleneck• Infinite retry nightmare • Long transaction • Runtime Exception 7
  8. 8. Future Works• How do we scale out by adding more instances? 8
  9. 9. ConclusionStorm is not the silver bullet, itdepends on your requirements 9
  10. 10. Q&AWe’re hiring!jobs@fliptop.com 10

×