Lesson learned of twitter storm

878 views

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
878
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Please update to include customer name and the date you’re presenting.
  • Lesson learned of twitter storm

    1. 1. Lessons learned fromTwitter Storm03/30/2013Robbie Cheng
    2. 2. Agenda• Why Twitter Storm• Real World Examples• Performance Gain• Lessons Learned• Future Works 2
    3. 3. Why Twitter Storm Queue Storm Hadoopmore concurrentjobs X V V highthroughput low (sync) (parallel) Vprogressmonitoring V Vpriority queue V ViTune Import(Stop/Resume) X VFault tolerance X V V 3
    4. 4. Real World Examples• simple query through social network API• detailed query through social network API• csv import/export 4
    5. 5. Performance Gain• simple query through social network API • daily throughput : 20MM • AWS instances : m1.large * 6• detailed query through social network API • daily throughput : 2MM • AWS instances : m1.large * 26 5
    6. 6. Performance Gain• Csv Export (60k) • queue : 4 min 33 sec • storm : 1 min 44 sec• Csv Import (60k) • TBD 6
    7. 7. Lessons Learned• simple task is better • http://en.wikipedia.org/wiki/File:AmdahlsLaw.svg• Batch processing is required if possible performance bottleneck• Infinite retry nightmare • Long transaction • Runtime Exception 7
    8. 8. Future Works• How do we scale out by adding more instances? 8
    9. 9. ConclusionStorm is not the silver bullet, itdepends on your requirements 9
    10. 10. Q&AWe’re hiring!jobs@fliptop.com 10

    ×