Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Monitoring theClouds   Gary Dusbabek   @gdusbabek
OutlineMotivation &FeaturesTechnologyPrognosticate
Motivation
Thousandsof servers
Complex Eventpipeline
Existing Solutions       New             Expensive                         but                         old                ...
Our Goals
Internal   vsexternal   Millions   of signals                        !
Geographicdistribution     &Redundancy
Simultaneous!Redundant Processing    & analysis
Fail!!safely
ResultsWhat we ended up with
Concurrent & redundant    event analysis
Robust! metricscollection
Remote collectorsin five datacenters!         Agent collectors anywhere!                https://github.com/racker/virgo
Advancing state of the art
How we did itTeam!&!Culture
How we did it No idea!     is!!too stupid
How we did itSlippery!iteration
How we did itReduceFriction
How we did it    Well-greased    Deploymentshttps://github.com/etsy/deployinatorhttps://github.com/racker/dreadnot        ...
How we did itWell-greasedDeployments     30 Days   146 Deploys  5 Datacenters
Nuts!&!Bolts!
Nuts!&!Bolts!            Cassandra!                      !                5x!replica5on!                3!Datacenter!
Nuts!&!Bolts!                Node.js!
Nuts!&!Bolts!             Reconnoiter!                 +!           Custom!ingestor!
Nuts!&!Bolts!              Esper!                 +!            Magic!sauce!
Nuts!&!Bolts!              Our!code!                       !          With!help!from!Zookeeper!
Nuts!&!Bolts!            Virgo!            hGps://github.com/racker/virgo!            !            Lua!            Luvit!e...
Nuts!&!Bolts!                Metrics!                inges5on!&!                processing!                !              ...
Nuts!&!Bolts!Mul5!DC!
Nuts!&!Bolts!Message!Redundancy!
Nuts!&!Bolts!Agent!crossRDC!
Prognostication
2011!Gartner!     Big Data!!“Emerging Technology”
IngestionSolved problem
Data!!   vsKnowledge
Insight    !powerful  rare
CorrelationPredictionAutomation!Aggregation
Bitrot So much datacan never change   databases
It will  getworse
Questions?@gdusbabek
Image!Credits!                                 !       All!images!for!this!presenta5on!come!from!the!Flickr!commons!collec...
Upcoming SlideShare
Loading in …5
×

Rackspace Cloud Monitoring - Strata NYC

1,799 views

Published on

Published in: Technology
  • Be the first to comment

Rackspace Cloud Monitoring - Strata NYC

  1. 1. Monitoring theClouds Gary Dusbabek @gdusbabek
  2. 2. OutlineMotivation &FeaturesTechnologyPrognosticate
  3. 3. Motivation
  4. 4. Thousandsof servers
  5. 5. Complex Eventpipeline
  6. 6. Existing Solutions New Expensive but old ! Lessons from Cloudkick
  7. 7. Our Goals
  8. 8. Internal vsexternal Millions of signals !
  9. 9. Geographicdistribution &Redundancy
  10. 10. Simultaneous!Redundant Processing & analysis
  11. 11. Fail!!safely
  12. 12. ResultsWhat we ended up with
  13. 13. Concurrent & redundant event analysis
  14. 14. Robust! metricscollection
  15. 15. Remote collectorsin five datacenters! Agent collectors anywhere! https://github.com/racker/virgo
  16. 16. Advancing state of the art
  17. 17. How we did itTeam!&!Culture
  18. 18. How we did it No idea! is!!too stupid
  19. 19. How we did itSlippery!iteration
  20. 20. How we did itReduceFriction
  21. 21. How we did it Well-greased Deploymentshttps://github.com/etsy/deployinatorhttps://github.com/racker/dreadnot
  22. 22. How we did itWell-greasedDeployments 30 Days 146 Deploys 5 Datacenters
  23. 23. Nuts!&!Bolts!
  24. 24. Nuts!&!Bolts! Cassandra! ! 5x!replica5on! 3!Datacenter!
  25. 25. Nuts!&!Bolts! Node.js!
  26. 26. Nuts!&!Bolts! Reconnoiter! +! Custom!ingestor!
  27. 27. Nuts!&!Bolts! Esper! +! Magic!sauce!
  28. 28. Nuts!&!Bolts! Our!code! ! With!help!from!Zookeeper!
  29. 29. Nuts!&!Bolts! Virgo! hGps://github.com/racker/virgo! ! Lua! Luvit!even5ng! Node.js!backend!
  30. 30. Nuts!&!Bolts! Metrics! inges5on!&! processing! ! Java! 20M!signals/min!
  31. 31. Nuts!&!Bolts!Mul5!DC!
  32. 32. Nuts!&!Bolts!Message!Redundancy!
  33. 33. Nuts!&!Bolts!Agent!crossRDC!
  34. 34. Prognostication
  35. 35. 2011!Gartner! Big Data!!“Emerging Technology”
  36. 36. IngestionSolved problem
  37. 37. Data!! vsKnowledge
  38. 38. Insight !powerful rare
  39. 39. CorrelationPredictionAutomation!Aggregation
  40. 40. Bitrot So much datacan never change databases
  41. 41. It will getworse
  42. 42. Questions?@gdusbabek
  43. 43. Image!Credits! ! All!images!for!this!presenta5on!come!from!the!Flickr!commons!collec5on! hGp://www.flickr.com/commons/Clouds ! ! !hGp://www.flickr.com/photos/sdasmarchives/7316444746!Biplane ! ! !hGp://www.flickr.com/photos/sdasmarchives/7316444552!Umbrellas ! ! !hGp://www.flickr.com/photos/statelibraryofnsw/4944459226!servers ! ! !hGp://www.flickr.com/photos/na5onaalarchief/4026288398!complex ! ! !hGp://www.flickr.com/photos/lselibrary/3990093924!pyramid ! ! !hGp://www.flickr.com/photos/brooklyn_museum/2488800747!kites ! ! ! !hGp://www.flickr.com/photos/usna5onalarchives/5506532956!target! ! ! !hGp://www.flickr.com/photos/nlireland/5963130143!map ! ! ! !hGp://www.flickr.com/photos/sdasmarchives/5018415335!safety! ! ! !hGp://www.flickr.com/photos/usna5onalarchives/3660777810!reports ! ! !hGp://www.flickr.com/photos/usna5onalarchives/6990370962!apparatus ! ! !hGp://www.flickr.com/photos/library_of_congress/2179184290!robust ! ! !hGp://www.flickr.com/photos/twm_news/5863492976!old!computer ! !hGp://www.flickr.com/photos/nasacommons/7605906886!dogs ! ! ! !hGp://www.flickr.com/photos/smithsonian/4010655153!team ! ! ! !hGp://www.flickr.com/photos/floridamemory/6730955597!oil ! ! ! !hGp://www.flickr.com/photos/usna5onalarchives/7065894867!classroom ! ! !hGp://www.flickr.com/photos/library_of_congress/2179067948!horses ! ! !hGp://www.flickr.com/photos/powerhouse_museum/2581833315!skeleton ! ! !hGp://www.flickr.com/photos/powerhouse_museum/2980051095!washing!machine ! !hGp://www.flickr.com/photos/na5onaalarchief/4397795513!genie ! ! ! !hGp://www.flickr.com/photos/library_of_congress/7267723488!waste! ! ! !hGp://www.flickr.com/photos/nlscotland/4699984659!truck!waste ! ! !hGp://www.flickr.com/photos/usna5onalarchives/3928424307!ques5ons ! ! !hGp://www.flickr.com/photos/nlscotland/4698823475!

×