Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Hootsuite Manages Its Growing Microservice Landscape


Published on

During our SOA transition at Hootsuite, we have noticed that visibility into our service relationships, dependencies and status is paramount to keeping our team, our build pipeline and application running smoothly. I’d like to share with you an API we baked into our SOA architecture that enables us to explore our applications service dependency graph in real time.

Published in: Software
  • Be the first to comment

  • Be the first to like this

How Hootsuite Manages Its Growing Microservice Landscape

  1. 1. Service Explosion HOW HOOTSUITE MANAGES ITS GROWING MICROSERVICE LANDSCAPE Specialist Software Developer - Mobile Web and APIs Adam Arsenault @Adam_Arsenault
  2. 2. ● Road to SOA ● Service Graph ● Voltron ● Demo What We’ll Talk about App S2 S3 S4 S1
  3. 3. ● Founded in 2008 ● PHP Monolith ● SOA started in 2013 ● Hyper growth ● Continuous Integration Road to SOA
  4. 4. “Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations” - Melvin Conway
  5. 5. Engagement MobilePublisher Analytics Platform Tools Labs Stable Teams
  6. 6. KABOOM!!!!
  7. 7. Ex. 1 - Integration Failures ● Dev merges changes ● Changes go to staging ● Integration tests fail ● Release pipeline frozen KABOOM!! !!
  8. 8. Ex. 2 - Production Downtime ● Service goes down in production ● On call / teams affected get notifications ● Sift through flood of notifications to figure out what’s broken KABOOM!! !!
  10. 10. The Service Graph
  11. 11. The Service Graph App
  12. 12. The Service Graph App S1 S2
  13. 13. The Service Graph App S2 S3 S4S1
  14. 14. The Service Graph App S2 S3 S4 S1
  15. 15. The Service Graph App S2 S3 S4 S1 S5 S6 S7
  16. 16. The Service Graph App S2 S3 S4 S1 S5 S6 S7
  17. 17. The Service Graph App S2 S3 S4 S1 S5 S6 S7
  18. 18. The API
  19. 19. Dependency Something that a service needs to function properly. Types: 1. Internal 2. Traversable App Cachedb S1
  20. 20. /status/about Returns metadata information about the service or app such as version, description, maintainers, links to documentation, and gets the status of each individual dependency.
  21. 21. /status/:dependency A configured status endpoint at '/status/:dependency Examples: ● '/status/service-core' ● '/status/db' OK
  22. 22. /status/aggregate Returns the overall status by checking all registered status checks and giving a simple response. Examples: ● OK ● CRIT - error message OK
  23. 23. /status/traverse Enables service graph traversal and execution of an "action" at the last level of traversal. App S2 S3 S4 S1
  24. 24. Usage
  25. 25. Monitor OK / CRIT
  26. 26. Debug S1 db S2
  27. 27. Explore and Learn App S2 S3 S4 S1 S5 S6 S7
  28. 28. Document
  29. 29. Status of single machine Overall status of application and services Alerts / notifications Troubleshoot by drilling down Monitoring Strategy
  30. 30. Technologies
  31. 31. Architecture Browser 1 Browser N... Play App Status Poller Actor WS Actor 1 WS Actor N ...
  32. 32. DEMO
  33. 33. “When there is a production issue, I see lots of people go to Voltron to perform diagnostics on what might be wrong” Geordie Henderson - VP Software Development
  34. 34. “Voltron is often the first to tell us when snowflake is down” Brandon Okert - Junior Software Developer Publisher
  35. 35. “When a critical service goes down, everything starts alerting and reporting problems, but Voltron gets through the noise by letting you drill down” Michael Reid - Senior Software Developer Platform
  36. 36. “We suspected the connection between dashboard and Billing Service was broken, but Voltron told us the communication channel was okay.” Martin Jung - Software Developer Mobile Web and APIs
  37. 37. Lessons Learned
  38. 38. • Productivity • Happiness Visibility Empowers
  39. 39. • Automate all the things • Identify problems early and fix • 10x factor SOA Tools Early
  40. 40. • Standardize • Add to a service framework • Share common status checks Make Checking Status Easy
  41. 41. • Synchronized views • Performance Websockets for Real time
  42. 42. Future Work
  43. 43. Analytics
  44. 44. Real Time Graph View
  45. 45. Open Source
  46. 46. Thank you! Questions? Specialist Software Developer - Mobile Web and APIs @Adam_Arsenault Adam Arsenault