Xtreme Deployment

1,918 views

Published on

User generated data is an old problem. Systems and network telemetry, page analytics and application state combine to form an ever growing mountain of data collected by today's tools. Collecting and storing this data requires more than just a single application, having no single point where the user touches the system and gets an answer makes debugging a nightmare and reproducing the error intractable. Distributed systems require a clear perspective on production systems and access to data in real time to have any hope of solving complex problems related to state, all while not impacting user experience.

We will explain the problem, the pains and how we solved them. Develop in production; push code to development.

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,918
On SlideShare
0
From Embeds
0
Number of Embeds
43
Actions
Shares
0
Downloads
14
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Xtreme Deployment

  1. 1. Xtreme Deployment / develop in production; push to devTuesday, October 2, 12
  2. 2. Who are we? Brian Clapper CTO of Circonus Whipping boy of the modern SaaS stack. Theo Schlossnagle CEO of Circonus Repeat denier of reality.Tuesday, October 2, 12
  3. 3. How the world used to look (less xtreme) Architectures driven by user interaction: (e.g. web page load ➠ web page response)Tuesday, October 2, 12
  4. 4. How the world used to look (less xtreme) isolated user A’s page load doesn’t effect user B. latent user A’s affect shows up “later” in user B’s view.Tuesday, October 2, 12
  5. 5. How the world used to look (less xtreme) Problem reproducibility is achievable (even if sometimes painful)Tuesday, October 2, 12
  6. 6. How the world looks now (more xtreme) Architectures driven by high volume data and usersTuesday, October 2, 12
  7. 7. How the world looks now (more xtreme) every piece of data entering the architecture can effect a user’s subsequent experience.Tuesday, October 2, 12
  8. 8. How the world looks now (more xtreme) sometimes the data must effect a user’s subsequent experience.Tuesday, October 2, 12
  9. 9. How the world looks now (more xtreme) Social systems cause indirect impact via user action eroding isolation.Tuesday, October 2, 12
  10. 10. How the world looks now (more xtreme) A more responsive web is eroding users’ acceptance of latent state.Tuesday, October 2, 12
  11. 11. How the world looks now (more xtreme) Highly distributed systems (due to complexity and scalability) means no single world stateTuesday, October 2, 12
  12. 12. How the world looks now (more xtreme) Highly distributed systems (due to complexity and scalability) means difficult repeatability of state and failuresTuesday, October 2, 12
  13. 13. How the world looks now (more xtreme) Highly distributed systems (due to complexity and scalability) means treacherous debugabilityTuesday, October 2, 12
  14. 14. Despair all ye’ who debug here I studied distributed systems long enough to understand “the suck” you do not wish to have in your lives.Tuesday, October 2, 12
  15. 15. Despair all ye’ who debug here Expedient debugging requires a quick realization of a precipitating state of failure.Tuesday, October 2, 12
  16. 16. Despair all ye’ who debug here In distributed systems, post-mortem techniques are less useful because recreating possible global state is a mind-bending (or brain-damaging) exercise.Tuesday, October 2, 12
  17. 17. Despair all ye’ who debug here Applying older techniques to small controlled distributed systems is still feasible, but complex, asynchronous architectures can be nearly impossible.Tuesday, October 2, 12
  18. 18. Some context about us Circonus ingests telemetry data across an entire businessTuesday, October 2, 12
  19. 19. Some context about us alerting • correlation • visualization • analyticsTuesday, October 2, 12
  20. 20. How we see the world today High velocity non-user-generated data inflow.Tuesday, October 2, 12
  21. 21. How we see the world today Users expect sub-second reflection of all input data and drawn-conclusions.Tuesday, October 2, 12
  22. 22. How we see the world today Users expect extremely high availability (100%, you can only disappoint)Tuesday, October 2, 12
  23. 23. How we see the world today I was surprised how impatient and demanding users are... until I used the tool... and felt exactly the same way.Tuesday, October 2, 12
  24. 24. Circonus architecture VoltronTuesday, October 2, 12
  25. 25. Circonus architecture Voltron • distributed data collection (via brokers)Tuesday, October 2, 12
  26. 26. Circonus architecture Voltron • distributed data collection (via brokers) • distributed data ingestion (via aggregators)Tuesday, October 2, 12
  27. 27. Circonus architecture Voltron • distributed data collection (via brokers) • distributed data ingestion (via aggregators) • distributed data distribution! (via message queueing)Tuesday, October 2, 12
  28. 28. Circonus architecture Voltron • distributed data collection (via brokers) • distributed data ingestion (via aggregators) • distributed data distribution! (via message queueing) • distributed data storage (via snowth)Tuesday, October 2, 12
  29. 29. Circonus architecture Voltron • distributed data collection (via brokers) • distributed data ingestion (via aggregators) • distributed data distribution! (via message queueing) • distributed data storage (via snowth) • distributed streaming fault detection (via muppet love)Tuesday, October 2, 12
  30. 30. Circonus architecture Voltron • distributed data collection (via brokers) • distributed data ingestion (via aggregators) • distributed data distribution! (via message queueing) • distributed data storage (via snowth) • distributed streaming fault detection (via muppet love) • ... distributed nightmare.Tuesday, October 2, 12
  31. 31. Despair actualized I shall not memorialize this in slides.Tuesday, October 2, 12
  32. 32. Despair actualized I shall only cry in front of you now.Tuesday, October 2, 12
  33. 33. Hope (keys to the city) • Leveraging • stream mirroring and • idempotent behavior.Tuesday, October 2, 12
  34. 34. Hope Allows for the recklessness that we all appreciate in a no-consequences development environment...Tuesday, October 2, 12
  35. 35. Hope with production data, production volume, and production “personality.”Tuesday, October 2, 12
  36. 36. Hope actualized Development In ProductionTuesday, October 2, 12
  37. 37. Hope actualized Ex #1: Dual storage backendsTuesday, October 2, 12
  38. 38. EXPLORATION PostgreSQLTuesday, October 2, 12
  39. 39. EXPLORATION SnowthTuesday, October 2, 12
  40. 40. EXPLORATIONTuesday, October 2, 12 { Safety & Happiness
  41. 41. EXPLORATION Safety & HappinessTuesday, October 2, 12
  42. 42. Hope actualized Ex #2: Dual streaming analysis toolsTuesday, October 2, 12
  43. 43. HellTuesday, October 2, 12
  44. 44. HellTuesday, October 2, 12
  45. 45. HellTuesday, October 2, 12
  46. 46. HellTuesday, October 2, 12
  47. 47. HellTuesday, October 2, 12
  48. 48. HellTuesday, October 2, 12
  49. 49. Hope actualized Ex #3: Dual messages queueing systemsTuesday, October 2, 12
  50. 50. The Future This is the foundation of real responsive design going forward.Tuesday, October 2, 12
  51. 51. Thank You Questions?Tuesday, October 2, 12

×