Gluecon 2013 - Dark Architecture and How to Forklift Upgrade Your System - Dyn Inc

2,430 views

Published on

Dyn's CTO Cory von Wallenstein walks through how to evolve a system architecture for scale, performance and looser coupling without putting the business at risk and while keeping high tech team morale using a Dark Architecture approach.

Published in: Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,430
On SlideShare
0
From Embeds
0
Number of Embeds
46
Actions
Shares
0
Downloads
19
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Gluecon 2013 - Dark Architecture and How to Forklift Upgrade Your System - Dyn Inc

  1. 1. Dark Architecture & How to Forklift UpgradeYour Infrastructure with Zero DowntimeCory von WallensteinChief Technology Officer,Dyn Inc.@cvonwallenstein@cvonwallenstein from @DynInc at #gluecon
  2. 2. But First, Who Is Dyn?• Internet Infrastructure as a Service– Managed DNS and Email Delivery• 230 Global Employees (we bootstrapped to 170)• Headquarters in Manchester, NH (offices in SFO & UK too)• Raised first financing in Oct 2012: $38MM from NorthBridge@cvonwallenstein from @DynInc at #gluecon
  3. 3. Problem We Are Trying To SolveInputsBlack Magic(Your Current System Architecture)OutputsDifferent Black Magic(Your New System Architecture)InputsInputsInputsOutputsOutputsOutputsScalex10, x102, etc.Performance(t2 - t0) <= (t1 - t0)t1t2t0t0CouplingTight -> Loose@cvonwallenstein from @DynInc at #gluecon
  4. 4. Pragmatic Engineeringover Unicorn Marketing@cvonwallenstein from @DynInc at #gluecon
  5. 5. Why Things Get This Way• Time to market reigns supreme– MVP was very… minimum… on… everything– Sooner is better than perfect• Prototype to production to scale withoutarchitectural rigor– Skillset for system engineering in high demand• Seen more often in small teams who findproduct market fit faster than expected– Inexperience, but we’ve all been there@cvonwallenstein from @DynInc at #gluecon
  6. 6. Dark Architecture• A way of thinking about, and technicalapproach to, solving thescale/performance/coupling problem whileenabling the business to succeed and keeping(some) of your hair• We stand on shoulders of giants– Fowler, Amazon, Netflix, etc.@cvonwallenstein from @DynInc at #gluecon
  7. 7. High Level of Dark Architecture• Legacy approach: Flag Day Upgrade/Deploy– Scope out 3 month upgrade to swap architecture Ato B, turns into 6 months, don’t get to anything else,cross fingers on flag day, fight fires where broken,gain weight, lose hair, girlfriend breaks up with you,team quits, FML…• Evolved approach: Fowler’s Blue/Green Deploy– Two copies of system, load balancing to rapidlydeploy new system version, rapidly fail back tolegacy on failure (only one active at a time)@cvonwallenstein from @DynInc at #gluecon
  8. 8. High Level of Dark Architecture• Dark Architecture Approach– Two copies of system, both active, send inputs for aworkflow to both, compare outputs and throw oneaway (the one you threw the output away from isthe “dark architecture”), log and inspect outputdifferences, gain confidence in new system whendifferences go away, swap which output you throwaway (effectively bringing the “dark” architecture“light”), achieve equilibrium on what workflows getprocessed by what system so your business hasflexibility, high five everyone, onward and upward.@cvonwallenstein from @DynInc at #gluecon
  9. 9. Tangible Examples• Scaling Global DNS Stats beyond 17 POPs– MySQL to Cassandra, Log file rsync to agg counts@cvonwallenstein from @DynInc at #gluecon
  10. 10. Tangible Examples• Scaling Email Delivery beyond 1 billion/month– Cron to daemon (2011), Perl to Node.js (now)
  11. 11. Dark Architecture Manifesto1. Clear definition of success over ambiguity– Likely scale/performance measured, may getblank stares on coupling2. Continuously deliver value over months of novisible progress3. Confidence in functional equivalence overscope creep4. ^5’s over finger pointing5. Plan for failure over cross fingers@cvonwallenstein from @DynInc at #gluecon
  12. 12. Dark Architecture Manifesto6. Customer impact over elegant systemdiagrams7. System flows over system components8. Operational confidence and familiarity overtrial by fire9. Having a ten item list over a nine item list10. Architecture evolution over architecturerevolution@cvonwallenstein from @DynInc at #gluecon
  13. 13. Scope and Priority• Prioritize a backlog of input/output workflowsby amount of pain– Don’t think on a system component level• “swap MySQL for Cassandra”– Think on a system workflow level• “retrieve query logs and render *.example.com graphs”– This exercise will force you to hone scope toexactly where the pain is so you can focus ondelivering the solution to this pain first and saveothers for later.@cvonwallenstein from @DynInc at #gluecon
  14. 14. Legacy Approach@cvonwallenstein from @DynInc at #gluecon
  15. 15. Legacy Approach: Week 0Legacy System100% of functionalityenabled100% of functionalityconsumed@cvonwallenstein from @DynInc at #gluecon
  16. 16. Legacy Approach: Week 1Legacy System100% of functionalityenabled100% of functionalityconsumedNew System0% of functionalityenabled0% of functionalityconsumed@cvonwallenstein from @DynInc at #gluecon
  17. 17. Legacy Approach: Week 4Legacy System100% of functionalityenabled100% of functionalityconsumedNew System25% of functionalityenabled0% of functionalityconsumedMost people start with easy pieces under amisguided “crawl walk run” philosophy. Quickwins on easy stuff while saving hard problemsfor later rarely ends well.@cvonwallenstein from @DynInc at #gluecon
  18. 18. Legacy Approach: Week 8Legacy System100% of functionalityenabled100% of functionalityconsumedNew System35% of functionalityenabled0% of functionalityconsumedProgress slows as harder problems encountered@cvonwallenstein from @DynInc at #gluecon
  19. 19. Legacy Approach: Week 12Legacy System100% of functionalityenabled100% of functionalityconsumedNew System80% of functionalityenabled0% of functionalityconsumed80% of projects spend 80% of their calendar timeat 80% perceived completion. I’m 80% sure.@cvonwallenstein from @DynInc at #gluecon
  20. 20. Legacy Approach: Week 24Legacy System100% of functionalityenabled100% of functionalityconsumedNew System100% of functionalityenabled0% of functionalityconsumedOther fires came up, things took longer thanexpected, you know… business. Morale neverbeen lower@cvonwallenstein from @DynInc at #gluecon
  21. 21. Legacy Approach: Flag Day!Legacy System100% of functionalityenabled0% of functionalityconsumedNew System100% of functionalityenabled100% of functionalityconsumed@cvonwallenstein from @DynInc at #gluecon
  22. 22. Legacy Approach: Flag Day!Legacy System100% of functionalityenabled0% of functionalityconsumedNew System100% of functionalityenabled100% of functionalityconsumed@cvonwallenstein from @DynInc at #gluecon
  23. 23. Dark Architecture Approach@cvonwallenstein from @DynInc at #gluecon
  24. 24. Dark Architecture Approach: Week 0Legacy System100% of functionalityenabled100% of functionalityconsumed@cvonwallenstein from @DynInc at #gluecon
  25. 25. Dark Architecture Approach: Week 1Legacy System100% of functionalityenabled100% of functionalityconsumedNew System0% of functionalityenabled0% of functionalityconsumed@cvonwallenstein from @DynInc at #gluecon
  26. 26. Dark Architecture Approach: Week 2Legacy System100% of functionalityenabled100% of functionalityconsumedNew System0% of functionalityenabled0% of functionalityconsumedNo functionality yet,just dark architectureframework for twoinputs and twooutputs (throwingone output away)
  27. 27. Dark Architecture Approach: Week 3Legacy System100% of functionalityenabled100% of functionalityconsumedNew System2% of functionalityenabled2% of functionalityconsumed (dark)Throw one away, but logand inspect differences!
  28. 28. Dark Architecture Approach: Week 4Legacy System100% of functionalityenabled98% of functionalityconsumedNew System2% of functionalityenabled2% of functionalityconsumedGain confidenceoperating with twoequal outputs, switchwhich one is thrownaway for that workflow.Goes horribly wrong?Switch back.
  29. 29. Dark Architecture Approach: Week 12Legacy System100% of functionalityenabled80% of functionalityconsumedNew System20% of functionalityenabled20% of functionalityconsumedWhere do we stand atexpected 3 months?Most painful 20% ofproblems resolved…now we haveflexibility for what todo next.
  30. 30. Customer impact over elegantsystem diagrams• Your customers are not paying you to havepretty whiteboards of elegant systemarchitectures• Your customers are paying you to make theirpain go away. This gets priority.• It’s OK to have different workflows handled bydifferent systems to give your team agility– Other priorities came up? System is stable.– Have technical debt time? Continue arch migration@cvonwallenstein from @DynInc at #gluecon
  31. 31. Parting Takeaways• Manifesto is a preference, not a rule• Think in flows not components• Deliver most painful pieces first so whenpriorities change, you’re not left half complete.• Process success >>> process name• Be realistic. DA provides flexibility and frequentvictories for morale and some value deliveredsooner, but it won’t necessarily make a fullarchitecture migration faster in calendar days.@cvonwallenstein from @DynInc at #gluecon
  32. 32. Cory von Wallenstein@cvonwallensteinQuestions?@cvonwallenstein from @DynInc at #gluecon

×