DEPLOYING NEXT-GENERATION SYSTEMS    TWILIO ENGINEERING
WHY NEXTGENERATION SYSTEMS?        #twiliocon
125,000,000                                                                    123,090,34100,000,000                      ...
WHY ZERODOWNTIME?       #twiliocon
HI. I’M TIMI’m Director of Engineering at twilio                              #twiliocon
THE CHALLENGE• Design, build, deploy replacements of 2 core systems.                                                 #twil...
THE CHALLENGE• Design, build, deploy replacements of 2 core systems.    ➡ They must be HA.                                ...
THE CHALLENGE• Design, build, deploy replacements of 2 core systems.    ➡ They must be HA.    ➡ They must be horizontally-...
THE CHALLENGE• Design, build, deploy replacements of 2 core systems.    ➡ They must be HA.    ➡ They must be horizontally-...
THE CHALLENGE• Design, build, deploy replacements of 2 core systems.    ➡ They must be HA.    ➡ They must be horizontally-...
HI. I’M FRANKI’m API-Team Engineering Lead at twilio                              #twiliocon
OUT WITH THE OLD...               #twiliocon
OUT WITH THE OLD...        • Monolithic Codebase                       #twiliocon
OUT WITH THE OLD...        • Monolithic Codebase        • Serving Millions of API Requests                         #twilio...
... AND IN WITH THE NEW                 #twiliocon
... AND IN WITH THE NEW• Python + Flask                   #twiliocon
... AND IN WITH THE NEW• Python + Flask• Designed to serve billions of  API requests                                  #twi...
... AND IN WITH THE NEW• Python + Flask• Designed to serve billions of  API requests• Zero Downtime, Zero Regressions     ...
TEST, TEST, TEST             #twiliocon
TEST, TEST, TEST• Unit & Functional tests for local development                                                  #twiliocon
TEST, TEST, TEST• Unit & Functional tests for local development• The same tests run in our Staging Cluster                ...
TEST, TEST, TEST• Unit & Functional tests for local development• The same tests run in our Staging Cluster• The same clust...
A TALE OF TWO APIS              #twiliocon
A TALE OF TWO APISCLUSTER TESTS                  #twiliocon
A TALE OF TWO APISCLUSTER TESTS                  #twiliocon
A TALE OF TWO APISCLUSTER TESTS                  #twiliocon
A TALE OF TWO APIS                     DIFF RESULTSCLUSTER TESTS                  #twiliocon
A TALE OF TWO APIS                     DIFF RESULTSCLUSTER TESTS                  #twiliocon
NGINX.CONF
NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stac...
NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stac...
NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stac...
NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stac...
NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stac...
NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stac...
SPOT THE DIFFERENCE                 GET /2010-04-01/Accounts/ACaaaaaaaaaaaaaaaaaaaaaaaaa/Applications/APaaaaaaaaaaaaaaaaaa...
SPOT THE DIFFERENCE    GET /2010-04-01/Accounts/ACaaaaaaaaaaaaaaaaaaaaaaaaa/Applications/APaaaaaaaaaaaaaaaaaaa  {TwilioRes...
ALL GROWN UP          #twiliocon
ALL GROWN UP     API           #twiliocon
ALL GROWN UP      API   UNIT TESTS                #twiliocon
ALL GROWN UP       API    UNIT TESTS FUNCTIONAL TESTS                 #twiliocon
ALL GROWN UP       API    UNIT TESTS FUNCTIONAL TESTS    DIFF TESTS                 #twiliocon
ALL GROWN UP       API    UNIT TESTS FUNCTIONAL TESTS    DIFF TESTS  CANARY DEPLOY                  #twiliocon
ALL GROWN UP       API    UNIT TESTS FUNCTIONAL TESTS    DIFF TESTS  CANARY DEPLOY       API                  #twiliocon
ALL GROWN UP       API    UNIT TESTS FUNCTIONAL TESTS    DIFF TESTS  CANARY DEPLOY       API                  #twiliocon
Flask-RESTful• A simple REST resource framework for python Flask applications• Simplifies argument parsing, generating out...
HI. I’M ADAMI’m MicroTransactions-Team Engineering Lead at twilio                                     #twiliocon
REWIND TO 2009
NERDY TELECOMSTARTUP SEEKS  RELIABLE HA         #twiliocon
MICRO-TRANSACTIONS @TWILIO                   #twiliocon
MICRO-TRANSACTIONS @TWILIO                   #twiliocon
MICRO-TRANSACTIONS @TWILIO                   #twiliocon
TRANSACTIONS @TWILIO V1                      BALANCE                       AGGREGAT                      UPDATER          ...
TRANSACTIONS @TWILIO V1                      BALANCE                       AGGREGAT                      UPDATER          ...
TRANSACTIONS @TWILIO V1                      BALANCE                       AGGREGAT                      UPDATER          ...
TX-                           TX-              MYSQL        INSERT only log dataAutomatically increments                  ...
ONLINE SYSTEM UPGRADES                #twiliocon
ONLINE SYSTEM UPGRADES• Double book-keeping with  Shadow Mode                             #twiliocon
ONLINE SYSTEM UPGRADES• Double book-keeping with  Shadow Mode• Rollback support with  Account Flags & Feature Flags       ...
ONLINE SYSTEM UPGRADES• Double book-keeping with  Shadow Mode• Rollback support with  Account Flags & Feature Flags• Gradu...
1. SHADOW MODE TX     TX-MASTER-QUEUE    DEQUEUER                      NGB     LEGACY                     QUEUE     QUEUE ...
1. SHADOW MODE TX     TX-MASTER-QUEUE    DEQUEUER                              ATOMIC         faucet controls             ...
Sid     Account Sid   Old System   New System                 SV...      AC...        29.5500      29.5500                ...
Sid     Account Sid   Old System   New System                 SV...      AC...        29.5500      29.5500                ...
2. ROLLBACK SUPPORT     Toggle Account Features                               #twiliocon
2. ROLLBACK SUPPORT      Toggle Cluster Features                                #twiliocon
3. GRADUAL ROLLOUT (PRACTICEe Week 1   Week 2    Week 3   Week 4   Week 5   Week 6    Week 7    Week 8            # of Err...
BETTER TOOLS                  Problem                                Solution• Redis killed by Linux OOM Killer       • Ro...
THANK YOUCOME ASK US QUESTIONS!
Deploying Next Gen Systems with Zero Downtime
Upcoming SlideShare
Loading in...5
×

Deploying Next Gen Systems with Zero Downtime

1,763

Published on

Hear what we learned from deploying next generation systems at Twilio, with zero downtime.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,763
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Lets rewind to 2009 and take a look at what we built.\n
  • At Twilio\n- \nnerd out to billing everyday. \n\nIts odd, quirky, but super powerful \n\n- mission critical to the advancement of twilio.\n\nWe decided to invest in building this early on in the company.\n
  • \n
  • high availability\nalways be processing\n
  • Realtime scoreboards, usage, metrics\n
  • Two piggy banks\nSingle process dequeuers - limited by the number of transactions you can process on a single database\n
  • Two piggy banks\nSingle process dequeuers - limited by the number of transactions you can process on a single database\n
  • \n
  • Once we understood the problems...\n\nWent to the drawing board...\n\nWe built something new to replace the infrastructure alongside it.\n\n&lt;button&gt;\n\nA system that disconnects our dequeuers from our processors. \n\n&lt;button&gt; \n\nProcessors powered by a REST API and uses status codes for success &amp; error resolution.\n\n&lt;button&gt;\n\nOnly inserting into our databases as a log server.\n\n&lt;button&gt;\n\nAnd to fix our realtime issue, we&amp;#x2019;re using redis as an in-flight datastore to atomically process metrics as we process transactions.\n
  • \n
  • \n
  • \n
  • Now we have two systems side-by-side, but we need to compare the two.\n&lt;click&gt;\nDouble book keeping lets us compare balances and metrics.\n\nIf we have a bug,\n&lt;click&gt;\nwe need to rollback. \n\nAnd we don&amp;#x2019;t want to do it all at once.\n&lt;click&gt;\n\n
  • Now we have two systems side-by-side, but we need to compare the two.\n&lt;click&gt;\nDouble book keeping lets us compare balances and metrics.\n\nIf we have a bug,\n&lt;click&gt;\nwe need to rollback. \n\nAnd we don&amp;#x2019;t want to do it all at once.\n&lt;click&gt;\n\n
  • Now we have two systems side-by-side, but we need to compare the two.\n&lt;click&gt;\nDouble book keeping lets us compare balances and metrics.\n\nIf we have a bug,\n&lt;click&gt;\nwe need to rollback. \n\nAnd we don&amp;#x2019;t want to do it all at once.\n&lt;click&gt;\n\n
  • With so much throughput, we couldn&amp;#x2019;t just shut down the billing system. We also couldn&amp;#x2019;t lose a billing event.\n&lt;click&gt;\nSo we built an abstraction between the two systems that would allow us to atomically control transactions.\n&lt;click&gt;\nWhen the faucet is turned off, it will wait till both queues are drained and send us a report.\n&lt;click&gt;\n
  • With so much throughput, we couldn&amp;#x2019;t just shut down the billing system. We also couldn&amp;#x2019;t lose a billing event.\n&lt;click&gt;\nSo we built an abstraction between the two systems that would allow us to atomically control transactions.\n&lt;click&gt;\nWhen the faucet is turned off, it will wait till both queues are drained and send us a report.\n&lt;click&gt;\n
  • Check the books. Can we turn any accounts online?\n
  • Nope, we should not enable any accounts yet.\n
  • If we had moved accounts, with a click, we can migrate them back.\n
  • Or if the cluster catches fire, we can turn off the entire new system and reroute billing traffic back to its legacy system.\n\n
  • &lt;TODO&gt;\nGraph build out each week with the story.\n\n\n\n\nPractice is good.\nWe tested our platform thoroughly in a practice mode with no account flags turned on.\n\nAs we progressed and fixed the edge-cases, we migrated 1%, %5, all the way up to all accounts over a period of time.\n\nPlanning with your tools lets you build a gradual deployment with ease.\n\n
  • Just to follow up\n\nbetter tools equal better deployments.\n\nWhen we had issues with our new in-flight store, we had a way to rollback.\nWhen we were seeing discrepancies in balances, we would investigate, fix, deploy, and compare.\n\nSo you get the idea, to migrate to a new micro-payments platform,\nwe must engineer tools that let us migrate back and\nforth with ease so that we can spend time on the solutions.\n
  • \n
  • Deploying Next Gen Systems with Zero Downtime

    1. 1. DEPLOYING NEXT-GENERATION SYSTEMS TWILIO ENGINEERING
    2. 2. WHY NEXTGENERATION SYSTEMS? #twiliocon
    3. 3. 125,000,000 123,090,34100,000,000 4 75,000,000 67,186,111 e 50,000,000 25,679,631 25,000,000 365,782 0 Dec 2009 Dec 2010 Dec 2011 Dec 2012 Twilio Transactions Per Month
    4. 4. WHY ZERODOWNTIME? #twiliocon
    5. 5. HI. I’M TIMI’m Director of Engineering at twilio #twiliocon
    6. 6. THE CHALLENGE• Design, build, deploy replacements of 2 core systems. #twiliocon
    7. 7. THE CHALLENGE• Design, build, deploy replacements of 2 core systems. ➡ They must be HA. #twiliocon
    8. 8. THE CHALLENGE• Design, build, deploy replacements of 2 core systems. ➡ They must be HA. ➡ They must be horizontally-scalable. #twiliocon
    9. 9. THE CHALLENGE• Design, build, deploy replacements of 2 core systems. ➡ They must be HA. ➡ They must be horizontally-scalable. #twiliocon
    10. 10. THE CHALLENGE• Design, build, deploy replacements of 2 core systems. ➡ They must be HA. ➡ They must be horizontally-scalable.... Oh, and don’t lose a single billing event or API request in the process. #twiliocon
    11. 11. HI. I’M FRANKI’m API-Team Engineering Lead at twilio #twiliocon
    12. 12. OUT WITH THE OLD... #twiliocon
    13. 13. OUT WITH THE OLD... • Monolithic Codebase #twiliocon
    14. 14. OUT WITH THE OLD... • Monolithic Codebase • Serving Millions of API Requests #twiliocon
    15. 15. ... AND IN WITH THE NEW #twiliocon
    16. 16. ... AND IN WITH THE NEW• Python + Flask #twiliocon
    17. 17. ... AND IN WITH THE NEW• Python + Flask• Designed to serve billions of API requests #twiliocon
    18. 18. ... AND IN WITH THE NEW• Python + Flask• Designed to serve billions of API requests• Zero Downtime, Zero Regressions #twiliocon
    19. 19. TEST, TEST, TEST #twiliocon
    20. 20. TEST, TEST, TEST• Unit & Functional tests for local development #twiliocon
    21. 21. TEST, TEST, TEST• Unit & Functional tests for local development• The same tests run in our Staging Cluster #twiliocon
    22. 22. TEST, TEST, TEST• Unit & Functional tests for local development• The same tests run in our Staging Cluster• The same cluster tests run against Both API Frameworks #twiliocon
    23. 23. A TALE OF TWO APIS #twiliocon
    24. 24. A TALE OF TWO APISCLUSTER TESTS #twiliocon
    25. 25. A TALE OF TWO APISCLUSTER TESTS #twiliocon
    26. 26. A TALE OF TWO APISCLUSTER TESTS #twiliocon
    27. 27. A TALE OF TWO APIS DIFF RESULTSCLUSTER TESTS #twiliocon
    28. 28. A TALE OF TWO APIS DIFF RESULTSCLUSTER TESTS #twiliocon
    29. 29. NGINX.CONF
    30. 30. NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stack $requested_stack_default_php { default @php; python @python;}
    31. 31. NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stack $requested_stack_default_php { default @php; python @python;}
    32. 32. NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stack $requested_stack_default_php { default @php; python @python;}
    33. 33. NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stack $requested_stack_default_php { default @php; python @python;}
    34. 34. NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stack $requested_stack_default_php { default @php; python @python;}location @python { proxy_pass 127.0.0.1:5555;}location @php { proxy_pass 127.0.0.1:12345;}
    35. 35. NGINX.CONF# Map the HTTP header X-Requested-Api-Stack: <value># to a named location in nginxmap $http_x_requested_api_stack $requested_stack_default_php { default @php; python @python;}location @python { proxy_pass 127.0.0.1:5555;}location @php { proxy_pass 127.0.0.1:12345;}location ~ / { try_files Kwijibo $requested_stack_default_php;}
    36. 36. SPOT THE DIFFERENCE GET /2010-04-01/Accounts/ACaaaaaaaaaaaaaaaaaaaaaaaaa/Applications/APaaaaaaaaaaaaaaaaaaa{TwilioResponse: {TwilioResponse:[{Application: [ [{Application: [ {Sid: APaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}, {Sid: APaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}, {DateCreated: Mon, 22 Aug 2011 20:58:45 +0000}, {DateCreated: Mon, 22 Aug 2011 20:58:45 +0000}, {DateUpdated: Mon, 22 Aug 2011 20:58:45 +0000}, {DateUpdated: Mon, 22 Aug 2011 20:58:45 +0000}, {AccountSid: ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}, {AccountSid: ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}, {FriendlyName: Application Friendly Name}, {FriendlyName: Application Friendly Name}, {ApiVersion: 2010-04-01}, {ApiVersion: 2010-04-01}, {VoiceUrl: http://www.example.com/voice}, {VoiceUrl: http://www.example.com/voice}, {VoiceMethod: GET}, {VoiceMethod: GET}, {VoiceFallbackUrl: http://www.example.com/voice-callback}, {VoiceFallbackUrl: http://www.example.com/voice-callback}, {VoiceFallbackMethod: GET}, {VoiceFallbackMethod: GET}, {StatusCallback: http://www.example.com/status-callback}, {StatusCallback: http://www.example.com/status-callback}, {StatusCallbackMethod: GET}, {StatusCallbackMethod: GET}, {VoiceCallerIdLookup: false}, {VoiceCallerIdLookup: False}, {SmsUrl: http://www.example.com/sms}, {SmsUrl: http://www.example.com/sms}, {SmsMethod: GET}, {SmsMethod: GET}, {SmsFallbackUrl: http://www.example.com/sms-fallback}, {SmsFallbackUrl: http://www.example.com/sms-fallback}, {SmsFallbackMethod: GET}, {SmsFallbackMethod: GET}, {SmsStatusCallback: http://www.example.com/sms-status-callback}, {SmsStatusCallback: http://www.example.com/sms-status-callback},
    37. 37. SPOT THE DIFFERENCE GET /2010-04-01/Accounts/ACaaaaaaaaaaaaaaaaaaaaaaaaa/Applications/APaaaaaaaaaaaaaaaaaaa {TwilioResponse: [{Application: [ {Sid: APaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}, {DateCreated: Mon, 22 Aug 2011 20:58:45 +0000}, {DateUpdated: Mon, 22 Aug 2011 20:58:45 +0000}, {AccountSid: ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}, {FriendlyName: Application Friendly Name}, {ApiVersion: 2010-04-01}, {VoiceUrl: http://www.example.com/voice}, {VoiceMethod: GET}, {VoiceFallbackUrl: http://www.example.com/voice-callback}, {VoiceFallbackMethod: GET}, {StatusCallback: http://www.example.com/status-callback}, {StatusCallbackMethod: GET},- {VoiceCallerIdLookup: false},? ^+ {VoiceCallerIdLookup: False},? ^ {SmsUrl: http://www.example.com/sms}, {SmsMethod: GET}, {SmsFallbackUrl: http://www.example.com/sms-fallback}, {SmsFallbackMethod: GET}, {SmsStatusCallback: http://www.example.com/sms-status-callback},
    38. 38. ALL GROWN UP #twiliocon
    39. 39. ALL GROWN UP API #twiliocon
    40. 40. ALL GROWN UP API UNIT TESTS #twiliocon
    41. 41. ALL GROWN UP API UNIT TESTS FUNCTIONAL TESTS #twiliocon
    42. 42. ALL GROWN UP API UNIT TESTS FUNCTIONAL TESTS DIFF TESTS #twiliocon
    43. 43. ALL GROWN UP API UNIT TESTS FUNCTIONAL TESTS DIFF TESTS CANARY DEPLOY #twiliocon
    44. 44. ALL GROWN UP API UNIT TESTS FUNCTIONAL TESTS DIFF TESTS CANARY DEPLOY API #twiliocon
    45. 45. ALL GROWN UP API UNIT TESTS FUNCTIONAL TESTS DIFF TESTS CANARY DEPLOY API #twiliocon
    46. 46. Flask-RESTful• A simple REST resource framework for python Flask applications• Simplifies argument parsing, generating output, & defining resources• ORM / Library independent. It’s only dependency is Flask itself. http://www.twilio.com/open-source #twiliocon
    47. 47. HI. I’M ADAMI’m MicroTransactions-Team Engineering Lead at twilio #twiliocon
    48. 48. REWIND TO 2009
    49. 49. NERDY TELECOMSTARTUP SEEKS RELIABLE HA #twiliocon
    50. 50. MICRO-TRANSACTIONS @TWILIO #twiliocon
    51. 51. MICRO-TRANSACTIONS @TWILIO #twiliocon
    52. 52. MICRO-TRANSACTIONS @TWILIO #twiliocon
    53. 53. TRANSACTIONS @TWILIO V1 BALANCE AGGREGAT UPDATER ION DEQUEUER DEQUEUERTX QUEUE DEQUEUER AGG DEQUEUER TX MYSQL AGG DEQUEUER DEQUEUER #twiliocon
    54. 54. TRANSACTIONS @TWILIO V1 BALANCE AGGREGAT UPDATER ION DEQUEUER DEQUEUERTX QUEUE DEQUEUER AGG DEQUEUER TX MYSQL AGG DEQUEUER DEQUEUER #twiliocon
    55. 55. TRANSACTIONS @TWILIO V1 BALANCE AGGREGAT UPDATER ION DEQUEUER DEQUEUERTX QUEUE DEQUEUER AGG DEQUEUER TX MYSQL AGG DEQUEUER DEQUEUER #twiliocon
    56. 56. TX- TX- MYSQL INSERT only log dataAutomatically increments MY SQLbalance and counters REDIS MYSQL REDIS POST-FLIGHT #twiliocon
    57. 57. ONLINE SYSTEM UPGRADES #twiliocon
    58. 58. ONLINE SYSTEM UPGRADES• Double book-keeping with Shadow Mode #twiliocon
    59. 59. ONLINE SYSTEM UPGRADES• Double book-keeping with Shadow Mode• Rollback support with Account Flags & Feature Flags #twiliocon
    60. 60. ONLINE SYSTEM UPGRADES• Double book-keeping with Shadow Mode• Rollback support with Account Flags & Feature Flags• Gradual rollout #twiliocon
    61. 61. 1. SHADOW MODE TX TX-MASTER-QUEUE DEQUEUER NGB LEGACY QUEUE QUEUE NGB- LEGACY- DEQUEUER DEQUEUER #twiliocon
    62. 62. 1. SHADOW MODE TX TX-MASTER-QUEUE DEQUEUER ATOMIC faucet controls NGB LEGACY QUEUE QUEUE NGB- LEGACY- DEQUEUER DEQUEUER #twiliocon
    63. 63. Sid Account Sid Old System New System SV... AC... 29.5500 29.5500 SV... AC... 144.8200 144.8200 COMPARING SV... SV... AC... AC... 35.6700 30.4200 35.6700 30.4200 FAUCETS: VS.OLD SYSTEM SV... SV... AC... AC... 106.9100 109.7900 106.9100 109.7900NEW SYSTEM SV... SV... AC... AC... 5.5900 10.8400 5.5900 10.8400 SV... AC... 74.0250 73.9450 SV... AC... 29.00 49.00 SV... AC... 13.9600 13.1200 SV... AC... 71.3800 91.2400 SV... AC... 671.0650 646.5050 SV... AC... 71.2600 71.2600 SV... AC... 44.5000 44.5000
    64. 64. Sid Account Sid Old System New System SV... AC... 29.5500 29.5500 SV... AC... 144.8200 144.8200 COMPARING SV... SV... AC... AC... 35.6700 30.4200 35.6700 30.4200 FAUCETS: VS.OLD SYSTEM SV... SV... AC... AC... 106.9100 109.7900 106.9100 109.7900NEW SYSTEM SV... SV... AC... AC... 5.5900 10.8400 5.5900 10.8400 SV... AC... 74.0250 73.9450 SV... AC... 29.00 49.00 SV... AC... 13.9600 13.1200 SV... AC... 71.3800 91.2400 SV... AC... 671.0650 646.5050 SV... AC... 71.2600 71.2600 SV... AC... 44.5000 44.5000
    65. 65. 2. ROLLBACK SUPPORT Toggle Account Features #twiliocon
    66. 66. 2. ROLLBACK SUPPORT Toggle Cluster Features #twiliocon
    67. 67. 3. GRADUAL ROLLOUT (PRACTICEe Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 # of Errors Accounts on Next Gen Billing
    68. 68. BETTER TOOLS Problem Solution• Redis killed by Linux OOM Killer • Rollback account flags• Inaccurate balances on NGB • Fix bug, deploy, compare• Credit card purchases failing • Rollback account flags• Failing to write log entries for NGB • Rollback feature flags #twiliocon
    69. 69. THANK YOUCOME ASK US QUESTIONS!

    ×