Strategies in continuous delivery


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Today I’m going to tell you some of the strategies we use that allow us to deploy 10 times a day
  • Management role is to help developer do its work
  • Management role is to help developer do its work
  • One of the key components to successful CD
  • Full load on a single serverOverride size limitation by setting a cookie on the client
  • Link to purchase on the editor was causing drop in conversion because users went there too soon without intent
  • Link to purchase on the editor was causing drop in conversion because users went there too soon without intent
  • Strategies in continuous delivery

    1. 1. Strategies In Continuous DeliveryLearn the “magic” behind the scenesAviran MordoServer Group Manager @ Wix@aviranm
    2. 2. About Wix21:34
    3. 3. Wix in Numbers• Over 35,000,000 users– Adding over 1,000,000 new users each month• Static storage is over 150TB of data– Adding over 1TB of files every day• 3 Data centers + 2 Clouds (Google AE, Amazon)– Around 300 servers• Over 40,000,000 Server API calls per day• ~400 people work at wix– Over 100 people in R&D21:34
    4. 4. Is Doing MultipleDeployments A Day TakesGuts ?21:34
    5. 5. 21:34From Jan’ 2013 – Jun’ 2013• 1500• 470• 200
    6. 6. 21:34From Jan’ 2013 – Jun’2013• 1500 Deployments• 470 A/B Tests• 200 Feature Toggles
    7. 7. Continuous Delivery at Wix• Abandon “VERSION” paradigm – move featurecentric life• Make small and frequent release as soon aspossible• Automate everything – TDD/CI/CD• Measure Everything–A/B test every new feature–Monitor real KPIs (business, not CPU)21:34
    8. 8. Continuous Delivery at Wix• Empower the developer–The developer is responsible from product idea to 1Mactive users–Remove every obstacle in the developer’s path–Big cultural change from waterfall – affects the wholecompany21:34
    9. 9. Test Driven Development• No new code is pushed to Git without being fully tested• Before fixing a bug first write a test to reproduce the bug• Cover legacy (untested) systems with Integration tests21:34
    10. 10. Test Driven Development• What people think is the impact on development–TDD slows down development–With TDD we write more code (product + test code).• Current Test Count (U-Tests + IT-Tests) – over10,00021:34
    11. 11. Test Driven Development• Actual impact on development–We develop products faster–Removes fear of change–Easier to enter some one else’s project–Do we really need QA? (Yes, they code tests)–Writing a feature is 10-30% slower, 45-90% less bugs–50% faster to reach production.–Considerably faster time to fix bugs21:34
    12. 12. Guidelines for successful TDD• Tests should run on project checkout to a randomcomputer.• Tests that cannot be debugged on a developermachine will never consistently run for any periodof time• Tests should run fast• Tests have to be readable – They are the project’sspecs21:34
    13. 13. Feature Toggles21:34
    14. 14. Feature Toggles• Everyone develops on the Trunk• Every piece of code can get to production atanytime21:34
    15. 15. Feature Toggle to the rescue• Unused new code can go to production – no harm done• Operational new code goes with a guard – use new or old code byfeature toggle21:34
    16. 16. Usage exampleSimple “if” statement in your code21:34
    17. 17. Feature Toggle Strategies (gradual expose users)• Company employees• Specific users or group of users• Percentage of traffic• By GEO• By Language• By user-agent• User Profile based• By context (site id or some kind of hash on site id)21:34
    18. 18. Feature Toggle Override• By specific server– Used to test system load– New database flows/migration– Refactoring that may affect performance and memory usage• By Url parameter– Enable internal testing– Product acceptance– Faking GEO• By FT cookie value– Testing– When working with API on a single page application21:34
    19. 19. Feature Toggles Management21:34
    20. 20. A/B Tests21:34
    21. 21. A/B Test• Every new feature is A/B tested• We open the new feature to a % of users– Define KPIs to check if the new feature isbetter or worse– If it is better, we keep it– If worse, we check why and improve– If we find flaws, the impact is just for % of ourusers (kind of Feature Toggle)21:34
    22. 22. An interesting site effect on product• How many times did you have the conversion“what is better”?– Put the menu on top / on the side• Well, how about building both and A/B Testing?21:34
    23. 23. Marking users with toss value in a cookie• Anonymous user– Toss is randomly determined– Can not guarantee persistent experience if changing browser• Registered User– Toss is determined by the user ID– Guarantee toss persistency across browsers– Allows setting additional tossing criteria (for example new users only)– Only use this for sections that a user has to be authenticated21:34
    24. 24. • Do not mix anonymous and registered tests• AB test parentage of users with optional filters– New Users Only (Registered users only)– By language– By GEO– By Browser– user-agent– OS– Any other criteria you have on your users21:34
    25. 25. A/B Test Features• A/B Test Override– Allows setting a value of a test for validation– Helps support experience what users experiencing• Override methods– Via URL parameter– Via cookie• Start/Stop Test• Pause tests• Bots always get “A”21:34
    26. 26. Manage your tests21:34
    27. 27. Refactoring21:34
    28. 28. Refactoring• Before refactoring make sure everything is covered withtests– Legacy code usually covered by IT tests• Refactor from inside out– Small iterations with tests– Refactor small methods - make sure the tests don’t break– Deploy often• Re-write from the outside in– Write from scratch– Code duplication sometimes needed (temporary)– Protected by Feature Toggle21:34
    29. 29. Code branch21:34New Code Old CodeFTOpenedYes No
    30. 30. DB Schema Changes WithoutDowntime• Adding columns– Use another table link by primary key– Use blob field for schema flexibility• Removing fields– Stop using. Do not do any DB schemachanges21:34
    31. 31. New DB schema• Plan a lazy migration path controlled by feature toggle1. Write to old / Read from old2. Write to both / Read from old3. Write to both / Read from new, fallback to old4. Write to new / Read from new, fallback to old5. Eagerly migrate data in the background6. Write to new / Read from new21:34
    32. 32. Gradual Deployment21:34• Assume two components• We shutdown one and install on it thenew version. It is not active yet• Do self test• Activate the new server it is passes self test• Continue deploying the other servers,a few at a time, checking each one withself testA 1.1 B 1.1A 1.1B 1.2A 1.1A 1.1 B 1.1B 1.1A 1.1A 1.1B 1.1B 1.2A 1.1B 1.2A 1.1A 1.1B 1.1B 1.2A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.2
    33. 33. Self Test / Post Deployment TestAfter each server deployment run a self test before deploying thenext server.• Checking server configuration and topology– Make sure database is accessible (DB connection string)– Is the schema the one I expect– Access required local resources (data files, other config files, templates,etc’)– Access remote resources– RPC / REST endpoints reachable and operational• Server will refuse requests unless it passes the self test• Allow a way to skip self test (and continue deployment)21:34
    34. 34. Tools - App-info21:34
    35. 35. Backward and Forward compatible• Assume two components• We release a new version of one• Now Rollback the other…21:34A 1.1B 1.2A 1.2B 1.1A 1.1A 1.1B 1.1B 1.2A 1.2A 1.1B 1.1B 1.1A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.1A 1.0A 1.2A 1.1 B 1.2B 1.1B 1.2 A 1.2A 1.2A 1.1 B 1.2B 1.1B 1.0
    36. 36. Production Visibility21:34Prepare for a rainy day
    37. 37. How does it work – CD Practices• Test driven development• Small Development Iterations• Backwards and Forwards compatible• Gradual Deployment & Self-Test• Feature Toggle• A/B Testing• Exception Classification• Production visibility21:34
    38. 38. Where are we today?• We have re-written our flash editor product as anHTML 5 editor– In just 4 months• Introduced Wix 3rd party applications (developers API)– In just 6 weeks• We are easily replacing significant parts of ourinfrastructure• And we are doing ~10 releases a day!21:34051015202530351/1/20131/4/20131/8/20131/11/20131/15/20131/18/20131/22/20131/27/20131/30/20132/3/20132/6/20132/10/20132/13/20132/18/20132/21/20132/25/20132/28/20133/4/20133/7/20133/12/20133/15/20133/19/20133/24/20134/3/20134/7/20134/10/20134/15/20134/19/20134/23/20134/28/20135/1/20135/6/20135/9/20135/13/20135/19/20135/22/20135/26/20135/29/20136/1/20136/4/20136/7/20136/11/20136/16/2013(blank)TotalTotal
    39. 39. 21:34
    40. 40. 21:34Aviran Mordo@aviranm more: The Road To Continuous Delivery: