Quality Leadership, Testing, and Governance Tactics that Make or Break Your Progressive Delivery System - Jeff Sing
•Download as PPTX, PDF•
0 likes•51 views
More and more companies are using Progressive Delivery to get all types of changes – new features, configuration changes, bug fixes, and experiments – into production in a safer, faster, and most importantly, a sustainable way. Software companies that shift to Progressive Delivery benefit from low risk releases, faster time to market, higher quality, and in general happier teams. Sounds great right? But what happens when your system isn’t implemented correctly, or worse, tested properly? This talk takes you on a journey of why teams use Progressive Delivery, and the path from basic to advanced feature flag usage. Make sure you build the right product, and build the product right! How to decide what testing strategies should be implemented (how to setup your unit test, end to end automation, etc) What tactics are most effective for keeping your implementation healthy and effective (feature flag governance!).
Recommended
More Related Content
Similar to Quality Leadership, Testing, and Governance Tactics that Make or Break Your Progressive Delivery System - Jeff Sing
Similar to Quality Leadership, Testing, and Governance Tactics that Make or Break Your Progressive Delivery System - Jeff Sing (20)
More from QA or the Highway
More from QA or the Highway (20)
Recently uploaded
Recently uploaded (20)
Quality Leadership, Testing, and Governance Tactics that Make or Break Your Progressive Delivery System - Jeff Sing
- 1. Jeff Sing (Iterable) Quality Leadership,Testing, and Governance Tactics that Make or Break Your Progressive Delivery System
- 2. Jeff Sing ● Sr Eng Manager - Quality & Operations @Iterable ● 15 Years in QA in a variety of industries ● Progressive Delivery Advocate jeff.sing@iterable.com https://www.linkedin.com/in/jeffsing/
- 3. Quality Leadership,Testing, and Governance Tactics that Make or Break Your Progressive Delivery System
- 4. Progressive Delivery “… a new new basket of skill and technologies concerned with modern software development, testing and deployment. I am thinking of Canarying, Feature Flags, A/B testing at scale.” - James Monk, RedMonk
- 5. Examples of Progressive Delivery
- 6. Continuous Delivery helps your team move faster by making each change small and manageable. Progressive Delivery helps your team move faster by reducing the risk of each change by controlling the audience exposed.
- 7. Data Driven Software Development Did I build the right Product? Did I build the Product right?
- 8. What happens when you don’t make good data driven decisions?
- 9. if feature_flag is ‘on’: do_something() else: do_something_else() So… Feature Flags?
- 10. enabled: False header_color: #F829EA header_sticky: True header_height: 20 header_message: ‘We love our users <3’ mobile_app_header Feature Rollout Disable header in production, no code change.
- 11. enabled: True header_color: #F829EA header_sticky: True header_height: 20 header_message: ‘We love our users <3’ mobile_app_header Feature Rollout Slowly rollout new header to only certain customers.
- 12. Software Development Life Cycle Build & Test ● Features don’t need to be completed to get merged ● No wasted time untangling messy merge conflicts Deploy ● Ship faster by decoupling deploys from releases to shorten dev cycle ● Deploy frequently and safety Release ● Empower teams to manage their own releases ● Quickly rollback production incidents (way faster MTTR)
- 13. Feature Flag Solutions Companies home build their own progressive delivery solution. Engineering usually builds solution and iterates on functionality as the system matures (Uber Piranha). Vendor bought solution such as Optimizely or LaunchDarkly. These vendors charge by seats and unique visits. Typically they provide a SDK to integrate with your system and custom dashboards. Homebuilt Solutions Vendor Solutions
- 14. Quality Leadership,Testing, and Governance Tactics that Make or Break Your Progressive Delivery System
- 15. Governance Tactics Mission Control Damage Control Testing Why governance and testing? Feature Flags
- 16. What could go wrong? What does this flag do? Who turned off my flag? Who touched my flag?
- 19. Feature Flag Governance Flag Info General Info - What does Flag do when it’s On/Off - What env is it deployed? Visibility - Who needs to know if this is On/Off? - Who is monitoring metrics? Risk Level How dangerous is this to roll out? What happens if something catastrophic happens to this flag? Access Level Who is allowed to make changes to this feature flag? How do we track this? Exit Strategy When is this feature flag no longer useful? When it is no longer useful, who is removing it?
- 20. Example: Launch Darkly Governance
- 21. Example: Launch Darkly Governance
- 22. Example: Governance Flag Info 1. Feature On Behavior 2. Feature Off Behavior 3. Who actually owns this Feature Flag
- 23. Example: Governance Risk Level 1. Can I roll back? 2. What happens if this flag fails?
- 24. Example: Governance Exit Strategy 1. When should I remove this? 2. When can I remove this? 3. Who should be removing this?
- 25. Feature Flag Testing Strategy
- 26. Testing Strategy Test Type High Level Strategy Manual Test Only most critical variations. E2E Test Focus only on important variations and test application still works if all features are on/off Integration Test Mock and stub to control feature states. Focus on individual code paths to ensure proper integration and business logic Unit Test Each code path should have its own set of independent unit tests. Ensure high code coverage, just as if you didn’t have any feature flags in your codebase.
- 27. General E2E Automation Test Strategy Test Runner + INDETERMINISM =
- 28. General E2E Automation Test Strategy Remove INDETERMINISM 1. Special Test User 2. Test Query Parameter to force a particular version 3. API Endpoint to set variation HELENA PATTERSON “Despite being red, Mars is a cold place. It’s full of iron oxide dust, which gives the planet its reddish cast”
- 29. Special Test User (Launch Darkly)
- 30. General E2E Automation Test Strategy (Query Parameter)
- 31. General E2E Automation Test Strategy (Automation Controls Path w/API calls)
- 33. Feature Flag Types Feature Rollout For deploying a new feature that will be permanent. Experimentation Perform A/B/n tests against our deployed features. Short Term
- 34. enabled: True header_color: #F829EA header_sticky: True header_height: 20 header_message: ‘We love our users <3’ mobile_app_header Experimentation Does header “We love our users <3” drive more conversions vs “Hi, welcome?” enabled: True header_color: #F829EA header_sticky: True header_height: 20 header_message: ‘Hi, welcome?’ mobile_app_header 50% 50% Customer A Customer B
- 35. Feature Flag Types Feature Rollout For deploying a new feature that will be permanent. Experimentation Perform A/B/n tests against our deployed features. Permission Change product experience that certain users receive Short Term Long Term
- 36. enabled: True user_tier: premium header_height: 20 premium_view Permissions Is user_tier type premium? If so, allow them to access the premium content.
- 37. Feature Flag Types Feature Rollout For deploying a new feature that will be permanent. Experimentation Perform A/B/n tests against our deployed features. Permission Change product experience that certain users receive Operation Controls operational aspects of your system; allows control of certain operational features in your production environment Short Term Long Term
- 38. enabled: True checkout_version: 3 quick_checkout: True header_height: 20 checkout_flow Operations Change the checkout_flow dynamically without having to deploy.
- 39. enabled: True checkout_version: 3 quick_checkout: True header_height: 20 checkout_flow Operations Change the checkout_flow dynamically without having to deploy. 4
- 40. Feature Flag Types Feature Rollout For deploying a new feature that will be permanent. Experimentation Perform A/B/n tests against our deployed features. Permission Change product experience that certain users receive Operation Controls operational aspects of your system; allows control of certain operational features in your production environment Circuit Breaker Turn on/off certain features or conditions in your production environment. Short Term Long Term
- 41. enabled: True fresh_boost: 1.25 profile_boost: 2.10 local_boost: 0.75 search_ranking_algorithm Circuit Breaker Turn on/off search_ranking_alogrithm.
- 42. enabled: False fresh_boost: 1.25 profile_boost: 2.10 local_boost: 0.75 search_ranking_algorithm Circuit Breaker Turn on/off search_ranking_alogrithm.
- 43. Governance Tactics Mission Control Damage Control Testing Same but Very Different... Feature Flags
- 45. Feature Flag Governance Flag Info Access Level Risk Level Exit Strategy Experiment Hypothesis driven Product/Eng Low Experiment End Permission Business Tier Provisioning Info CS/AE/GTM Restrictive Access High Rare Operational Eng Runbook Eng High/Medium If Feature Deprecates Circuit Breaker Eng Runbook Eng Low If Feature Deprecates
- 46. Example: Governance (Flag Info)
- 51. Permission: E2E Test Strategy Permissions 1. Checking that “provisioning” is correct 2. Automation should check all paths 3. High sensitivity in regression that we don’t remove a customers features
- 52. Operational: E2E Test Strategy Operational 1. Validate “default” condition 2. Strategy to validate operational values may vary... 3. Doesn’t always make sense to automate all paths... enabled: True checkout_version: 3 quick_checkout: True header_height: 20 checkout_flow
- 53. Circuit Breaker: E2E Test Strategy Circuit Breaker 1. Regression Automation: Product still works if flag is ON, and still works if OFF 2. 3rd Party Integrations - usually manually tested
- 54. TAKEAWAYS POWERING MODERN DEPLOYMENT Feature Flags used correctly are an incredibly powerful engineering tool. CONTROLLED BLAST RADIUS Quality champions rejoice! Plus our MTTR should be factors better! 01 03 02 04 05 06 THOUGHTFUL IMPLEMENTATION Ensure your organization has a strong process for releasing/managing/removing your feature flag, or else technical debt can punish you. FEATURE FLAG DIFFERENTIATION Not all Feature Flags are alike, why would you treat them so? DETERMINISTIC TESTING Ensure you are properly testing your feature flags and that you have good feature coverage for each path. QUALITY LEADERSHIP Every organization needs leadership around governance and testing.
- 55. Interested in learning more about Feature Flags? Or trying them? VENDORS ■ Optimizely (Free Rollouts): https://www.optimizely.com/rollouts/ ■ LaunchDarkly: https://launchdarkly.com/ ■ Split: https://split.io/ FURTHER READING ■ Feature Toggles: https://martinfowler.com/articles/feature-toggles.html ■ Getting Started WIth Feature Flags: https://dzone.com/refcardz/getting-started-with-feature-flags ■ Unleash (open source Feature Flag): https://github.com/Unleash/unleash RESOURCES
- 56. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik. Please keep this slide for attribution. jeff.sing@iterable.com https://www.linkedin.com/in/jeffsing / Stay in Touch
Editor's Notes
- Currently am leading the Quality Program and Engineering Operations at Iterable - a cross channel marketing platform that enables e-commerce customers to control their customer experience and understand and measure every interaction across their entire customer journey. We serve customers like Doordash, MadisonReed, zillow, and asics. 15 Years Web Security/Fraud Prevention Medical Devices Marketing Tech (most recently) Everything from small medium to humongous
- Continuous Delivery is a set of practices that ensure your code is always in a deployable state. You accomplish this by increasing the frequency at which code is committed, built, tested, and deployed—steps that in the past only occurred at the end of a project when it was ‘code complete’. Because your team is delivering frequently, every change is smaller. Changes are more isolated and less risky. Problems are identified more readily, and bugs are fixed more easily because they are discovered immediately. Progressive Delivery is the technique of delivering changes first to small, low-risk audiences, and then expanding to larger and riskier audiences, validating the results as you go. We can accomplish this by deploying first to a certain environment or certain user.
- What does that mean? Well in QA, a lot of our orchestration is around to answer this question - Did I build the product right? Automation test, test planning, feature testing matrixes, etc is to answer did we release correctly? CI helps us with this because it makes it easier for us to do this, smaller changes, smaller releases... On the other hand, all of this cannot help us determine if we built the right product. Using Progressive Delivery we can not only control our feature rollouts, but we can experiment on who and what our customers experience is, and determine if this is the right feature to release.
- So let’s talk orchestration of Progressive Delivery: The primary method in the software development industry to orchestrate this are Feature Flags. Essentially you are wrapping your features behind a flag and orchestrating it so that when you deploy to production, it isn’t visible until you are ready to flip the switch. Now obviously their are more complexities to this, but this is the core concept. You can build on top to have a method to deploy in a sliding scale (so 20% of your audience, 40%, 60% etc) or pass in certain values or configurations that only affect features behind that flag.
- So let’s look at an example, here is what I would call a feature rollout which i just mentioned. Here my feature flag (mobile_app_header) is disabled. When my users come to my app, nothing is in the header. However when I’m ready to flip it on, it will appear in production.
- Build & Test - no stale feature branches, smaller code reviews Deploy - we actually deploy constantly at Iterable. This allows us to test certain features in production. Release - SE can release certain features, teams can control betas. QA perspective MTTR has improved
- For todays talk - I’ll talk around the 2 major ways companies are implementing FF - either as as homebuilt solution or vendors (in this case Launch Darkly). Each have their pros and cons but this talk easily applies to both.
- So now that I’ve talked a bit about progressive delivery and the power of feature flags, let’s get to the meat of this talk… testing and governance tactics
- What does this flag do: You have a feature flag out forever, the developer leaves, and now you have no idea what it does, I guess you can never turn it off safely… Who turned off my flag: A new engineer comes, misreads some doc, and turns off a flag not realzing it causes a catastrophic outage Who touched my flag: This one is more insidious, we didn’t remove a feature flag, and the other dev teams didn’t realize it was in your code base. Someone builds a feature flag upstream and turns it off, suddenly your flag no longer behaves the way you think it does. Who touched your flag? Why?
- Knights capital - high frequency trading algorith - KNIGHT was largets trader in US equities market with a mrket share of 17.3% on NYSE and 16.9 on NASDAQ. Developers reusing a flag that was no longer thought to be in use. Dead code coming back to life. One server started making as many trades as fast as possible without limit Rolled back within 45 minutes. ½ billion dollars
- Flag info x4 Access Level x2 Risk Level x2 Exit Strategy x2
- Unit testing - your developers should be writing these with the mentality that you didn’t have any feature flags in your codebase Integration tests - this is important to have orchestration to mock or stub your feature states. You need to make sure the code paths are tested The top part of our pyramids - automation gets expensive, and you may have many feature paths, so for E2E you really want to make sure you are testing your critical variations - also don’t forget to test OFF state, as your flags may be in for abit and at any point you could be turning off your feature Manual Testing - is super important. You should be checking your critical variations. What is important though is you need to expose your manual testers tooling to be able to control the flags to let them get into the right variation. … also make sure your tooling lets you know what variation you are in.
- Experimentation - allows me to drive traffic to different features and measure conversions
- Permission - change product experience - bucket your premium users into certain experiences
- Operation - controls operational aspects - some examples, you can hook a feature variable to your features, so if the code base is activated it dynamically will pull that value in, meaning you can do something like point to where the source of an image is dynamically change it by modifying the resource location, or maybe a devops config change, where you can modify how many instances are being spun up, etc.
- Since permission flags are acting as gates for a tested feature, testing for permission feature flags revolve around ensuring the right customer is correctly bucketed rather than if the feature is working. This is usually handled by the business systems team as we add users into the permission flag. In general they perform this by picking a customer and emulating their experience on our product and validating that they are getting the correct set of features purchased.
- Since permission flags are acting as gates for a tested feature, testing for permission feature flags revolve around ensuring the right customer is correctly bucketed rather than if the feature is working. This is usually handled by the business systems team as we add users into the permission flag. In general they perform this by picking a customer and emulating their experience on our product and validating that they are getting the correct set of features purchased.