Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lessons Learned in Test Automation From Zombieland
Similar to Lessons Learned in Test Automation From Zombieland (20)
Recently uploaded
Recently uploaded (20)
Lessons Learned in Test Automation From Zombieland
- 1. DRAFT Lessons learned on end to end testing from Matt Barbour Sr. Director Engineering Comcast Silicon Valley matt_barbour@cable.comcast.com STAREAST 2016 – May 4th, 2016
- 2. Confidential and Proprietary What can we LEARN from a ZOMBIE MOVIE? • Characters in zombieland live by a set of rules in order to survive the chaos • Many of these apply in non- zombie situations too! Specifically: End-to-end testing in real world chaos. FOL 2014 Process Improvement Project – IDD Process1 Confidential and Proprietary
- 3. Confidential and Proprietary End To End Testing as compared to distributed computing • Distributed systems expect failure because the real-world is chaotic • Strive to be fault tolerant by optimizing for desired characteristics of the end application according to the CAP Theorum 2 • End to end automated tests should have similar expectations about the environment they are operating in Automated Tests Performant Meaningful Repeatable Accurate PARM THEORY Tests must balance being: Performant, Accurate, Repeatable, and Meaningful
- 4. Confidential and Proprietary KNOW YOUR WAY OUT BE FORWARD LOOKING AND OWN THE PROBLEM!! • Aim for excellence – don’t tolerate mediocrity • “that’s the way we have always done it”™ • Low pass rate • Tests cant finish quickly enough to give feedback on a checkin • Don’t know what a test is doing or why it exists • Inconsistent results • Aim for being in the center of the PARM matrix 3 Automa ted Tests P M R A
- 5. Confidential and Proprietary BUDDY SYSTEM LEARN AND APPLY DEVELOPMENT BEST PRACTICES Developing E2E tests should be no different than developing an application GTAC 20154 • Design Patterns • Code Review • Unit Tests • Static Code Analysis Automat ed Tests P M R A
- 6. Confidential and Proprietary RULE #31 CHECK THE BACK SEAT MAKE SURE YOURE NOT BRINGING ALONG UNWANTED DEPENDENCIES • Use a dependency management tool! • Explicitly declare all dependencies • Beware of depending on uber jars in a project… 5 Automat ed Tests P M R A
- 7. Confidential and Proprietary A.B.T. ALWAYS BE TESTING Trim the fat Look for opportunities to obtain useful test metrics outside of a traditional test Even idle devices can tell you something! 6 Automat ed Tests P M R A
- 8. Confidential and Proprietary MAKE SURE YOUR TEST TOOLS SCALE • Use a test framework which allow you to scale in both directions (more devices in parallel) (more tests in parallel) • Use open source tools where possible • WebDriver isnt just for web and mobile!! 7 Automat ed Tests P M R A
- 9. Confidential and Proprietary TRYING TO DO TOO MUCH IN A TEST IS A RECIPE FOR FAILURE! • Tests should focus on one thing, and one thing only. • BDD is helpful here • BDD scenarios should favor DAMP over DRY • DAMP = Descriptive and Meaningful Phrases • DRY = Don’t repeat yourself 8 Automated Tests P M R A
- 10. Confidential and Proprietary Choose your test methodologies wisely • You can lose the war against unreliable, slow tests before you even start by your choice of test methodology • Known offenders – Image Comparison – Xpath – Record/Playback – Not controlling your data – The list goes on and on 9 RULE #7 TRAVEL LIGHT Automated Tests P M R A
- 11. Confidential and Proprietary RULE #5 NO ATTACHMENTS CONTROL ALL THE DATA IN YOUR SYSTEM • If you have a data dependent test where you are not controlling the data you probably have an unstable test, or worse… • Mocking production data is time consuming… so is triaging false failures • Tests must be able to start from a known state 1 0 Automated Tests P M R A
- 12. Confidential and Proprietary WORK AROUND OR ELIMINATE ARCHITECTURAL CONGESTION 11 •Shared resources are the enemy of test parallelism. •Your system’s architecture can get in the way here… don’t assume the application is infallible. Either work with it or modify it •Cloud deployments can mitigate architectural problems •VMs are cheap and easy to spin up and tear down Automated Tests P M R A
- 13. Confidential and Proprietary Avoid Strangers This one is just plain wrong • The only way we can survive the slow unreliable test zombie apocalypse is by sharing tools and techniques. 12 All the test tools used by Comcast Silicon Valley are open-sourced: https://github.com/Comcast • Cucumber testing in parallel • Zucchini • Device Management • DAWG • Selenium Driver Management • Magic Wand • Theres a great deal of knowledge in the community out there, use it!!
Editor's Notes
- Chaos of zombie world vs chaos of real world Describe the chaos we are dealing with Service oriented architecture One application which runs server side but communicates with hardware in customers homes as well as 5-10 webservices managed by both internal and external groups and released on whatever intervals make sense for that team 15+ different models of hardware and growing The application itself running in a HA cluster across multiple data centers Cable TV – no tolerance for downtime
- CAP Theorem (Consistent, Available, Partition Tolerant - pick 2) Lots of analogies between distributed computing (DevOps) and automated testing PARM theory (Performant, Accurate, Repeatable/Precise, Meaningful - must have all 4) E2E testing should really be the smallest part of your “test pyramid”
- First step is to admit you have a problem Don’t tolerate mediocrity You want to have an engineering team which is empowered to solve problems and consider alternative solutions It’s a fine line – new and shiny doesn’t always mean good for your product Optimze for being in the middle of the PARM matrix
- - For automated tests to be really useful they need to be MORE reliable than the code it is testing - Analogy to semiconductor measurement Test Development and application development should be considered one and the same Treat test authoring as a first class citizen and apply industry standards / best practices
- This is in line with treating test development as a first class citizen, but is important enough that it deserves its own slide Use a dependency management tool – one exists for every language Lock down your dependencies Avoid transitive dependencies Dependency ranges Maven example Parent POM for dep mgmt tools which allow inheritance Uber Jars – leading cause of hair loss among test automation engineers
- Trim the fat by : removing hardcoded sleeps and unnecessary synchronization Bunch tests which have similar setup steps Pipeline and parallelize where it makes sense (more on that next) Look for other ways to get test coverage without adding additional tests Memory consumption, concurrency, etc Compare to idle devices
- Testing in real world conditions is going to be slower and more problematic than you would like ideally Combat this by scaling horizontally and vertically and avoiding proprietary frameworks where you can Scaling horizontal will allow you to get coverage of all your device compatibility tests - make sure you account for tests which cant run on certain device types Run same test on multiple instances of the same make/model/browser/os/etc - will help you suss out inconsitent tests Just scaling horizontally will still mean you are limited by slow tests. One way to combat this is to scale vertically. It can also be combined with horizontal scaling. For ex: Run your slow tests in parallel on one set of devices and simultaneously run your other tests in parallel on a separate set of devices Open source tools (especially the popular ones) have a community around them and are easy to hire people for Even if your application isnt web or mobile, you can make existing tools work. Things like WebDriver can be adapted to run against other device types
- Big monolithic tests are less desireable than narrowly focused tests. BDD is helpful in this regard Be careful to avoid antipatterns Too generic Too specific so that they lose meaning Trying to do DRY vs DAMP One case where good development practices don’t apply
- How you implement your tests can have a drastic impact on P, A, & R When you find anti-patterns in your test methodologies, the first step is to know WHY the test is authored that way When you get to root cause you may find that there is a better way to do it If theres no better way to do it you have the knowledge to minimize the damage or just kill the test Image Comparison example
- Failure to control your data is quite often the biggest contributor to unstable tests If you have a data dependent test you quite likely have an unstable test or worse, youre letting defects get through to production Why? Mocking prod code is HARD and SLOW… so is triaging false failures or HOPE is NOT a strategy Sports example: Live events are unpredictable Rain delays, overtime, cancellations, etc Live events generally happen in off hours Don’t boil the ocean though – look at what your tests require and mock out only that
- Identify architectural congestion in your application or tests Shared resources are the enemy of test parallelism Singletons are a big problem here (but not the only one) First step is to define whether the arch congestion is the correct design for your users. If its not, congrats, you’ve found a defect. If it is, then you have to figure out how to work around the congestion Sports example again Data layer serves multiple clients. If your tests are controlling the data like they should then you will start clobbering yourself once you strat trying to scale our horizontally Cloud deployments can help here – the brute force approach is to throw up identical clusters. In the absence of that (either by necessity or choice) you are left with trying to synchronize across tests