Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DRAFT
Lessons learned on end to end testing from
Matt Barbour
Sr. Director Engineering
Comcast Silicon Valley
matt_barbour...
Confidential and Proprietary
What can we LEARN from a
ZOMBIE MOVIE?
• Characters in zombieland
live by a set of rules in o...
Confidential and Proprietary
End To End Testing as compared to distributed computing
• Distributed systems expect failure
...
Confidential and Proprietary
KNOW YOUR WAY OUT
BE FORWARD LOOKING AND
OWN THE PROBLEM!!
• Aim for excellence – don’t
toler...
Confidential and Proprietary
BUDDY SYSTEM
LEARN AND APPLY DEVELOPMENT BEST
PRACTICES
Developing E2E tests should be no dif...
Confidential and Proprietary
RULE #31
CHECK THE BACK SEAT
MAKE SURE YOURE NOT BRINGING ALONG UNWANTED DEPENDENCIES
• Use a...
Confidential and Proprietary
A.B.T.  ALWAYS BE TESTING
Trim the fat
Look for opportunities to
obtain useful test metrics
...
Confidential and Proprietary
MAKE SURE YOUR TEST TOOLS
SCALE
• Use a test framework
which allow you to scale
in both direc...
Confidential and Proprietary
TRYING TO DO TOO MUCH IN A
TEST IS A RECIPE FOR FAILURE!
• Tests should focus on one
thing, a...
Confidential and Proprietary
Choose your test methodologies
wisely
• You can lose the war against
unreliable, slow tests b...
Confidential and Proprietary
RULE #5
NO
ATTACHMENTS
CONTROL ALL THE DATA
IN YOUR SYSTEM
• If you have a data dependent tes...
Confidential and Proprietary
WORK AROUND OR ELIMINATE ARCHITECTURAL CONGESTION
11
•Shared resources are the enemy of test
...
Confidential and Proprietary
Avoid Strangers
This one is just plain wrong
• The only way we can survive the slow unreliabl...
DRAFT
THANK YOU!!
Upcoming SlideShare
Loading in …5
×

Lessons Learned in Test Automation From Zombieland

280 views

Published on

This presentation will tell you how to apply the rules for surviving the zombie apocalypse to surviving your own zombie apocalypse - end to end testing on real devices.

https://www.youtube.com/watch?v=9IShc6tfj2E

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Lessons Learned in Test Automation From Zombieland

  1. 1. DRAFT Lessons learned on end to end testing from Matt Barbour Sr. Director Engineering Comcast Silicon Valley matt_barbour@cable.comcast.com STAREAST 2016 – May 4th, 2016
  2. 2. Confidential and Proprietary What can we LEARN from a ZOMBIE MOVIE? • Characters in zombieland live by a set of rules in order to survive the chaos • Many of these apply in non- zombie situations too! Specifically: End-to-end testing in real world chaos. FOL 2014 Process Improvement Project – IDD Process1 Confidential and Proprietary
  3. 3. Confidential and Proprietary End To End Testing as compared to distributed computing • Distributed systems expect failure because the real-world is chaotic • Strive to be fault tolerant by optimizing for desired characteristics of the end application according to the CAP Theorum 2 • End to end automated tests should have similar expectations about the environment they are operating in Automated Tests Performant Meaningful Repeatable Accurate PARM THEORY Tests must balance being: Performant, Accurate, Repeatable, and Meaningful
  4. 4. Confidential and Proprietary KNOW YOUR WAY OUT BE FORWARD LOOKING AND OWN THE PROBLEM!! • Aim for excellence – don’t tolerate mediocrity • “that’s the way we have always done it”™ • Low pass rate • Tests cant finish quickly enough to give feedback on a checkin • Don’t know what a test is doing or why it exists • Inconsistent results • Aim for being in the center of the PARM matrix 3 Automa ted Tests P M R A
  5. 5. Confidential and Proprietary BUDDY SYSTEM LEARN AND APPLY DEVELOPMENT BEST PRACTICES Developing E2E tests should be no different than developing an application GTAC 20154 • Design Patterns • Code Review • Unit Tests • Static Code Analysis Automat ed Tests P M R A
  6. 6. Confidential and Proprietary RULE #31 CHECK THE BACK SEAT MAKE SURE YOURE NOT BRINGING ALONG UNWANTED DEPENDENCIES • Use a dependency management tool! • Explicitly declare all dependencies • Beware of depending on uber jars in a project… 5 Automat ed Tests P M R A
  7. 7. Confidential and Proprietary A.B.T.  ALWAYS BE TESTING Trim the fat Look for opportunities to obtain useful test metrics outside of a traditional test Even idle devices can tell you something! 6 Automat ed Tests P M R A
  8. 8. Confidential and Proprietary MAKE SURE YOUR TEST TOOLS SCALE • Use a test framework which allow you to scale in both directions (more devices in parallel) (more tests in parallel) • Use open source tools where possible • WebDriver isnt just for web and mobile!! 7 Automat ed Tests P M R A
  9. 9. Confidential and Proprietary TRYING TO DO TOO MUCH IN A TEST IS A RECIPE FOR FAILURE! • Tests should focus on one thing, and one thing only. • BDD is helpful here • BDD scenarios should favor DAMP over DRY • DAMP = Descriptive and Meaningful Phrases • DRY = Don’t repeat yourself 8 Automated Tests P M R A
  10. 10. Confidential and Proprietary Choose your test methodologies wisely • You can lose the war against unreliable, slow tests before you even start by your choice of test methodology • Known offenders – Image Comparison – Xpath – Record/Playback – Not controlling your data – The list goes on and on 9 RULE #7 TRAVEL LIGHT Automated Tests P M R A
  11. 11. Confidential and Proprietary RULE #5 NO ATTACHMENTS CONTROL ALL THE DATA IN YOUR SYSTEM • If you have a data dependent test where you are not controlling the data you probably have an unstable test, or worse… • Mocking production data is time consuming… so is triaging false failures • Tests must be able to start from a known state 1 0 Automated Tests P M R A
  12. 12. Confidential and Proprietary WORK AROUND OR ELIMINATE ARCHITECTURAL CONGESTION 11 •Shared resources are the enemy of test parallelism. •Your system’s architecture can get in the way here… don’t assume the application is infallible. Either work with it or modify it •Cloud deployments can mitigate architectural problems •VMs are cheap and easy to spin up and tear down Automated Tests P M R A
  13. 13. Confidential and Proprietary Avoid Strangers This one is just plain wrong • The only way we can survive the slow unreliable test zombie apocalypse is by sharing tools and techniques. 12 All the test tools used by Comcast Silicon Valley are open-sourced: https://github.com/Comcast • Cucumber testing in parallel • Zucchini • Device Management • DAWG • Selenium Driver Management • Magic Wand • Theres a great deal of knowledge in the community out there, use it!!
  14. 14. DRAFT THANK YOU!!

×