Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Automated Testing - Cast conference workshop august 2014


Published on

CAST 2014 New York: The Art and Science of Testing

The Association for Software Testing


Automated tools provide test professionals with the capability to make relevant observations even in the fastest-paced environments. Automated testing is also a powerful tool for improving communication between software engineers. This is important because good communication is a prerequisite for growing a great software engineering organization.

This workshop will explore the continuous testing of software systems. Special focus will be given to the situation where the engineering team is deploying code to production so frequently that it is not possible to perform deep regression testing before each release.

People who participate in this course will learn pragmatic automated testing strategies like:

* Data analysis on the command line with find, grep and wc.
* Network analysis with Chrome Inspector, Charles and netcat.
* Using code churn to predict hotspots where bugs may occur.
* Putting stack traces in context with automated SCM blame emails.
* Using statsd to instrument a whole application.
* Testing in production.
* Monitoring-as-testing.

Technical level: participants should have some familiarity with the command line and with editing code using a text editor or IDE. Familiarity with Git, SVN or another version control system is helpful but not required. Likewise some knowledge of Web servers is helpful but not required. It is desirable for participants to bring laptops.


From 2010 to 2012 Noah was a Test Architect at Etsy. He helped build Etsy's continuous integration system, and has helped countless other engineers develop successful automated testing strategies.These days Noah is an independent consultant in New York. He is passionate about helping engineers understand and use automated tools as they work to scale their applications more effectively.

Published in: Software
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • Substandard data quality is harming organizations to an immense extent. Inaccurate and erroneous data impacts organizations with budget increments, customer displeasure, and organizational uncertainty. This is where automation testing can resolve the vulnerability by verifying the data.
    Are you sure you want to  Yes  No
    Your message goes here

Continuous Automated Testing - Cast conference workshop august 2014

  1. 1. Noah Sussman and Joanna Burgess! ! CAST NYC, August 11, 2014 Continuous Automated Testing: A Communication System That Scales! Jared Tarbell, Substrate
  2. 2. Continuous AutomatedTesting (C.A.T.) Schedule ✤ 9:00-10:30: CAT! ✤ 10:30-10:50: Break ! ✤ 10:50-12:30: CAT! ✤ 12:30-1:30: Lunch! ✤ 1:30-3:00: CAT! ✤ 3:00-3:30: Snack! ✤ 3:30-5:00: CAT
  3. 3. Load it Up ! ✤ Jenkins ! ✤ Golly
  4. 4. “When I use a word it means just what I choose it to mean-neither more nor less.” What does QA mean to you?
  5. 5. Safetyisnotaboutthebreakageor lackofqualityofsinglecomponents. Managingqualityisaboutsingle components,aboutseeinghowthey meetparticularspecifications. ! —SidneyDekker Guy Sie on Flickr
  6. 6. In other words…when are we, and when aren’t we speaking the same language? ! More to the point…how much does it matter? Rolling the Dice
  7. 7. ContestedTerminology ✤ Quality Assurance! ✤ Testing! ✤ Manual Testing ! ✤ Bugs! ✤ Accountability! ✤ QA Environment !
  8. 8. 4REALZ:“When I use a word it means just what I choose it to mean” (ymmv) ✤ Like Humpty Dumpty, I am using contested terminology to support my own viewpoints! ✤ When I use a word, it means what I intend it to mean! ✤ If you are unsure what I mean, raise your hand!!!
  9. 9. ETTO ✤ Efficiency-to-thoroughness-trade-offs (ETTO)! ✤ We do necessarily not pick the best option, we pick the option that best satisfies our immediate needs namely:! ✤ efficiency! ✤ thoroughness
  10. 10. AutomatedTests Are… ✤ Part of the software design process ✤ Runnable documentation ✤ NOT useful for debugging releases ✤ NOT useful for catching bugs in code
  11. 11. Tractable Systems A system is tractable if the principles of the functioning are known and have simple descriptions with few details. Tractable systems do not change while being described.! !
  12. 12. Intractable Systems Intractable systems are complex systems. A defining feature of intractable systems is that they are unsafe and defy documentation.
  13. 13. DefiningTerminology ✤ Abstraction! ✤ Orders of Magnitude Growth ! ✤ Cellular Automaton! ✤ Finite Automaton! ✤ Halting Problem ! ✤ CAP Theorem ! ✤ Acyclic Directed Graphs ! ✤ Path reduction
  14. 14. Core Concepts ✤ Ironies of automation ! ✤ Hawthorne Effects! ✤ Goodhart’s Law! ✤ Diseconomies of scale
  15. 15. Such dev.Very ops.Wow.
  16. 16. Autonomy Mastery & Purpose
  17. 17. devops: where is the QA? ✤ STOP ASKING ME where QA fits into devops.! ✤ ROFLMAO if QA and testing aren’t ALREADY part of dev for you!! ✤ The idea that QA is distinct from development is a convenient fiction.! ✤ QA and testing have always been software engineering disciplines.
  18. 18. Sufficiently Advanced Monitoring is Indistinguishable fromTesting ✤ A Google engineer proposed this idea in 2007.! ✤ Nothing much has changed since then.! ✤ Sufficiently advanced technology is indistinguishable from magic. —Arthur C. Clarke! ✤ Sufficiently advanced technology is indistinguishable from a rigged demo. —Silicon Valley aphorism
  19. 19. Monitoring vs.Testing ✤ Yes you should monitor all the things.! ✤ Yes you should build real time dashboards.! ✤ Yes you should deploy continuously.! ✤ Yes you should fix production bugs on the fly.! ✤ No, none of the above replaces QA and testing.! ✤ Really, no.
  20. 20. QA andTesting are part of the software design process ✤ Monitoring has nothing to do with design.! ✤ Monitoring provides visibility into implementation.! ✤ QA and testing address design flaws.! ✤ That is, QA and testing are design tools.
  21. 21. Each Necessary but Only Jointly Sufficient ✤ Monitoring is part of the picture of overall system safety ✤ QA and testing are part of the picture of overall system safety ✤ Manual and automated testing — part of the overall picture ✤ Developers following standards — part of the picture ✤ People cooperating — part of the picture ✤ The system safety picture is intrinsically incomplete!
  22. 22. Limiting Risk Nancy Leveson on Limiting Risk:! ! ✤ Oversight! ! ✤ Limit complexity! ! ✤ Systems thinking
  23. 23. Applythebestsoftware engineeringprinciples… qualityassurance,testing… thehigheststandards. It'snotgoingtobeenough. —NancyLeveson Guy Sie on Flickr
  24. 24. Law of Leaky Abstractions “…tools which pretend to abstract out something, like all abstractions, leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don't save us time learning.” ! —Joel Spolsky
  25. 25. Failure is Scary But Also Inevitable Everyone—even cats—must deal with failure.
  26. 26. Leaky Abstractions in Software ✤ How does this relate to QA? ✤ in the new view of systems safety, safety is derived from being able to predict the behavior of the system. ✤ Since abstractions leak, no general abstraction can ever provide granular insight into system behavior.
  27. 27. The Mechanistic Fallacy: A Leaky Abstraction PushedToo Far ✤ Systems Thinking! ✤ Normal Failure! ✤ ETTO (Efficiency to Thoroughness Tradeoff)! ✤ Intractable Systems! ✤ The New View of system safety
  28. 28. Complex Systems are Intractable —Erik Hollnagel
  29. 29. “You can take a bicycle apart and put it back together… you cannot do that with a frog.” —Liz Keogh Bicycle vs. Frog
  30. 30. How is a Software System Like a Frog? ✤ It’s a running system.! ✤ It’s changing all the time.! ✤ Conceptually can’t be decomposed into discrete components.! ✤ A frog is intractable.
  31. 31. Powers ofTen
  32. 32. Let’s compare a bicycle to the universe ✤ How complex is a bicycle?
  33. 33. Bicycles and universes are not on the same scale of complexity ✤ About 30 parts ~ 1/30 = 0.03
  34. 34. ✤ How many components does a program have?! ✤ Consider the range of computational “parts” that exist between 1 microsecond and a half hour of computational time [Edsger Dijkstra]! ✤ 0.001 / 1800 = 5.55-7! ✤ Notice how there’s an exponent this time? Now let’s compare a the universe to a computer program
  35. 35. Computers and universes are on the same scale of complexity
  36. 36. What does n-7 levels of hierarchy look like?
  37. 37. The Pesticide Paradox and the Complexity Barrier “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”! “Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity.”! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! —Boris Beizer
  38. 38. The Pesticide Paradox ✤ Roaches have decided to terraform your apartment to suit their needs. Bug terminator summoned; pesticides released. ! ✤ Dead roaches. Problem solved or another problem created? ! ✤ An unintended consequence: the more efficient the pesticide is the greater the chance that the genes for pesticide resistance will dominate the gene pool going forward. ! ✤ This killing “all the bugs” actually results, over the long term, in large generations of bugs that will not be so easy to kill.
  39. 39. The Complexity Barrier ✤ There is an upper limit to how much complexity the human brain can handle. ! ✤ Simplifying an existing system is a very difficult problem.! ✤ There are no general solutions for simplification of systems.
  40. 40. ! ! ! ✤ Simplicity and design are equally important until you have to pick one; then pick simplicity. ! ! ✤ Modern business practices assume that there is no functional upper limit on system complexity. ! ! MIT vs. New Jersey! a.k.a “worse is better”
  41. 41. Want to Reduce Cost? Communicate!
  42. 42. Conway’s Game of Life 1. Survivals. Every counter with two or three neighboring counters survives for the next generation. 2. Deaths. Each counter with four or more neighbors dies (is removed) from overpopulation. Every counter with one neighbor or none dies from isolation. 3. Births. Each empty cell adjacent to exactly three neighbors--no more, no fewer--is a birth cell. A counter is placed on it at the next move. Delightfully Simple Rules ! ! -Martin Gardner
  43. 43. In real systems the rules aren’t as delightfully simple. Jonathan Borofsky, Counting 1 to 3227146, 1969/1976!
  44. 44. Jenkins lab (frequently-running job) 1. We will need test data. So we will create two jobs: a frequently running job and a long running job! 2. Set up a Jenkins instance (download the installer from! 3. Go to http://localhost: 8080/! 4. Click “new item”! 5. Type a name for your frequently-running project! 6. Click “build a freestyle software project” and click “OK”! 7. Create a new build step and type the code for your frequently running project ! 8. Schedule the job to run once a minute (using the Cron syntax as described in the inline help)
  45. 45. Jenkins User Conference New York, May 17 2012 #jenkinsconf A Review of My Previously Published Approach to Jenkins API Automation
  46. 46. Jenkins User Conference New York, May 17 2012 #jenkinsconf The Jenkins JSON API To read the documentation, go to You can append /api/json to the end of nearly any Jenkins URL to get JSON data. for latest builds json for history of a specific build. for slave information.
  47. 47. Jenkins User Conference New York, May 17 2012 #jenkinsconf Using depth= to get more granular data If the API response doesn't contain some data that you expected, try appending ? depth=1 to the URL. If you still don't get what you want, increase the integer value. Usually you'll keep getting more data up until around ?depth=5 Exactly what and how much data you'll get is dependent on the configuration of your Jenkins instance.
  48. 48. Jenkins User Conference New York, May 17 2012 #jenkinsconf Drawbacks of using depth= Depending on how deep you go into the API response, you can wind up with a lot of data. Such large responses can be expensive to download. In some cases you can request a response so large that you will wind up DDoSing Jenkins!
  49. 49. Jenkins User Conference New York, May 17 2012 #jenkinsconf Using tree= to filter the API response The tree= URL parameter is like a SQL query. Use depth= to look at the wealth of information available. Then use tree= to select only the information you actually need. This can dramatically reduce the size of your API responses.
  50. 50. Jenkins User Conference New York, May 17 2012 #jenkinsconf
  51. 51. Jenkins User Conference New York, May 17 2012 #jenkinsconf ?tree=busyExecutors
  52. 52. Jenkins User Conference New York, May 17 2012 #jenkinsconf ?tree=computer[displayName]
  53. 53. Suggested Reading ✤ Drive: The Surprising Truth About What Motivates Us by Dan Pink with RSAnimate! ✤ Ironies of Automation by Lisanne Bainbridge! ✤ Big Ball of Mud by Brian Foote and Joseph Yoder! ✤ The Rise of Worse is Better by Richard Gabriel! ✤ Worse is Better by Richard Gabriel! ✤ To Err is Human: ETTO Principle by Erik Hollnagel! ✤ The Law of Leaky Abstractions by Joel Spolsky! ✤ Designers and Women in Open Source by Vitorio Miliano! ✤ The fantastic combinations of John Conway’s new solitaire game “life” by Martin Gardner! Books! ✤ The Field Guide to Understanding Human Error by Sidney Dekker! ✤ How Google Tests Software by James Whittaker! ✤ Software Testing Techniques (2nd edition) by Boris Beizer! ✤ The Timeless Way of Building by Christopher Alexander!