CAST 2014 New York: The Art and Science of Testing The Association for Software Testing www.associationforsoftwaretesting.org COURSE DESCRIPTION Automated tools provide test professionals with the capability to make relevant observations even in the fastest-paced environments. Automated testing is also a powerful tool for improving communication between software engineers. This is important because good communication is a prerequisite for growing a great software engineering organization. This workshop will explore the continuous testing of software systems. Special focus will be given to the situation where the engineering team is deploying code to production so frequently that it is not possible to perform deep regression testing before each release. People who participate in this course will learn pragmatic automated testing strategies like: * Data analysis on the command line with find, grep and wc. * Network analysis with Chrome Inspector, Charles and netcat. * Using code churn to predict hotspots where bugs may occur. * Putting stack traces in context with automated SCM blame emails. * Using statsd to instrument a whole application. * Testing in production. * Monitoring-as-testing. Technical level: participants should have some familiarity with the command line and with editing code using a text editor or IDE. Familiarity with Git, SVN or another version control system is helpful but not required. Likewise some knowledge of Web servers is helpful but not required. It is desirable for participants to bring laptops. BIO From 2010 to 2012 Noah was a Test Architect at Etsy. He helped build Etsy's continuous integration system, and has helped countless other engineers develop successful automated testing strategies.These days Noah is an independent consultant in New York. He is passionate about helping engineers understand and use automated tools as they work to scale their applications more effectively.

1. Noah Sussman and Joanna Burgess! ! CAST NYC, August 11, 2014
Continuous Automated Testing:
A Communication System That Scales!
Jared Tarbell, Substrate

2. Continuous AutomatedTesting (C.A.T.)
Schedule
✤ 9:00-10:30: CAT!
✤ 10:30-10:50: Break !
✤ 10:50-12:30: CAT!
✤ 12:30-1:30: Lunch!
✤ 1:30-3:00: CAT!
✤ 3:00-3:30: Snack!
✤ 3:30-5:00: CAT

3. Load it Up
!
✤ Jenkins http://jenkins-ci.org/ !
✤ Golly http://golly.sourceforge.net/

5. “When I use a word it means just what I
choose it to mean-neither more nor less.”
What does QA mean to you?

6. Safetyisnotaboutthebreakageor
lackofqualityofsinglecomponents.
Managingqualityisaboutsingle
components,aboutseeinghowthey
meetparticularspecifications.
!
—SidneyDekker
Guy Sie on Flickr

7. In other words…when are we, and when aren’t we
speaking the same language? !
More to the point…how much does it matter?
Rolling the Dice

8. ContestedTerminology
✤ Quality Assurance!
✤ Testing!
✤ Manual Testing !
✤ Bugs!
✤ Accountability!
✤ QA Environment !

9. 4REALZ:“When I use a word it means
just what I choose it to mean” (ymmv)
✤ Like Humpty Dumpty, I am
using contested terminology
to support my own
viewpoints!
✤ When I use a word, it means
what I intend it to mean!
✤ If you are unsure what I mean,
raise your hand!!!

10. ETTO
✤ Efficiency-to-thoroughness-trade-offs (ETTO)!
✤ We do necessarily not pick the best option, we pick
the option that best satisfies our immediate needs
namely:!
✤ efficiency!
✤ thoroughness

11. AutomatedTests Are…
✤ Part of the software design process
✤ Runnable documentation
✤ NOT useful for debugging releases
✤ NOT useful for catching bugs in code

12. Tractable Systems
A system is tractable if the principles of the functioning
are known and have simple descriptions with few details.
Tractable systems do not change while being described.!
!

13. Intractable Systems
Intractable systems are complex systems. A defining
feature of intractable systems is that they are unsafe
and defy documentation.

14. DefiningTerminology
✤ Abstraction!
✤ Orders of Magnitude Growth !
✤ Cellular Automaton!
✤ Finite Automaton!
✤ Halting Problem !
✤ CAP Theorem !
✤ Acyclic Directed Graphs !
✤ Path reduction

15. Core Concepts
✤ Ironies of automation !
✤ Hawthorne Effects!
✤ Goodhart’s Law!
✤ Diseconomies of scale

16. Such dev.Very ops.Wow.

17. Autonomy Mastery & Purpose

18. devops: where is the QA?
✤ STOP ASKING ME where QA fits into devops.!
✤ ROFLMAO if QA and testing aren’t ALREADY part of dev for
you!!
✤ The idea that QA is distinct from development is a convenient
fiction.!
✤ QA and testing have always been software engineering disciplines.

19. Sufficiently Advanced Monitoring is
Indistinguishable fromTesting
✤ A Google engineer proposed this idea in 2007.!
✤ Nothing much has changed since then.!
✤ Sufficiently advanced technology is indistinguishable from
magic. —Arthur C. Clarke!
✤ Sufficiently advanced technology is indistinguishable from a
rigged demo. —Silicon Valley aphorism

20. Monitoring vs.Testing
✤ Yes you should monitor all the things.!
✤ Yes you should build real time dashboards.!
✤ Yes you should deploy continuously.!
✤ Yes you should fix production bugs on the fly.!
✤ No, none of the above replaces QA and testing.!
✤ Really, no.

21. QA andTesting are part of the
software design process
✤ Monitoring has nothing to do with design.!
✤ Monitoring provides visibility into implementation.!
✤ QA and testing address design flaws.!
✤ That is, QA and testing are design tools.

22. Each Necessary but Only Jointly Sufficient
✤ Monitoring is part of the picture of overall system
safety
✤ QA and testing are part of the picture of overall
system safety
✤ Manual and automated testing — part of the overall
picture
✤ Developers following standards — part of the picture
✤ People cooperating — part of the picture
✤ The system safety picture is intrinsically incomplete!

23. Limiting Risk
Nancy Leveson on Limiting Risk:!
!
✤ Oversight!
!
✤ Limit complexity!
!
✤ Systems thinking

24. Applythebestsoftware
engineeringprinciples…
qualityassurance,testing…
thehigheststandards.
It'snotgoingtobeenough.
—NancyLeveson
Guy Sie on Flickr

25. Law of Leaky Abstractions
“…tools which pretend to abstract out something, like all
abstractions, leak, and the only way to deal with the leaks
competently is to learn about how the abstractions work
and what they are abstracting. So the abstractions save us
time working, but they don't save us time learning.”
!
—Joel Spolsky

26. Failure is Scary But Also Inevitable
Everyone—even cats—must deal with failure.

27. Leaky Abstractions in Software
✤ How does this relate to QA?
✤ in the new view of systems safety, safety is
derived from being able to predict the
behavior of the system.
✤ Since abstractions leak, no general abstraction
can ever provide granular insight into system
behavior.

28. The Mechanistic Fallacy:
A Leaky Abstraction PushedToo Far
✤ Systems Thinking!
✤ Normal Failure!
✤ ETTO (Efficiency to Thoroughness Tradeoff)!
✤ Intractable Systems!
✤ The New View of system safety

29. Complex Systems are Intractable
—Erik Hollnagel

30. “You can take a bicycle apart and put it back together…
you cannot do that with a frog.” —Liz Keogh
Bicycle vs. Frog

31. How is a Software System Like a Frog?
✤ It’s a running system.!
✤ It’s changing all the time.!
✤ Conceptually can’t be
decomposed into discrete
components.!
✤ A frog is intractable.

32. Powers ofTen

33. Let’s compare a bicycle to the universe
✤ How complex is a bicycle?

34. Bicycles and universes are not on
the same scale of complexity
✤ About 30 parts ~ 1/30 = 0.03

35. ✤ How many components does a program have?!
✤ Consider the range of computational “parts” that exist
between 1 microsecond and a half hour of
computational time [Edsger Dijkstra]!
✤ 0.001 / 1800 = 5.55-7!
✤ Notice how there’s an exponent this time?
Now let’s compare a the universe to
a computer program

36. Computers and universes are on the
same scale of complexity

37. What does n-7 levels of
hierarchy look like?

38. The Pesticide Paradox
and the Complexity Barrier
“Every method you use to prevent or find bugs leaves a
residue of subtler bugs against which those methods are
ineffectual.”!
“Software complexity (and therefore that of bugs) grows
to the limits of our ability to manage that complexity.”!
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! —Boris Beizer

39. The Pesticide Paradox
✤ Roaches have decided to terraform your apartment to suit
their needs. Bug terminator summoned; pesticides released. !
✤ Dead roaches. Problem solved or another problem created? !
✤ An unintended consequence: the more efficient the pesticide
is the greater the chance that the genes for pesticide resistance
will dominate the gene pool going forward. !
✤ This killing “all the bugs” actually results, over the long term,
in large generations of bugs that will not be so easy to kill.

40. The Complexity Barrier
✤ There is an upper limit to how much complexity the
human brain can handle. !
✤ Simplifying an existing system is a very difficult
problem.!
✤ There are no general solutions for simplification of
systems.

41. !
!
!
✤ Simplicity and design are equally important
until you have to pick one; then pick simplicity. !
!
✤ Modern business practices assume that there is
no functional upper limit on system complexity. !
!
MIT vs. New Jersey!
a.k.a “worse is better”

42. Want to Reduce Cost?
Communicate!

43. Conway’s Game of Life
1. Survivals. Every counter with two or three neighboring counters
survives for the next generation.
2. Deaths. Each counter with four or more neighbors dies (is removed)
from overpopulation. Every counter with one neighbor or none dies
from isolation.
3. Births. Each empty cell adjacent to exactly three neighbors--no more,
no fewer--is a birth cell. A counter is placed on it at the next move.
Delightfully Simple Rules
! ! -Martin Gardner

44. In real systems the rules aren’t as delightfully simple.
Jonathan Borofsky, Counting 1 to 3227146, 1969/1976!

45. Jenkins lab (frequently-running job)
1. We will need test data. So we will create two jobs: a frequently running job and a
long running job!
2. Set up a Jenkins instance (download the installer from jenkinsci.org)!
3. Go to http://localhost: 8080/!
4. Click “new item”!
5. Type a name for your frequently-running project!
6. Click “build a freestyle software project” and click “OK”!
7. Create a new build step and type the code for your frequently running project !
8. Schedule the job to run once a minute (using the Cron syntax as described in the
inline help)

46. Jenkins User Conference New York, May 17 2012 #jenkinsconf
A Review of My Previously Published
Approach to Jenkins API Automation

47. Jenkins User Conference New York, May 17 2012 #jenkinsconf
The Jenkins JSON API
To read the documentation, go to
http://ci.example.com/api
You can append /api/json to the end of
nearly any Jenkins URL to get JSON data.
http://ci.example.com/api/json for latest
builds
http://ci.example.com/job/unit-tests/api/
json for history of a specific build.
http://ci.example.com/computer/api/json
for slave information.

48. Jenkins User Conference New York, May 17 2012 #jenkinsconf
Using depth= to get more granular data
If the API response doesn't contain some
data that you expected, try appending ?
depth=1 to the URL.
If you still don't get what you want, increase
the integer value.
Usually you'll keep getting more data up
until around ?depth=5
Exactly what and how much data you'll get
is dependent on the configuration of your
Jenkins instance.

49. Jenkins User Conference New York, May 17 2012 #jenkinsconf
Drawbacks of using depth=
Depending on how deep you go into the
API response, you can wind up with a lot of
data.
Such large responses can be expensive to
download.
In some cases you can request a response
so large that you will wind up DDoSing
Jenkins!

50. Jenkins User Conference New York, May 17 2012 #jenkinsconf
Using tree= to filter the API response
The tree= URL parameter is like a SQL query.
Use depth= to look at the wealth of
information available.
Then use tree= to select only the information
you actually need.
This can dramatically reduce the size of your
API responses.

51. Jenkins User Conference New York, May 17 2012 #jenkinsconf

52. Jenkins User Conference New York, May 17 2012 #jenkinsconf
?tree=busyExecutors

53. Jenkins User Conference New York, May 17 2012 #jenkinsconf
?tree=computer[displayName]

54. Suggested Reading
✤ Drive: The Surprising Truth About What Motivates Us by Dan Pink with RSAnimate https://www.youtube.com/watch?v=u6XAPnuFjJc!
✤ Ironies of Automation by Lisanne Bainbridge http://www.ise.ncsu.edu/nsf_itr/794B/papers/Bainbridge_1983_Automatica.pdf!
✤ Big Ball of Mud by Brian Foote and Joseph Yoder http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.316.433&rep=rep1&type=pdf!
✤ The Rise of Worse is Better by Richard Gabriel http://www.jwz.org/doc/worse-is-better.html!
✤ Worse is Better by Richard Gabriel http://dreamsongs.com/worseisbetter.html!
✤ To Err is Human: ETTO Principle by Erik Hollnagel http://www.namahn.com/we-share/interviews/erik-hollnagel-err-human-etto-principle!
✤ The Law of Leaky Abstractions by Joel Spolsky http://www.joelonsoftware.com/articles/LeakyAbstractions.html!
✤ Designers and Women in Open Source by Vitorio Miliano http://old.vi.to/designers-and-women-in-open-source.html!
✤ The fantastic combinations of John Conway’s new solitaire game “life” by Martin Gardner http://www.ibiblio.org/lifepatterns/october1970.html!
Books!
✤ The Field Guide to Understanding Human Error by Sidney Dekker http://amzn.to/1mdwNdU!
✤ How Google Tests Software by James Whittaker http://amzn.to/XvMtUl!
✤ Software Testing Techniques (2nd edition) by Boris Beizer http://amzn.to/YqtevB!
✤ The Timeless Way of Building by Christopher Alexander http://amzn.to/1tgW3UJ!