0
7 Deadly Sins
of Agile
Software Test
Automation
Craig Smith
Welcome…
Image: © Gracie Films / 20th Century Fox Television http://www.paleodietrecipes.org.uk/paleo-images/content/homer...
Adrian Smith
@adrianlsmith
Geeks Hate Repetition
It Started
With…
Image: © New Line Cinema http://www.imfdb.org/images/5/5e/SevenCover.jpg
© United Plankton Pictures https...
Discussion:
What is
stopping you
from being
great at
automation?
Image: http://toworkandback.com/wp-content/uploads/2012/0...
Image: http://scitascienda.files.wordpress.com/2013/04/barney-stinson-quote-when-i-get-sad-i-stop-being-sad-and-be-awesome...
Image: http://www.cupojoy.com/images/image/Quality%20Inn%20Exterior.jpg
Building Quality In
AUTOMATED
TESTING
Automated
Testing
Image: http://www.lastwordonnothing.com/wp-content/uploads/2011/12/iStock_000007001634...
Envy
Flawed comparison of manual testing & automation
Image: http://workingtropes.lmc.gatech.edu/wiki/images/Man-vs-machin...
How Management
See Testing
Image: http://www.craigsfire.com/wp-content/uploads/2010/02/pool.jpg
How Management
Would Like To See
Testing
Image: http://us.123rf.com/400wm/400/400/dotshock/dotshock1012/dotshock101200006/...
Repeat After Me…
Agile
≠
Automated Testing
Software is a Series of Loops
Image: http://2.bp.blogspot.com/_4G2ptSlZZn0/R9nRjimkLII/AAAAAAAAAqA/tzYJjRyhLkk/s400/brekky...
 End to end automated tests and cover 10 conditions per
test, manual covers 1 condition per test
 Automated tests cover ...
Manual vs Automated…
A Flawed Comparison
Image: http://fvhs.com/wp-content/uploads/2012/11/Screen-shot-2012-11-15-at-7.49....
Testing is more than a series
of merely repeatable actions
Image: http://cdn.themis-media.com/media/global/images/library/...
Look
Left…
Look
Right…
Image: http://3.bp.blogspot.com/_RdF7S0TNAHY/TOqn8MLYajI/AAAAAAAAAJI/QDBFdixiq0E/s1600/2006-06-17%252520Falling-piano.gif
...
21
Image: © Jerry Bruckheimer Television http://images2.fanpop.com/images/photos/4700000/CSI-Las-Vegas-9x18-Mascara-csi-47...
 Regression testing –
assessing current state
 Automation of test support
activities
Ideal Automation Targets
 Data gen...
Common “Envy” Symptoms
 Relying on automation as
the basis for all testing
activities
 All tests are built by
developers...
Approach: Desired Role
Automation:
I work with developers to
automate tests that
provide business value
and identify syste...
Approach: Report
on Confidence
“Envy” Lessons Learned
 Avoid comparison between
manual and automated
testing - both are needed
 Distinguish between the...
•Over indulging on commercial test tools
Gluttony
Over indulging on commercial test tools…
Image: © Universal Pictures htt...
Discussion: What Automated
Tools Do You Use & Why?
Image: http://www.newnorfolkmitre10.com.au/pics/66/66718b.jpg
Some vendors
believe they
have the
“secret
sauce”
for
automation
Image: http://topnews.ae/images/HP-Sauce.jpg
Promise of Automation
Image: http://thebsreport.files.wordpress.com/2009/10/traveling-salesmen-medicine-show.jpg?w=500&h=3...
Image: http://www.melhotornot.com/wp-content/uploads/Rowboat.jpg
Commercial Restricts Usage
Justifying The Expense
I have spent
a lot of
money on
this tool, we
better be
using it for
everything!
Image: © Warner Bro...
 Underlying commercial tools
technology often not compatible
with development tool chain
Incompatible Technology
 Specia...
Common “Gluttony” Symptoms
 A commercial tool forms
the basis of a testing
strategy
 Only certain teams or
individuals c...
Approach: Tools Guide
Approach: Craftsmanship
Image: © Wild Dancer Productions / Touchstone Television http://www.tvacres.com/images/props_tool_...
“Gluttony” Lessons Learned
 Favour open software tools
where ever possible
 Use tools that can easily be
supported by th...
•User interface forms the basis
for all testing
Lust
User interface forms the basis for all testing…
Image: http://stufici...
Traditionally Test via the UI
Image: http://stovallorganizing.com/wp-content/uploads/2013/04/Turtleing.jpg
Manual
Exploratory
Collaboratively
built around
system behaviour
Developer built
optimised for fast
feedback
Confidence
Sp...
Understand The Architecture
Image: http://harrison.kyschools.us/hcms/TechProjects2011.htm
Test Design
F Fast
I Isolated
R Repeatable
S Self Verifying
T Timely
FIRST Test Properties
Image: http://www.moto-choice.com/PRImages/...
Common “Lust” Symptoms
 Testers cannot draw the
application or system
architecture
 Large proportion of tests
are run th...
Discussion: What
technologies do you need to
understand to automate?
Image: © Paramount Pictures http://americanobjective....
Agile Testing
Command Line
Interface
Continuous
Integration
Version Control
Build Tools
Capture Replay
(Selenium)
HTML
Tes...
Approach: Use
UI Tools for
Right Purpose
Use tools like Selenium
sparingly, good for
exploratory testing
Only use tools li...
“Lust” Lessons Learned
 Limit the investment in
automated tests that are
executed through the user
interface
 Collaborat...
Pride
Too proud to
collaborate
when creating
tests
Pride
Too proud to collaborate when creating tests…
Image: http://latim...
Poor
Collaboration
= Poor Tests
Image: http://2.bp.blogspot.com/-RYkFdryP-Co/Tj7-0q3DyPI/AAAAAAAABJU/6tETrv8zyb4/s1600/Ops...
Poor
Collaboration
or
Dedicated
Roles
=
Duplication
Image: © Universal Pictures http://cdn3.whatculture.com/wp-content/upl...
No
Definition
of
Quality
Image: https://lh3.googleusercontent.com/-46OEX5ffUX4/TtVNoEnFPmI/AAAAAAAADBo/g55cQRy5im0/s800/zi...
Discussion:
What is
something
you regard
as quality?
Image: http://farm7.staticflickr.com/6115/6287632596_5cc5a681c8_z.jpg
Analyst /
Customer
Developer Tester
Automation
Elaboration /
Specification
Acceptance
Criteria
High
Performing
AGILE
Proje...
Developer
Tester
Analyst
More
Technical
Less
Technical
Code
Design
UnitTest
Automated
Functional/
Specialist
Tests
Manual
...
Red
GreenRefactor
Red
GreenRefactor
ATDD
ATDD
TDD
Image: http://concordion.org/image/example/online-shop/AcceptanceTest.png
Specification
by Example
Common “Pride” Symptoms
 Automated tests are being
built in isolation from team
 Intent of tests is unclear
or not match...
Approach:
Defining Quality
Quality
Advocates
What does quality mean
to the different roles in
the team?
Quality
Taxonomy
W...
Approach:
Disband Testing
Teams
Image: http://prince2pm.files.wordpress.com/2012/01/industry-groups-product-groups.jpg
Approach:
Simple
Strategy
“Pride” Lessons Learned
 Collaborate to create good
tests and avoid duplication
 Limit the investment in UI
based automa...
Sloth
Too lazy to properly maintain automated tests…
Image: http://photos.pcpro.co.uk/blogs/wp-content/uploads/2011/06/Asl...
New
Feature
System
Interface
Change
OS
Patch
Reference
Data
Changes
Time
Many Causes of
Automated Test Failure
Time
Cost/Effort
Potential
Value of
Maintained
Automated Test
Suite
Value of
Unmaintained
Automated Test
Suite
Manual test...
Discussion:
How often
are you
red?
How do you
know?
Who cares?
Image: http://www.eriding.net/media/photos/environment/traf...
Continuous Integration
Watch
Code
Build /
Compile
Run
Tests /
Analysis
Publish
Results
Stop
The
Line!
Failure?
Continuous Delivery
Image: http://www.thoughtworks.com/imgs/drawing-cd-header.png
#notesting
Image: http://3.bp.blogspot.com/-jy4Sc0m-oTA/UU9qrGJw56I/AAAAAAAAAC8/zSlPJS9Ed7c/s200/No+testing.JPG
Common “Sloth” Symptoms
 Test suite has not been recently
run - state is unknown
 Continuous Integration history
shows c...
Approach:
Visible
Status /
Stop The
Line
“Sloth” Lessons Learned
 Ensure automated tests are
executed using a Continuous
Integration environment
 Ensure tests ar...
Rage
Frustration with slow, brittle or unreliable tests…
Image: © 20th Century Fox http://kei.dizkartes.nl/static/img/_com...
Fast
Feedback Image: http://www.digitallanding.com/wp-content/uploads/2012/07/Faster_Internet-e1346361972417.jpg
Discussion:
What is a
good…
Time
benchmark
for a build?
Amount of
coverage?
Image: http://www.mdr.de/sachsen/bild235170_v-...
Image: http://1.bp.blogspot.com/-pKB3xu6zOc8/TWU1W_4OEqI/AAAAAAAAAzk/ygEOBNumv-Q/s400/nespresso-clooney.jpg
No Waiting…
100%*
Image: http://shirtoid.com/wp-content/uploads/2013/02/100-percent-sci-fi1.jpg
Common “Rage” Symptoms
 Slow Automated Tests
 Brittle
 Unreliable
 Large datasets
 Unnecessary integrations
 Inadequ...
HEALTH
PROJECT
DEVELOPMENT
TESTING
USER
0
1
2
3
4
5
0
1
2
3
4
5
6
7
1 2 3 4 5 6 7 8
NewRisks&IssuesRaised
NumberofRisks&Is...
Approach:
Cloud /
Central /
Open
Source ALM
“Rage” Lessons Learned
 Treat automated tests with
the same importance as
production code
 Review, refactor, improve ......
Greed
Trying to cut costs through automation…
Image: © 20th Century Fox http://blogs.telegraph.co.uk/news/files/2010/09/go...
Lure of saving labour
Image: http://www.caradvice.com.au/thumb/640/556/wp-content/uploads/2010/11/Car-salesman.jpg
Automation is not cheap!
Image: http://designbuildsource.com.au/wp-content/uploads/2012/03/australian-money.jpg
Common “Greed” Symptoms
 Investment in commercial
tools using a business-case
based on reducing headcount
 Using a predi...
New Software
Test Engineer
Role
New Graduates,
Recruits, Consultants
& Partners
BA
SME
Other
Up
Skill
Assessment
Current T...
Approach:
Outsource
Teams Not
Testing
Image: https://sphotos-b-ord.xx.fbcdn.net/hphotos-ash3/p480x480/942876_5051836328882...
Approach: Avoid Test
Automation Projects
Image: http://www.hifi-writer.com/he/progscan/Files/abcestablish.jpg
“Greed” Lessons Learned
 Ensure the reasons for
automation are clear
and are NOT based
purely on saving
money/headcount
...
Quality
is Key
Image: http://gs-press.com.au/images/news_articles/cache/key_hand_resized-600x0.jpg
Approach:
Quality
Assessment
 Tests are NOT executed on a regular basis
 State of current defects is unknown
 Functional testing occurs regularly
 ...
Wrap
Up
Image: http://www.pria.com.au/sb_cache/priablog/id/924/f/7%20deadly%20sins.jpg
Envy Flawed comparison of manual testing & automation
Gluttony Over indulging on commercial test tools
Lust User interface...
Why Use Automation?
Image: © Fuzzy Door Productions / 20th Century Fox Television
Don’t Lose
Sight of the
Goal
How geeks really workHow Geeks Can Work Together
Questions?
Craig Smith
http://www.craigsmith.id.au
craig@craigsmith.id.au
@smithcdau
Upcoming SlideShare
Loading in...5
×

7 Deadly Sins of Agile Software Test Automation

6,243

Published on

Talk delivered by Craig Smith at Agile 2013 in Nashville, USA on 8 August 2013.

Automated software testing is a key enabler for teams wanting to build high quality software that can be progressively enhanced and continuously released. To ensure development practices are sustainable, automated testing must be treated as a first-class citizen and not all approaches are created equal. Some approaches can accumulate technical debt, cause duplication of effort and even team dysfunctions.

The seven deadly sins of automated software testing are a set of common anti-patterns that have been found to erode the value of automated testing resulting in long term maintenance issues and ultimately affecting the ability of development teams to respond to change and continuously deliver.

Taking the classic seven sins (Gluttony, Sloth, Lust, Envy, Rage, Pride, Greed) as they might be applied to test automation we will discuss how to identify each automated sin and more importantly provide guidance on recommended solutions and how to avoid them in the first place.

Published in: Technology, Education
1 Comment
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,243
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
64
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "7 Deadly Sins of Agile Software Test Automation"

  1. 1. 7 Deadly Sins of Agile Software Test Automation Craig Smith
  2. 2. Welcome… Image: © Gracie Films / 20th Century Fox Television http://www.paleodietrecipes.org.uk/paleo-images/content/homer-confession.jpg
  3. 3. Adrian Smith @adrianlsmith
  4. 4. Geeks Hate Repetition
  5. 5. It Started With… Image: © New Line Cinema http://www.imfdb.org/images/5/5e/SevenCover.jpg © United Plankton Pictures https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/p480x480/429897_565831523456105_395040135_n.jpg
  6. 6. Discussion: What is stopping you from being great at automation? Image: http://toworkandback.com/wp-content/uploads/2012/08/stop-sign.jpg
  7. 7. Image: http://scitascienda.files.wordpress.com/2013/04/barney-stinson-quote-when-i-get-sad-i-stop-being-sad-and-be-awesome-instead-true-story.jpg?w=500&h=353
  8. 8. Image: http://www.cupojoy.com/images/image/Quality%20Inn%20Exterior.jpg Building Quality In
  9. 9. AUTOMATED TESTING Automated Testing Image: http://www.lastwordonnothing.com/wp-content/uploads/2011/12/iStock_000007001634Small.jpg
  10. 10. Envy Flawed comparison of manual testing & automation Image: http://workingtropes.lmc.gatech.edu/wiki/images/Man-vs-machine.jpg
  11. 11. How Management See Testing Image: http://www.craigsfire.com/wp-content/uploads/2010/02/pool.jpg
  12. 12. How Management Would Like To See Testing Image: http://us.123rf.com/400wm/400/400/dotshock/dotshock1012/dotshock101200006/8437158-internet-network-server-room-with-computers-racks-and-digital-receiver-for-digital-tv.jpg
  13. 13. Repeat After Me… Agile ≠ Automated Testing
  14. 14. Software is a Series of Loops Image: http://2.bp.blogspot.com/_4G2ptSlZZn0/R9nRjimkLII/AAAAAAAAAqA/tzYJjRyhLkk/s400/brekky05011049.jpg
  15. 15.  End to end automated tests and cover 10 conditions per test, manual covers 1 condition per test  Automated tests cover all high priority business flows, manual tests mostly cover lower priority flows  Metrics need context! % Regression test cases automated Automated Manual Discussion: How Much Automated Testing Is Enough?
  16. 16. Manual vs Automated… A Flawed Comparison Image: http://fvhs.com/wp-content/uploads/2012/11/Screen-shot-2012-11-15-at-7.49.37-AM.png
  17. 17. Testing is more than a series of merely repeatable actions Image: http://cdn.themis-media.com/media/global/images/library/deriv/71/71335.jpg
  18. 18. Look Left… Look Right…
  19. 19. Image: http://3.bp.blogspot.com/_RdF7S0TNAHY/TOqn8MLYajI/AAAAAAAAAJI/QDBFdixiq0E/s1600/2006-06-17%252520Falling-piano.gif Look Up!
  20. 20. 21 Image: © Jerry Bruckheimer Television http://images2.fanpop.com/images/photos/4700000/CSI-Las-Vegas-9x18-Mascara-csi-4772098-2000-1333.jpg CSI: Software
  21. 21.  Regression testing – assessing current state  Automation of test support activities Ideal Automation Targets  Data generation / sub-setting  Load generation  Non-functional testing - performance, security, ...)  Deterministic problems  Big data problems Image: http://media.merchantcircle.com/30134145/mobile%20target_full.png
  22. 22. Common “Envy” Symptoms  Relying on automation as the basis for all testing activities  All tests are built by developers  Absence of code reviews  Absence of exploratory testing  Absence of user testing Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG
  23. 23. Approach: Desired Role Automation: I work with developers to automate tests that provide business value and identify system risks Collaboration: I work with analysts and SMEs to ensure that testable acceptance criteria are created for all stories Strategy & Planning: I am involved in the project at all stages to ensure that testing provides the greatest value and quality objectives are achieved Tools & Techniques: I use both manual and automated techniques using the preferred testing tools based on the situation Architecture: I have an understanding of the system architecture and can create tests that verify individual components and the system as a whole Development: I take an interest in development practices and monitor code quality metrics Estimating: I am involved in developing estimates for projects at a story level, for tools and infrastructure and also for deployment/release activities Reporting: I provide metrics that give insight into project health and system quality Agile: I understand Agile project delivery and the differences between testing in the different phases of an phase Qualifications & Training: I have recognised qualifications in testing and continually update and maintain my skills Recruitment & Development: I am attracted to the organisation because testing is a cool career path that offers heaps of opportunities and a way of continually developing my skills Community & Teams: I am part of a community of testers that are embedded within teams but share common values
  24. 24. Approach: Report on Confidence
  25. 25. “Envy” Lessons Learned  Avoid comparison between manual and automated testing - both are needed  Distinguish between the automation and the process that is being automated  Use automation to provide a baseline  Use automation in conjunction with manual techniques Image: http://idahoptv.org/dialogue4kids/images/season10/robots/bookbot.gif
  26. 26. •Over indulging on commercial test tools Gluttony Over indulging on commercial test tools… Image: © Universal Pictures http://static.ifood.tv/files/external-images/photo-26589.jpg
  27. 27. Discussion: What Automated Tools Do You Use & Why? Image: http://www.newnorfolkmitre10.com.au/pics/66/66718b.jpg
  28. 28. Some vendors believe they have the “secret sauce” for automation Image: http://topnews.ae/images/HP-Sauce.jpg
  29. 29. Promise of Automation Image: http://thebsreport.files.wordpress.com/2009/10/traveling-salesmen-medicine-show.jpg?w=500&h=364
  30. 30. Image: http://www.melhotornot.com/wp-content/uploads/Rowboat.jpg Commercial Restricts Usage
  31. 31. Justifying The Expense I have spent a lot of money on this tool, we better be using it for everything! Image: © Warner Bros Television Distribution / Screen Gems http://3.bp.blogspot.com/-1L3LVOu0cZg/UPs4gju7DJI/AAAAAAAANx0/mrdx5RYSSW0/s1600/jetsons_L74.jpg
  32. 32.  Underlying commercial tools technology often not compatible with development tool chain Incompatible Technology  Special file formats or databases  Lack of version control for tests and/or cannot be versioned within the software  Not easily combined with Continuous Integration Image: http://virtualmarketingofficer.com/wp-content/uploads/2009/06/square-peg.jpg  Not easily adapted or extended by the developers
  33. 33. Common “Gluttony” Symptoms  A commercial tool forms the basis of a testing strategy  Only certain teams or individuals can access a tool or run tests  Developers have not been consulted in the selection of a testing tools  “We always use <insert tool-name> for testing!” Image: http://resources2.news.com.au/images/2008/09/02/va1237328073953/burger-6230645.gif
  34. 34. Approach: Tools Guide
  35. 35. Approach: Craftsmanship Image: © Wild Dancer Productions / Touchstone Television http://www.tvacres.com/images/props_tool_time_binford.jpg
  36. 36. “Gluttony” Lessons Learned  Favour open software tools where ever possible  Use tools that can easily be supported by the development team and play nicely with existing development tool chain  Ensure any commercial tools can be executed in a command-line mode so it can be automated Image: https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-prn1/p480x480/9346_506859889386718_1005343687_n.jpg  Educate!
  37. 37. •User interface forms the basis for all testing Lust User interface forms the basis for all testing… Image: http://stuficionado.net/wp-content/uploads/2012/11/computer_hug.jpeg
  38. 38. Traditionally Test via the UI Image: http://stovallorganizing.com/wp-content/uploads/2013/04/Turtleing.jpg
  39. 39. Manual Exploratory Collaboratively built around system behaviour Developer built optimised for fast feedback Confidence Speed/Feedback Exercises components and systems Investment Profile
  40. 40. Understand The Architecture Image: http://harrison.kyschools.us/hcms/TechProjects2011.htm
  41. 41. Test Design
  42. 42. F Fast I Isolated R Repeatable S Self Verifying T Timely FIRST Test Properties Image: http://www.moto-choice.com/PRImages/EditorImages/148330-bathurst-finish-2.jpg
  43. 43. Common “Lust” Symptoms  Testers cannot draw the application or system architecture  Large proportion of tests are run through the UI  Testers have limited technical skills  No collaboration with developers  Intent of tests is unclear Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG
  44. 44. Discussion: What technologies do you need to understand to automate? Image: © Paramount Pictures http://americanobjective.com/wp-content/uploads/2012/04/original.jpg
  45. 45. Agile Testing Command Line Interface Continuous Integration Version Control Build Tools Capture Replay (Selenium) HTML Test Maintenance & Data Specification By Example / ATDD / BDD Concordion / Cucumber SQL Web Services Approach: Test Automation Course
  46. 46. Approach: Use UI Tools for Right Purpose Use tools like Selenium sparingly, good for exploratory testing Only use tools like HTMLUnit or scrape screens when no other options available
  47. 47. “Lust” Lessons Learned  Limit the investment in automated tests that are executed through the user interface  Collaborate with developers  Focus investment in automation at lowest possible level with clear test intent  Ensure automation gives fast feedback Image: http://foodcandy.com/wp-content/uploads/2013/03/flowers-and-chocolate1.jpg
  48. 48. Pride Too proud to collaborate when creating tests Pride Too proud to collaborate when creating tests… Image: http://latimesblogs.latimes.com/.a/6a00d8341c630a53ef0120a955baf0970b-600wi
  49. 49. Poor Collaboration = Poor Tests Image: http://2.bp.blogspot.com/-RYkFdryP-Co/Tj7-0q3DyPI/AAAAAAAABJU/6tETrv8zyb4/s1600/OpsSilo.jpg
  50. 50. Poor Collaboration or Dedicated Roles = Duplication Image: © Universal Pictures http://cdn3.whatculture.com/wp-content/uploads/2013/03/twins-poster.jpg
  51. 51. No Definition of Quality Image: https://lh3.googleusercontent.com/-46OEX5ffUX4/TtVNoEnFPmI/AAAAAAAADBo/g55cQRy5im0/s800/zimogs_kvalitates_zime.jpg
  52. 52. Discussion: What is something you regard as quality? Image: http://farm7.staticflickr.com/6115/6287632596_5cc5a681c8_z.jpg
  53. 53. Analyst / Customer Developer Tester Automation Elaboration / Specification Acceptance Criteria High Performing AGILE Project Manager Good Collaboration
  54. 54. Developer Tester Analyst More Technical Less Technical Code Design UnitTest Automated Functional/ Specialist Tests Manual Test Exploratory Test Requirements Customer Collab. Developers need more testing involvement Testers need more technical involvement Analysts need more testing involvement Skills
  55. 55. Red GreenRefactor Red GreenRefactor ATDD ATDD TDD
  56. 56. Image: http://concordion.org/image/example/online-shop/AcceptanceTest.png Specification by Example
  57. 57. Common “Pride” Symptoms  Automated tests are being built in isolation from team  Intent of tests is unclear or not matched to quality  Poor automation design (abstraction, encapsulation, ...)  Maintainability or compatibility issues Image: http://farm8.staticflickr.com/7120/7683912508_b23c4c9ff0_b.jpg
  58. 58. Approach: Defining Quality Quality Advocates What does quality mean to the different roles in the team? Quality Taxonomy What are quality attributes? Quality Prioritisation How do we know what quality attributes to include? Quality Tradeoff Risks What are risks with the quality attributes we are trading off? Quality Measurement How do we test and measure quality? Success Sliders How does quality relate to the sliders? Quality Definition What does quality mean? Next Steps How do we apply quality to our work 1 2 3 4 5 6 7 8
  59. 59. Approach: Disband Testing Teams Image: http://prince2pm.files.wordpress.com/2012/01/industry-groups-product-groups.jpg
  60. 60. Approach: Simple Strategy
  61. 61. “Pride” Lessons Learned  Collaborate to create good tests and avoid duplication  Limit the investment in UI based automated tests  Collaborate with developers to ensure good technical practices (encapsulation, abstraction, reuse, ... )  Test code = Production code Image: https://c479107.ssl.cf2.rackcdn.com/files/7178/article/width668/rfwyhksh-1327469166-1327469251.jpg
  62. 62. Sloth Too lazy to properly maintain automated tests… Image: http://photos.pcpro.co.uk/blogs/wp-content/uploads/2011/06/Asleep-at-computer.jpg
  63. 63. New Feature System Interface Change OS Patch Reference Data Changes Time Many Causes of Automated Test Failure
  64. 64. Time Cost/Effort Potential Value of Maintained Automated Test Suite Value of Unmaintained Automated Test Suite Manual test execution Maintained automation Unmaintained automation Importance of Maintenance
  65. 65. Discussion: How often are you red? How do you know? Who cares? Image: http://www.eriding.net/media/photos/environment/traffic/070328_jbean_mp_environment_traffic_0135.jpg
  66. 66. Continuous Integration Watch Code Build / Compile Run Tests / Analysis Publish Results Stop The Line! Failure?
  67. 67. Continuous Delivery Image: http://www.thoughtworks.com/imgs/drawing-cd-header.png
  68. 68. #notesting Image: http://3.bp.blogspot.com/-jy4Sc0m-oTA/UU9qrGJw56I/AAAAAAAAAC8/zSlPJS9Ed7c/s200/No+testing.JPG
  69. 69. Common “Sloth” Symptoms  Test suite has not been recently run - state is unknown  Continuous Integration history shows consistent failures following development changes / release  Test suite requires manual intervention  Duplication within automation code Image: http://commons.wikimedia.org/wiki/File:Cairns_Australia_Beer_Run.jpg  Small changes trigger a cascade of failures
  70. 70. Approach: Visible Status / Stop The Line
  71. 71. “Sloth” Lessons Learned  Ensure automated tests are executed using a Continuous Integration environment  Ensure tests are always runnable - even if the system in not being actively developed  Make test results visible - create transparency of system health  Ensure collaboration between developers and testers Image: http://1.bp.blogspot.com/_yEAvQXpDW6I/S-8HgogJWpI/AAAAAAAAAY0/phpZuCfP610/s400/Linea%2520Berocca.jpg
  72. 72. Rage Frustration with slow, brittle or unreliable tests… Image: © 20th Century Fox http://kei.dizkartes.nl/static/img/_common/header_404.jpg
  73. 73. Fast Feedback Image: http://www.digitallanding.com/wp-content/uploads/2012/07/Faster_Internet-e1346361972417.jpg
  74. 74. Discussion: What is a good… Time benchmark for a build? Amount of coverage? Image: http://www.mdr.de/sachsen/bild235170_v-standardBig_zc-3ad1f7a1.jpg?version=49443
  75. 75. Image: http://1.bp.blogspot.com/-pKB3xu6zOc8/TWU1W_4OEqI/AAAAAAAAAzk/ygEOBNumv-Q/s400/nespresso-clooney.jpg No Waiting…
  76. 76. 100%* Image: http://shirtoid.com/wp-content/uploads/2013/02/100-percent-sci-fi1.jpg
  77. 77. Common “Rage” Symptoms  Slow Automated Tests  Brittle  Unreliable  Large datasets  Unnecessary integrations  Inadequate environments  Too many / UI / manual tests  Time bound data  External / Prod integrations  Reliance on UI / sequence  False positives / confidence  Failures being ignored  Workarounds investigated
  78. 78. HEALTH PROJECT DEVELOPMENT TESTING USER 0 1 2 3 4 5 0 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 NewRisks&IssuesRaised NumberofRisks&Issues Iteration TEST COVERAGE 0 5 10 15 20 25 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 NumberofTests NumberofDefects Iteration MAINTAINABILITY PERFORMANCE 0 200 400 600 800 1000 1200 1400 1600 1800 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 1 2 3 4 5 6 7 8 LinesofTestCode LinesofCode Iteration OVERALL $- $5 $10 $15 $20 $25 $30 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 BusinessValue NumberofFeatures Iteration Approach: BVC
  79. 79. Approach: Cloud / Central / Open Source ALM
  80. 80. “Rage” Lessons Learned  Treat automated tests with the same importance as production code  Review, refactor, improve ...  Apply a “Stop the line” approach to test failure  Eliminate (quarantine) unreliable tests Image: http://us.123rf.com/400wm/400/400/edma/edma0907/edma090700217/5226661-businessman-doing-yoga-exercises-in-a-city-park.jpg  Ensure collaboration with developers
  81. 81. Greed Trying to cut costs through automation… Image: © 20th Century Fox http://blogs.telegraph.co.uk/news/files/2010/09/gordon-gekko.jpg
  82. 82. Lure of saving labour Image: http://www.caradvice.com.au/thumb/640/556/wp-content/uploads/2010/11/Car-salesman.jpg
  83. 83. Automation is not cheap! Image: http://designbuildsource.com.au/wp-content/uploads/2012/03/australian-money.jpg
  84. 84. Common “Greed” Symptoms  Investment in commercial tools using a business-case based on reducing headcount  Using a predicted ROI as a way of reducing budget for testing  Consolidating automated testing within a special group Image: © Gracie Films / 20th Century Fox Television http://thegreeneconomy.com/wp-content/uploads/2012/05/greedy-246x220.gif
  85. 85. New Software Test Engineer Role New Graduates, Recruits, Consultants & Partners BA SME Other Up Skill Assessment Current Test Analysts Capable + Current Skills + Desire Capable + Desire Not Capable + No Desire Not Capable + Desire **Assessment = Technical Test + Interview** Approach: Assist & Assess
  86. 86. Approach: Outsource Teams Not Testing Image: https://sphotos-b-ord.xx.fbcdn.net/hphotos-ash3/p480x480/942876_505183632888249_1411782978_n.jpg
  87. 87. Approach: Avoid Test Automation Projects Image: http://www.hifi-writer.com/he/progscan/Files/abcestablish.jpg
  88. 88. “Greed” Lessons Learned  Ensure the reasons for automation are clear and are NOT based purely on saving money/headcount  Ensure business case for automation includes costs for ongoing maintenance Image: http://resources3.news.com.au/images/2010/12/02/1225964/296651-piggy-bank-and-coins-chart.jpg
  89. 89. Quality is Key Image: http://gs-press.com.au/images/news_articles/cache/key_hand_resized-600x0.jpg
  90. 90. Approach: Quality Assessment
  91. 91.  Tests are NOT executed on a regular basis  State of current defects is unknown  Functional testing occurs regularly  All testers can execute the functional tests  Production verification testing is used to ensure success  A majority of functional tests have been automated  Exploratory testing forms part of test execution  Customers verify implemented features prior to deployment  A majority of non-functional tests (performance, reliability, ...) are completed prior to deployment, many are automated  Testers pair with developers to automate tests  Customers verify implemented features as they are completed  Functional and non-functional testing occurs continuously within development iterations  Developers and testers are performing test first practices -1 1 2 3 4 Discussion: Test Execution
  92. 92. Wrap Up Image: http://www.pria.com.au/sb_cache/priablog/id/924/f/7%20deadly%20sins.jpg
  93. 93. Envy Flawed comparison of manual testing & automation Gluttony Over indulging on commercial test tools Lust User interface forms the basis for all testing Pride Too proud to collaborate when creating tests Sloth Too lazy to maintain automated tests Rage Frustration with slow, brittle or unreliable tests Greed Trying to cut costs through automation 7 Deadly Sins
  94. 94. Why Use Automation?
  95. 95. Image: © Fuzzy Door Productions / 20th Century Fox Television Don’t Lose Sight of the Goal
  96. 96. How geeks really workHow Geeks Can Work Together
  97. 97. Questions? Craig Smith http://www.craigsmith.id.au craig@craigsmith.id.au @smithcdau
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×