Agile Testing
Principles and Practice
 Test manager, Program and project
manager
 11+ years’ delivery experience
 6+ years’ agile experienceMary Jiang
姜 融
About you?
 Managers, testers, ScrumMasters…?
 How long has your team used agile
principles/practices?
 Which kind of testing your team are
working on?
Group Discussion (10 minutes)
What is your biggest concern regarding agile testing?
Or
What interests you most about agile testing?
Presentation Time
After discussion every Group needs to select and present
your top 3 items
Agile Context
(Image provided by Wikimedia, reproduced here
under a Creative Commons license.)
Agile context –Manifesto
Individuals and
interactions
over processes and tools
Responding to change
over following a plan
Customer collaboration
over contract negotiation
Working software
over comprehensive documentation
Scrum Flow
Agile Testing
What is agile testing?
 A software testing practice that follows the principles
of agile software development
 Involves all members of a cross-functional agile team,
with special expertise contributed by testers
 Testing from the customer perspective as early as
possible
 Includes everything beyond unit and component level
testing
 Does not just mean testing on agile project
Agile Testing vs. Traditional Testing
Traditional testing V-Model
The pain of traditional testing
Testers’ pain from traditional testing
Is it agile testing?
Is it agile testing?
Is it agile testing?
What does agile testing mean?
Challenge
 Continuous Change
 Less documentation
 Keep everyone on the same page
 Face to customer
 More testing effort
Opportunity
 Everyone owns quality
 Earn business value earlier
 Break the boundaries
 Involve the customer
 Quick feedback
Whole team approach
 Scrum Team commitment to quality
 Scrum Team responsible for testing
 Product owner responsible for acceptance testing
 Development team responsible for all level’s testing
Agile testing mind-set
 Don’t be afraid
 Enjoy learning new skills and taking on new challenges
 Not limited in the areas you can test
 No judgment
 Results-oriented
 Collaborative
 Passionate
Ten principles for agile testers
 Provide continuous feedback
 Deliver value to the customer
 Enable “face-to-face” communication
 Have courage
 Keep it simple
 Practice continuous improvement
 Respond to change
 Self-organize
 Focus on people
 Enjoy
-Lisa Crispin and Janet Gregory
Change doesn’t come easy
 Be patient
 Let them feel pain
 Build your credibility
 Work on your own professional development
 Beware of the quality police mentality
Typical activities
for agile testing
• PROJECT INITIATION AND
PRE-PLANNING
• SPRINT N-1 AND SPRINT N
• CLEAN UP AND DELIVERY
Testing activities in
project initiation
and pre-planning
 Definition of “Done”
 Release planning
 Test planning
 Preparation for visibility
 Team commitment
“
”
Nothing is done until it is tested
FAMOUS SAYING IN AGILE
Definition of “Done”
Acceptance criteria
Agile tester through release planning
 A big picture view
 Sizing stories from the testing perspective
 Testing considerations while prioritizing
 What is in scope?
Sample big picture
Web Svc
SIEBEL SVCEIM LOAD CTI
UCM
REMEDY TOOLS
IVR
ESP
Portal
Siebel
UI
Alarm Etc
Web Interaction
Agent Ticket Application
Phone
Phone
W/S
Test
Perf
Load
Tech
UCM
Ubiquity
Portal
Web Interaction
GDC
US
Test planning
 Type of testing
 Infrastructure
 Test environments
 Test data
 Test results
Lightweight test plans
 Lightweight documents
 Test matrix
 Test spreadsheet
 A white board
 Online tools
Sample test plan
Sample test plan
Preparation for visibility
 Tracking test tasks and status
 Communicating testing results/bugs
 Release metrics
 Test status
 Traceability/coverage
 Defect status
Consideration of Bugs
 Is it a bug or is it a new feature?
 Use a Defect Tracking System?
 Which bugs need to be logged?
 When to fix bugs?
 Rules for dealing with bugs
Team commitment
 Definition of “Done”
 Ensure enough involvement
 Communication channels
 Agreement for visibility
 Collaboration in a distribution environment
Testing activities
in Sprint N - 1 and
Sprint
 Sprint planning
 Driving Requirement
 Driving development
 Collaborate with developers
 Facilitate communication
 Completing testing tasks
 Regression Tests
 Sprint Metrics
 Demo
 Retrospective
Sprint N-1
Are the requirements for Sprint N ready or
not?
What will testers do during Sprint Planning?
 Learn more details
 Considering all viewpoints
 Think about testability
 Write task cards for testing
 Ensure testing effort is considered in estimation
Samples of task cards for testing
Samples of task cards for testing
Samples of testers sprint plan
Driving requirement
 High level tests and examples
Flowchart diagram
Test/verification points
 Test driven requirements
Q&A with customer/representative
High level tests review
Sample for high level flowchart
Places Bid
Purchases
Item
Creates
Account
Searches
listings for
item
Creates an
Auction
Ships Item
Buyer
Seller
User
Samples of Verification Points
Q&A and review
 Collaboration tools to track Q&A and review
comments
 Online and offline review with
customer/business representative
Driving Development
 Start Simple
 Add complexity
 Risk analysis
 Coding and testing progress together
 Focus on one story at a time
Collaborate with developers
 Pair Testing
 “Show Me”
Facilitate communication
 Daily Stand-up
 Bridge the gap between business and
development
 Collaboration tools for distributed teams
 Understand and sacrifice a little time
Completing Testing Tasks
 Proactive - test as early as possible
 Address the testing crunch
 Anyone can do testing
 Non-functional testing
Regression Tests
 Automation will be preferred
 Keep the build “Green”
 Keep the build quick
 Building a Regression Suite
 Checking the “Big Picture”
Sprint metrics
 Measuring progress
 Burn Down Chart
 Is testing done?
 Coverage
 Unit Test
 Code
 Test coverage
 Defect Status
Samples for metrics – Test Status
Samples for metrics –
Traceability/Coverage
Category
Priority
TC Covered Defects TC NON-Covered Defects Total
# % # %
4-Business Critical 25 60.98% 16 39.02% 41
3-High 34 58.62% 24 41.38% 58
2-Intermediate 28 62.22% 17 37.78% 45
1-Low 19 76.00% 6 24.00% 25
Total 106 62.72% 63 37.28% 169
Samples for metrics – Defect Status
0
50
100
150
200
250
300
350
400
450
500
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8
Total Bugs
Total Fixed Bugs Ignore Reopen Situation(Forcast)
Incremeantal Won't Fixed Bugs
Total Fixed Bugs With Reopen Situation(Forcast)
Sprint Review (demo)
 Conduct the demo
 Listen to the customers
 Ask questions if possible
 Do not forget the “big picture”
Sprint Retrospective
 Start, stop, continue
 Identify test-related obstacles
 Focus on a few action items for the next sprint
Clean Up and
Delivery
 Planning enough time for testing
 Testing release candidate
 Testing on a staging environment
 Final non-functional testing
 Integration with external applications
 Product data conversion and load
 Installation testing
 Communication
 Support customer testing
System Testing
 Planning in advance
 Test environment(QA/testing and staging)
 Final testing including non-functional testing
 Cooperation with third-party teams
 ETL/data conversion
 Communication constantly
 Installation testing
 Defect trend
What if it’s not ready?
 Was the build broken in the last day?
 Any show stopper bugs not detected until final
functional testing?
 Any new non-functional issues found?
 Any test data problems?
 Integration issue?
User Acceptance Testing
 High level plan and scope
 Support and collaborate with customer
 Get feedback from customer
 Understand how customers view products
Deliverables
 External Customer
 Internal Customer
 Team and Organization Assets
Transition from
traditional tester
to an agile one
Know yourself
 Change your testing thinking
 Values are more important than process and
practice
 Existing testing skills are useful
 Build you own credibility
Agile Testing Quadrants
Functional Tests
Examples
Story Tests
Prototype
Simulations
Exploratory Testing
Scenarios
Usability testing
User Acceptance Testing
Alpha/Beta
Unit Tests
Component Tests
Performance & Load testing
Security Testing
“ility” testing
Application
under test
Business View
Technology View
Supportingtheteam
Critiqueproduct
Business supporting test
 Support Team
 Right tools to elicit requirements
 From big picture to details
 Driving development with business – facing testing
 Critique product
 Demonstrate applications to stakeholders to get early
feedback
 Use scenarios and workflow to test from end to end
Business supporting testers’ role
 As a bridge between business and development
 Required skills and capability
 Domain knowledge
 Solid testing skills and technical skills as supplement
 Critical and logic thinking
 Soft skills
 Personalities
 Automation skills
Ashely and Mina's story
Technical Supporting Test
 Support Team
 Unit tests
 Component Tests
 Continuous build
 Critique product
 Performance/load testing
 Security testing
 “ility” testing
Technical supporting testers’ role
 Driving development and non-functional needs
 Required skills and capability
 Solid testing and technical skills
 Architecture level understanding as preferred
 Developing and Programming skills
 Requirement and results analysis skills
Alex and Colin’s Story

Agile testing practice

  • 1.
  • 2.
     Test manager,Program and project manager  11+ years’ delivery experience  6+ years’ agile experienceMary Jiang 姜 融
  • 3.
    About you?  Managers,testers, ScrumMasters…?  How long has your team used agile principles/practices?  Which kind of testing your team are working on?
  • 4.
    Group Discussion (10minutes) What is your biggest concern regarding agile testing? Or What interests you most about agile testing?
  • 5.
    Presentation Time After discussionevery Group needs to select and present your top 3 items
  • 6.
  • 7.
    (Image provided byWikimedia, reproduced here under a Creative Commons license.)
  • 8.
    Agile context –Manifesto Individualsand interactions over processes and tools Responding to change over following a plan Customer collaboration over contract negotiation Working software over comprehensive documentation
  • 9.
  • 10.
  • 11.
    What is agiletesting?  A software testing practice that follows the principles of agile software development  Involves all members of a cross-functional agile team, with special expertise contributed by testers  Testing from the customer perspective as early as possible  Includes everything beyond unit and component level testing  Does not just mean testing on agile project
  • 12.
    Agile Testing vs.Traditional Testing
  • 13.
  • 14.
    The pain oftraditional testing
  • 15.
    Testers’ pain fromtraditional testing
  • 16.
    Is it agiletesting?
  • 17.
    Is it agiletesting?
  • 18.
    Is it agiletesting?
  • 19.
    What does agiletesting mean? Challenge  Continuous Change  Less documentation  Keep everyone on the same page  Face to customer  More testing effort Opportunity  Everyone owns quality  Earn business value earlier  Break the boundaries  Involve the customer  Quick feedback
  • 20.
    Whole team approach Scrum Team commitment to quality  Scrum Team responsible for testing  Product owner responsible for acceptance testing  Development team responsible for all level’s testing
  • 21.
    Agile testing mind-set Don’t be afraid  Enjoy learning new skills and taking on new challenges  Not limited in the areas you can test  No judgment  Results-oriented  Collaborative  Passionate
  • 22.
    Ten principles foragile testers  Provide continuous feedback  Deliver value to the customer  Enable “face-to-face” communication  Have courage  Keep it simple  Practice continuous improvement  Respond to change  Self-organize  Focus on people  Enjoy -Lisa Crispin and Janet Gregory
  • 23.
    Change doesn’t comeeasy  Be patient  Let them feel pain  Build your credibility  Work on your own professional development  Beware of the quality police mentality
  • 24.
    Typical activities for agiletesting • PROJECT INITIATION AND PRE-PLANNING • SPRINT N-1 AND SPRINT N • CLEAN UP AND DELIVERY
  • 25.
    Testing activities in projectinitiation and pre-planning  Definition of “Done”  Release planning  Test planning  Preparation for visibility  Team commitment
  • 26.
    “ ” Nothing is doneuntil it is tested FAMOUS SAYING IN AGILE Definition of “Done” Acceptance criteria
  • 27.
    Agile tester throughrelease planning  A big picture view  Sizing stories from the testing perspective  Testing considerations while prioritizing  What is in scope?
  • 28.
    Sample big picture WebSvc SIEBEL SVCEIM LOAD CTI UCM REMEDY TOOLS IVR ESP Portal Siebel UI Alarm Etc Web Interaction Agent Ticket Application Phone Phone W/S Test Perf Load Tech UCM Ubiquity Portal Web Interaction GDC US
  • 29.
    Test planning  Typeof testing  Infrastructure  Test environments  Test data  Test results
  • 30.
    Lightweight test plans Lightweight documents  Test matrix  Test spreadsheet  A white board  Online tools
  • 31.
  • 32.
  • 33.
    Preparation for visibility Tracking test tasks and status  Communicating testing results/bugs  Release metrics  Test status  Traceability/coverage  Defect status
  • 34.
    Consideration of Bugs Is it a bug or is it a new feature?  Use a Defect Tracking System?  Which bugs need to be logged?  When to fix bugs?  Rules for dealing with bugs
  • 35.
    Team commitment  Definitionof “Done”  Ensure enough involvement  Communication channels  Agreement for visibility  Collaboration in a distribution environment
  • 36.
    Testing activities in SprintN - 1 and Sprint  Sprint planning  Driving Requirement  Driving development  Collaborate with developers  Facilitate communication  Completing testing tasks  Regression Tests  Sprint Metrics  Demo  Retrospective
  • 37.
    Sprint N-1 Are therequirements for Sprint N ready or not?
  • 38.
    What will testersdo during Sprint Planning?  Learn more details  Considering all viewpoints  Think about testability  Write task cards for testing  Ensure testing effort is considered in estimation
  • 39.
    Samples of taskcards for testing
  • 40.
    Samples of taskcards for testing
  • 41.
    Samples of testerssprint plan
  • 42.
    Driving requirement  Highlevel tests and examples Flowchart diagram Test/verification points  Test driven requirements Q&A with customer/representative High level tests review
  • 43.
    Sample for highlevel flowchart Places Bid Purchases Item Creates Account Searches listings for item Creates an Auction Ships Item Buyer Seller User
  • 44.
  • 45.
    Q&A and review Collaboration tools to track Q&A and review comments  Online and offline review with customer/business representative
  • 46.
    Driving Development  StartSimple  Add complexity  Risk analysis  Coding and testing progress together  Focus on one story at a time
  • 47.
    Collaborate with developers Pair Testing  “Show Me”
  • 48.
    Facilitate communication  DailyStand-up  Bridge the gap between business and development  Collaboration tools for distributed teams  Understand and sacrifice a little time
  • 49.
    Completing Testing Tasks Proactive - test as early as possible  Address the testing crunch  Anyone can do testing  Non-functional testing
  • 50.
    Regression Tests  Automationwill be preferred  Keep the build “Green”  Keep the build quick  Building a Regression Suite  Checking the “Big Picture”
  • 51.
    Sprint metrics  Measuringprogress  Burn Down Chart  Is testing done?  Coverage  Unit Test  Code  Test coverage  Defect Status
  • 52.
    Samples for metrics– Test Status
  • 53.
    Samples for metrics– Traceability/Coverage Category Priority TC Covered Defects TC NON-Covered Defects Total # % # % 4-Business Critical 25 60.98% 16 39.02% 41 3-High 34 58.62% 24 41.38% 58 2-Intermediate 28 62.22% 17 37.78% 45 1-Low 19 76.00% 6 24.00% 25 Total 106 62.72% 63 37.28% 169
  • 54.
    Samples for metrics– Defect Status 0 50 100 150 200 250 300 350 400 450 500 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Total Bugs Total Fixed Bugs Ignore Reopen Situation(Forcast) Incremeantal Won't Fixed Bugs Total Fixed Bugs With Reopen Situation(Forcast)
  • 55.
    Sprint Review (demo) Conduct the demo  Listen to the customers  Ask questions if possible  Do not forget the “big picture”
  • 56.
    Sprint Retrospective  Start,stop, continue  Identify test-related obstacles  Focus on a few action items for the next sprint
  • 57.
    Clean Up and Delivery Planning enough time for testing  Testing release candidate  Testing on a staging environment  Final non-functional testing  Integration with external applications  Product data conversion and load  Installation testing  Communication  Support customer testing
  • 58.
    System Testing  Planningin advance  Test environment(QA/testing and staging)  Final testing including non-functional testing  Cooperation with third-party teams  ETL/data conversion  Communication constantly  Installation testing  Defect trend
  • 59.
    What if it’snot ready?  Was the build broken in the last day?  Any show stopper bugs not detected until final functional testing?  Any new non-functional issues found?  Any test data problems?  Integration issue?
  • 60.
    User Acceptance Testing High level plan and scope  Support and collaborate with customer  Get feedback from customer  Understand how customers view products
  • 61.
    Deliverables  External Customer Internal Customer  Team and Organization Assets
  • 62.
  • 63.
    Know yourself  Changeyour testing thinking  Values are more important than process and practice  Existing testing skills are useful  Build you own credibility
  • 64.
    Agile Testing Quadrants FunctionalTests Examples Story Tests Prototype Simulations Exploratory Testing Scenarios Usability testing User Acceptance Testing Alpha/Beta Unit Tests Component Tests Performance & Load testing Security Testing “ility” testing Application under test Business View Technology View Supportingtheteam Critiqueproduct
  • 65.
    Business supporting test Support Team  Right tools to elicit requirements  From big picture to details  Driving development with business – facing testing  Critique product  Demonstrate applications to stakeholders to get early feedback  Use scenarios and workflow to test from end to end
  • 66.
    Business supporting testers’role  As a bridge between business and development  Required skills and capability  Domain knowledge  Solid testing skills and technical skills as supplement  Critical and logic thinking  Soft skills  Personalities  Automation skills
  • 67.
  • 68.
    Technical Supporting Test Support Team  Unit tests  Component Tests  Continuous build  Critique product  Performance/load testing  Security testing  “ility” testing
  • 69.
    Technical supporting testers’role  Driving development and non-functional needs  Required skills and capability  Solid testing and technical skills  Architecture level understanding as preferred  Developing and Programming skills  Requirement and results analysis skills
  • 70.