Traditionally, testing is often thought of as the "Quality" function of the software development process. One that happens after the "creative work" of the product and development team are finished. Over the past decade, Agile testing methods like Test Driven Development and technology advancements in test automation have improved team responsiveness and release cycle time. Yet while Agile Testing methods have made software development more efficient there is still a great need to make it more effective - with a tighter alignment to features and capabilities that actually matter to the users and cycle times that enable faster learning. Lean startup is a methodology for developing businesses and products, which aims to shorten product development cycles by adopting a combination of business-hypothesis-driven experimentation, iterative product releases, and validated learning. Stephen speaks on how to add the most relevant concepts of Lean Startup to your Agile Testing regime in order to guide backlog prioritization and discover new features and capabilities. The natural feedback loops of testing, especially during UAT, provide a continual opportunity to see from the users perspective. By using Lean Startup methods and making testing an integral part of the innovation process your team can make products not only that work but that matter. Speaker Profile Stephen Graves is currently the CEO of eTruVal, a startup providing property data and analytics to commercial real estate investors. Previously, he was Director of Global Business Operations at CA Technologies, where he ran multiple programs initiatives including New Product Incubation across all the business units. Prior to that, Stephen has held positions as a management consultant, a senior Business Analyst for a NYC government agency managing multi-million dollar projects and has founded two tech startups. Additionally, Stephen has earned an MBA from Edinburgh Business School and certifications as a Project Management Professional, Professional Scrum Master, ITSL and SAFeAgilist. http://www.1point21gws.com/testingsummit/2017/Agile_Testing_and_Test_Automation_Devops_QA_Newyork/

1. Copyright 2017 | All Right Reserved

2. USING AGILE TESTING TO
DRIVE PRODUCT INNOVATION
Agile Testing and Test Automation Summit 2017 | October 13, NYC

3. Copyright 2017 | All Right Reserved 3
About Me
(Stephen Graves).
Currently CEO of eTruVal, a startup providing property
data and analytics to commercial real estate investors.
Previous Director of Global Business Operations at CA
Technologies, heading up New Product Incubation.
Founded 4 startups (2 tech, 2 ecommerce)
Previous Senior Business Analyst for NYC gov.
MBA, PMP, SAFe Agilist (Scaled Agile Framework),
Professional Scrum Master, ITIL Certifications
Startup Weekend NYC Fintech Organizer

4. Copyright 2017 | All Right Reserved 4
Testing Levels.
Acceptance
System
Integration
Unit

5. Copyright 2017 | All Right Reserved 5
Testing Types.
Installation testing
Compatibility testing
Smoke and sanity testing
Regression testing
Acceptance testing
Alpha testing
Beta testing
Functional vs non-functional testing
Continuous testing
Destructive testing
Software performance testing
Usability testing
Accessibility testing
Security testing
Internationalization and localization
Development testing
A/B testing
Concurrent testing
Conformance testing
or type testing

6. Copyright 2017 | All Right Reserved 6
Traditional (Waterfall) VS Agile Testing.
▪ Whole team approach to testing
▪ Self-managed test team
▪ Iterative, incremental testing
life-cycle
▪ Inter-mixed test phases
▪ Exploratory testing is predominant
▪ Face-to-face communication with
the project team
AgileTesting
TraditionalTesting
▪ Independent test team
▪ Test team managed by a test lead
▪ V-model to handle the testing
life-cycle
▪ Sequential test phases: planning, analysis,
design, implementation, execution, reporting
▪ Test cases based execution of tests is
predominant
▪ Written communication with the project team

7. Copyright 2017 | All Right Reserved 7
Agile Testing Approach.
TDD
Test-Driven
(Test-First)
Development
BDD
Behavior Driven
Development /
Design
ATDD
Acceptance
Test-Driven
Development

8. WHAT ABOUT
AUTOMATION?

9. Copyright 2017 | All Right Reserved 9
Agile Testing Model.
Functional tests:
– Story acceptance tests
– Feature/Capability
Acceptance tests
– Enabler acceptance tests
Validate
Solution / System acceptance
tests:
– Scenario tests
– Exploratory tests
– User acceptance tests
– Alpha and beta tests
Test Continuously
System qualities tests
– Performance and load
– Security
– Other NFRs
– Enabler tests
– Unit tests
– Component tests
BUSINESS-FACING
CRITIQUINGTHESOLUTION
SUPPORTINGDEVELOPMENT
TECHNOLOGY-FACING
Manual
ToolsAutomated
Automated
& Manual
Q2 Q3
Q1 Q4
Test-First

10. Copyright 2017 | All Right Reserved 10
Traditional Product Innovation.
Stage
1
Stage
2
Stage
3
Stage
4
Stage
5
Gate
2
Gate
3
Gate
4
Gate
5
Idea
Screen
Gate
1
Scoping Build Business case Development Testing & Validation Launch
Go to LaunchGo to TestingGo to Development
Second
Screen
Driving new Products to Market

11. “Are you
building
the thing
right?”
It’s
“Are you
building the
RIGHT THING?”
Your Biggest Danger is Not

12. ENTER LEAN
STARTUP

13. Copyright 2017 | All Right Reserved 13
Build Measure Learn.
BuildLearn
Measure
Ideas
Data Code
Minimize TOTAL time
through the loop

14. Copyright 2017 | All Right Reserved 14
Focus on the Problem.
Problem
Solution
Understand WHAT is the problem and WHO you are solving it for.

15. Copyright 2017 | All Right Reserved 15
Minimum Viable Product.
Not This
This

16. Copyright 2017 | All Right Reserved 16
Validated Learning.
Data must support
hypothesis
Most important feedback
is from actual customers
and end users.

17. Copyright 2017 | All Right Reserved 17
Kanban.
Limit WIP
Event Driven
Maximize Flow

18. Copyright 2017 | All Right Reserved 18
Small Batches.

19. Copyright 2017 | All Right Reserved 19
Continuous Deployment.
Requirements Dev QA Release Requirements
Continuous
Deployment
Release
DEV
QARELEASE
COMMIT
TEST
DEPLOY
MONITOR
FROM THIS TO THIS
2 Week Release Cycle
Hundreds of lines of code
Delayed Learning
Coding Days vs Customer Days
Multiple Releases per Day
<25 lines of code
Rapid Learning
Every Day is Customer Day

20. Copyright 2017 | All Right Reserved 20
Jidoka Principle (Andon Cord).

21. Copyright 2017 | All Right Reserved 21
5Ws+H and 5Ys.
Use
5Ws+H to
Understand
the Problem
Who
What
When
Where
Why
Why?
Why?
Why?
Why?
Why?

22. Copyright 2017 | All Right Reserved 22
Actionable Metrics.
Conclusions around success or failure should be made based
on measuring real things that matter.

23. Copyright 2017 | All Right Reserved 23
Lean Startup Model.
Code Faster
Unit Tests
Usability Tests
Continuous Integration
Incremental Deployment
Free & Open-Source Components
Cloud Computing
Cluster Immune System
Just-in-time Scalability
Refactoring
Developer Sandbox
Learn Faster
Split Tests
Customer Interviews
Customer Development
Five Whys Root Cause Analysis
Customer Advisory Board
Falsifiable Hypotheses
Product Owner Accountability
Custom Archetypes
Cross-functional Teams
Smoke Tests
Measure Faster
Split Tests
Clear Product Owner
Continuous Deployment
Usability Tests
Real-time Monitoring
Custom Liaison
Funnel Analysis
Cohort Analysis
Net Promoter Score
Search Engine Marketing
Real-Time Alerting
Predictive Monitoring
BuildLearn
Measure
Ideas
Data Code

24. Copyright 2017 | All Right Reserved 24
Lean Startup Agile Testing.

25. Copyright 2017 | All Right Reserved 25
Benefits of Lean Startup in Agile Testing.
Focus on things
that matter
Customer / User
Engagement
Rapid
Feedback
Backlog
prioritization / pruning
Feature
discovery
New Horizon
1 ideas
Finding
Adjacencies
Eliminate
Bloat

26. w w w . s t e p h e n g r a v e s . u s
THANK YOU