Analytics that Agile Development and Testing Teams Can Use to Improve Quality and Efficiency In recent years, businesses have raised their expectations for development projects, cutting back on financial resources and demanding shorter production cycles. In order to meet the growing demands and succeed as a testing manager you need to leverage data and insights to make critical decisions about your team and strategy. However, this is no easy task. Managers today have more data than they know what to do with. How do you distinguish between what is meaningful and what is meaningless? In this webinar, Jonathan Alexander, CTO of QASymphony shared analytics that agile development and testing teams can use to improve quality and efficiency. Data presented comes from the QASymphony Labs, where the analytics team works with anonymized data gathered from hundreds of development organizations to find meaningful and useful testing metrics. By watching this on-demand webinar you will learn about testing and quality metrics specifically useful to agile teams, analytics to improve quality and testing efficiency, and processes to make test teams more agile. Don’t miss this opportunity to learn: Testing and quality metrics specifically useful to agile teams Analytics to improve quality and testing efficiency Processes to make test teams more agile

1. September 25, 2015
Introduction to QASymphony
for [INSERT COMPANY
NAME]
#AgileTransformation
Agile Transformation: People,
Process and Tools to Make
Your Transformation Successful
The audio for this webinar is delivered through your computer. There is no dial-in
number. Make sure your speakers are turned up or use a pair of headphones.
#testmetrics
Agile Testing
Analytics

2. HOW TO WATCH THIS WEBINAR
• Audio for this webinar is delivered through your computer. Make
sure your speakers are turned up or use a set of headphones.
• If your audio quality is choppy, it could be your internet connection.
• You can customize your webinar viewing experience by increasing,
decreasing or minimizing the size of the widgets on your screen.
• If you have questions, enter them in the widget on the right.
#testmetrics

3. MAXIMIZING THE PRESENTATION
#RiskBasedTesting
Click the icon in the
top right of the
presentation to
make the slides full
screen.
#testmetrics

4. • This webinar will be recorded and available on-demand
tomorrow. You will get an email when it is available.
• Join the conversation on Twitter using the hashtag
#testmetrics
• Use the Q&A widgets to ask questions during the webinar.
• At the end of the webinar, you will be asked to take a short
survey.
HOUSEKEEPING
#RiskBasedTesting#testmetrics

5. Live Demo of QASymphony Products:
Agile Testing and Tools
NEXT WEBINAR
#RiskBasedTesting
WHAT?
Join us for a 30-minue live demo of
the entire qTest Platform including,
Test Management, Test Analytics,
Exploratory Testing and more
WHY?
Learn how to easily track, manage,
organize and report on software
testing with the qTest Platform
WHEN?
January 11th, 2-2:30 PM EST
Ryan Yackel
Senior Product Engineer
QASymphony
@ryanyackattack

6. April 18-19, 2017 | qualityjam.com
Email marketing@qasymphony.com to be entered to win a free ticket.
MARK YOUR CALENDARS!
#RiskBasedTesting#TestAutomation#testmetrics

7. Test Case Management | Exploratory Testing | Test Analytics | BDD and more...
ABOUT QASYMPHONY
The #1 Agile Enterprise Testing Platform

8. POWERFUL, EASY-TO-USE TEST MANAGEMENT
• Central test repository to organize and
manage test cases quickly and efficiently
• Seamless integration with JIRA for
requirements and issue coverage
• Integrates with your CI/CD processes
through tools like Bamboo and Jenkins
• Flexible options to organize their tests and
powerful reporting to track their results
across projects
• Customization abilities allows for flexible
project deployments and integration settings
#testmetrics

9. ACTIONABLE, REAL-TIME TEST METRICS
• Your data, your way. Arrange and
display charts and graphs exactly
the way you want. Tabs let you
create multiple dashboards for
different projects
• Customized reporting. Zoom in
and dynamically adjust the charts.
Click on a chart to view and
download the data
• Visualize your data with interactive
heat maps that help quickly
identify where the issues are in the
application. Then drill down to get
more insights
#testmetrics

10. Our Presenter
Jonathan Alexander
CTO of QASymphony
@jalex206
#testmetrics

11. In This Session
Analytics - What and Why
Quality Analytics
Coverage and Risk Analytics
Velocity Analytics
Test Case Optimization
How to Get Started
#testmetrics

12. Analytics - Data Sources
Need to create links: dev tickets -> code check-ins, test cases, defects, support cases
#testmetrics
Many potential data sources

13. The Goal: Leverage Data for Improvement
● Use analytics to improve:
○ Test coverage
○ Forecasting completion dates
○ Efficiency and most effective use of resources
○ Test case quality
○ Productivity
● Think of analytics as an objective input to the planning process
#testmetrics

14. Quality Analytics
● Core:
○ Test result %s by project/release
■ separate out latest runs
○ Defect priority and status %s
● Extra:
○ Test results by day or week
○ Defect status/priority crosstab
○ Defects per test run
○ Defect leakage (found by users)
● Tips:
○ Use color-coding to identify potential issues
○ Put manual and automated results side-by-side
#testmetrics

15. Coverage Analytics
● Core:
○ Test cases by requirement
■ Latest run results
■ Breakdown by type
○ Defects by requirement
● Extra:
○ Test case complexity
○ Test time per requirement
○ Last date of test run(s)
● Tip:
○ Use data visualization to spot risks
#testmetrics

16. Velocity Analytics
● Core:
○ Requirements inflow rate
○ Test case creation rate
○ Test run rate (cases & steps)
○ % tests complete and blocked
○ Defects opened and closed
● Extra:
○ Avg. and total testing time spent
○ Mean time to test(s) created, run, passed
○ Forecast time and defects remaining
● Tip:
○ Breakdown analytics by tester
#testmetrics

17. Test Case Optimization
● Start to think of test cases like source code
● Track manual and automated test cases, exploratory scripts
● Track analytics that will help optimize your test case library
○ Days since last run
■ Archive test cases that are not used anymore
○ Flapping (# of times consecutive runs have different results)
■ Examine these tests and code/functional areas, might
indicate need to refactor one or the other
○ Percentile complexity (steps and time spent per test)
■ Refactor highly complex tests for greater efficiency and
more pinpoint understanding of results
○ Cumulative execution time
■ Automate the manual tests that are taking the most time
■ Refactor automated tests that are running the longest
#testmetrics

18. How To Do This Yourself
● Setup a Test Analytics Reporting Server
○ Use an open source or 3rd party BI tool (such as qTest Insights)
○ Keep it simple
● Identify team members that will Work on Test Analytics
○ Depending on tool(s) may need technical and “analyst”
○ Commit to a certain # of hours per week or per month
● Start with Requirements, Test Results, and Defects data
○ For most companies data size will be very manageable
○ Don’t tackle big data problems (partition data if necessary)
● Start with Quality analytics, then add Coverage, then Velocity
○ Focus on weekly project reports
● Add More Detail and More Data Sources Over Time
#testmetrics

19. Thank You for Listening!
Questions?
#testmetrics