Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we measure quality of JIRA deployments to Cloud?

1,533 views

Published on

Tens thousands of customers, a few millions of users, frequent deployments and immediate feedback about bugs. This is how I would shortly describe the context of deploying JIRA Cloud releases. In Spartez, based on our partnership with Atlassian, not only we take part in the process of developing new functionalities but also we take care of the quality of JIRA Cloud releases. It is not difficult to notice that taking into account above mentioned context measuring the quality of our product is a challenge. How to manage thousands of customer tickets a year? How to handle the fact that in case of Cloud solutions we are service providers and not only product providers? How to automate the process of measuring quality at least partially? These are the kind of challenges we as Quality Assistance Engineers have to face.

During the presentation I will answer the above questions. I will present our approach to measuring quality, its advantages and disadvantages, data and experiences. I will also go beyond the specifics of our product and process. Measuring quality of any product is difficult. Very often either we give up getting valid data completely or we have metrics in which no one believes. What is even worse, sometimes although we have proper data we do not use them to improve our process and product. In the presentation I will describe our best practices to get valid measurements and how we use this information to avoid defects in the future.

Published in: Software
  • Be the first to comment

  • Be the first to like this

How we measure quality of JIRA deployments to Cloud?

  1. 1. Measuring Quality of JIRA Cloud Michał Kujałowicz, JIRA QA Team Lead michal.kujalowicz@spartez.com
  2. 2. • Involved in development of core Atlassian products • Commercial, Open-Source add-ons, customisations • Developers, Product Managers, Designers, QAs, Agile Team Leads • QA for Quality Assistance Partner
  3. 3. • 1 QA per 10 developers • Testing in hands of developers (both manual and automatic) • Teaching/coaching to become great testers Quality Assistance
  4. 4. • Process improvement • Test tools, test environments • Customer insight • Prevention, data-driven decisions • Defect analysis, continuous improvement Those other things
  5. 5. Agenda 1. Metrics’ principles 2. Context of JIRA Cloud 3. Measuring quality 4. Learning from measurements 5. Takeaways
  6. 6. Metrics’ principles
  7. 7. Software Quality Measurements Theory SW Quality Metrics
  8. 8. Software Quality is the degree to which software possesses a desired combination of attributes (e.g. reliability, interoperability) [IEEE 1061]. refers to two distinct notions: - software functional quality - software structural quality [wikipedia]
  9. 9. Quality Attributes All kinds of -ilities ISO 9126-3 ISO 25000:2005 Functionality Reliability or
  10. 10. Measurements • It’s not about counting things • It’s about estimating the value of something http://kaner.com/pdfs/SoftwareRelatedMeasurements.pdf
  11. 11. Measurements • the thing you want to measure. • the thing you use to take a measurement • what the instrument tells you when you use it to measure something • is the READING • the function that assigns a value to the attribute, based on the reading http://kaner.com/pdfs/SoftwareRelatedMeasurements.pdf ATTRIBUTE INSTRUMENT READING MEASUREMENT METRIC
  12. 12. Metrics Classification Improvement Metrics Identification Metrics • Goal: Gaining full/real picture for topic • Driven by: worrying signals, gossips, misinformation, curiosity, duty, process, standards • Goal: Change in process/product/people • Driven by: Results of Improvement metrics
  13. 13. Principles
  14. 14. Start with questions you need answers for Principles
  15. 15. Collecting metric is not the goal Principles
  16. 16. Buy in for the goal and the metric Principles
  17. 17. Metrics will not be perfect Principles
  18. 18. Drive the decisions Principles
  19. 19. Do not be afraid to throw them away Principles
  20. 20. Example - Story Rejection Rate 0% 25% 50% 75% 100% 2009 2011 2013 2013 2014 4%7% 15% 30% 100% “Quality at Speed” video: https://www.youtube.com/watch?v=yRP29wFqu20
  21. 21. Context of JIRA Cloud
  22. 22. JIRA Cloud
  23. 23. JIRA Cloud contd. • Incident management process • Regressions with the biggest impact • Thousands of issues reported in official Bug tracker per year
  24. 24. Challenges for quality
  25. 25. My story BTCYS
  26. 26. Measuring Quality
  27. 27. Start with questions you need answers for • What is the quality of our releases to JIRA Cloud? (Functional + Perceived by customers) • What bugs are we letting through? • What is their severity? • What are the root causes of those bugs? • How can we prevent them? Principles in action
  28. 28. • Goal: Being able to provide facts about quality of releases to JIRA Cloud • Goal: Being able to take further decisions Collecting metric is not the goal Principles in action Identification Metrics!
  29. 29. Metrics/Measures Number of incidents per each release Average number of Support Cases per incident for each release Metric: Metric: Incident = Every failure causing more than 4 Support Cases
  30. 30. Customer feedback Support JIRA instance Official bugtracker BUG-12345SUPPORT-7893 Customer Problem report Problem report Incident instance !!! Incident! Bug report Incident management process DEV, SOPS
  31. 31. Why? • Why incidents? • Why based on Support Cases?
  32. 32. Measurement system - part 1 Support JIRA instance Official bugtrackerIncident instance DB Get all links and store number of Support Cases for every bug
  33. 33. Measurement system - part 2 Support JIRA instance Official bugtrackerIncident instance DB Get all links and store number of Support Cases for every bug Internal JIRA instance Defect analysis scripts
  34. 34. Measurement system - part 2 Get all issues reported in bug tracker yesterday Check support cases number in DB More than 2 support cases? Create or update issue in internal JIRA Repeat process for issues created 3, 7 and 14 days ago X
  35. 35. Defect information Automatic Manual
  36. 36. Defect Analysis Dashboard
  37. 37. • Some incidents do not have Support Cases • Is not normalised against number of customers • Difficulty in driving severity imagination • Influenced by time of fixing (good or bad?) Metrics will not be perfect! Principles in action
  38. 38. Do we have problem with quality? It is less than 0.2 % of customers You cannot prevent all bugs It is significant number Even 1 Support Case is too much 0.2% reported, how many could not work?
  39. 39. Learning from measurements
  40. 40. • Improve Post Incident Review process • Create Tech Debt team • Kick off several initiatives/projects for better prevention Drive the decisions Principles in action • Improve Post Incident Review process
  41. 41. • Ticket • Severity • Document Owner • Report Status • Executive Summary • Do we know the root cause? • Has the root cause been mitigated? • Root Cause • Outage Description • Was this a repeat of a previous incident? • Affected users • Start Date/Time (UTC) • End Date/Time (UTC) • Duration • Time to Detection • Time to Recovery • What went well? PIR • What went well? • What could have gone better? • Where we got Lucky? • Priority actions to fix root cause(s) • Actions to improve service quality and/ or mitigate risk • Names of people involved • Approvers
  42. 42. PIR Improvements • Focus on prevention action items • Created in team’s backlogs • Measurable and Trackable • SLA for development teams to fix those issues
  43. 43. Action Items • Do not fix the root cause only • Try to prevent whole class of issues
  44. 44. Projects/initiatives for better prevention • Knowledge sharing tool • Better integration testing on system level • JS errors on WD tests • Test Coverage week • …
  45. 45. So what’s next?
  46. 46. Improvement metrics • % of completed action items • Number of incidents and support cases per release • Number of incidents and support cases per released change
  47. 47. Number of incidents and support cases per released change • LOC or tasks (issues)?
  48. 48. There is more to measuring quality• Customer value for new features - analytics, feature specific • Usability • Performance • Availability • Security
  49. 49. Takeaways
  50. 50. Metrics’ principles • Start with questions you need answers for • Collecting metric is not the goal • Buy in for goals and metrics • Metrics will not be perfect • Drive the decisions • Do not be afraid to throw them away
  51. 51. Measurement system • You need facts • Not easy to build but this is not an excuse • Use your opportunities • Automate
  52. 52. Learning from defects • Focus on follow-up items not descriptions • Do not fix the root cause only, try to prevent classes of issues • Stating: We need to be better in … next time - will not work
  53. 53. QA Role • Yes, it belongs to you! • Drive • Do not be afraid to question existing metrics
  54. 54. Thank you !
  55. 55. Images - credits • Żuraw in Gdańsk - by JM_GD - CC BY 2.0 • Chess - by DGlodowska - CC0 Public Domain • Land Rover Wolf XD - by Bob Bob - CC BY 2.0 • Cress keyboard - by wetwebwork - cc-by-sa-2.0 • definition - by PDPics - CC0 Public Domain • ‘Quality ... is like buying oats. signage’ by antefixus U.E. - CC BY-NC-ND 2.0 • ‘Customers Needed NO Experience Required’ - Matthew Burpee - CC BY-NC-SA 2.0 • Eintracht hooligans - by Heptagon - CC BY-SA 3.0 • ‘iceland sources’ - by Barthwo - CC0 Public Domain • Countryliner - by Arriva436 - CC BY-SA 3.0 • ‘McRae Fire. Low-Severity Fire.’ - Kaibab National Forest - by CC BY-SA 2.0 • The slug in the water - by Daniel Mietchen - CC BY-SA 2.0 • TasmanianDevil 1888 - by Mike Switzerland - CC BY-SA 3.0 • ‘Typhoid Dragon, Slain by Prevention’ - by VCU Tompkins-McCaw Library Special Collections - CC BY-NC 2.0 • Raise_your_hand_if_you_can’t_swim - by National Photo Company - CC0 1.0 • ‘Signs of the Times: If Anything is Not to your Satisfaction.... (pingnews)’ - by pingnews.com - CC BY-SA 2.0 • Crowd - by James Cridland - CC BY 2.0 • whole world in my hands - by sewingstars - CC BY-NC-ND 3.0 • ShippingContainerSFBay - by Mgunn - CC0 Public Domain • Anger Controlls Him - Jessica Flavin - CC BY 2.0 • houses - by OpenClipartVectors - CC0 Public Domain • Under Floor Cable Runs Ell - by Robert.Harker - CC BY-SA 3.0 • Angry... ? - by Navaneeth KN - CC BY 2.0

×