Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
Insert Picture Here
15 Lessons Learned from
15 Years...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.2
Program Agenda
 NetBeans > About
 15 Lessons Learn...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
Graphic Section Divider
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4
NetBeans > About
●
Integrated Development Environmen...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5
NetBeans > About
●
almost 35 000 000 downloads
●
~4 ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6
NetBeans Experience with Quality Control
You are a l...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7
●
Recurring problems with lack of time
for necessary...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8
Lesson
●
Add quality into feature set and keep that ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9
NetBeans 7.0
 quality didn't get enough
attention r...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10
●
How to set the right Release Criteria?
●
How to d...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11
Lesson
●
Listen to your customer
●
Find a metric ma...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12
NetBeans 3.0 – 7.0
●
guessed overall bugs target
Ne...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13
Notice:
 promising trend
 decreasing difference
...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14
Lack of structured view on complex data
in issue tr...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15
Lesson
Create presentation layer on raw data from y...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16
# 3 – Metrics and dashboards
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17
●
Relying only on absolute bug count
is like planni...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18
Lesson
●
Income of bugs for brand new features is e...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19
JavaScript Editor in
NetBeans 7.3:
 week #2: high ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20
●
Testing on artificial projects & use cases
●
Seri...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21
Lesson
●
Engage real users of your product during i...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22
NetCAT 7.3:
 71 tested configurations
 116 review...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23
Low quality bug reports
●
have low fix rate
●
consu...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24
Lesson
●
Encourage users to follow bug reporting gu...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25
#151578 #213130
34months
Day #1: bug reportedDay #1...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26
●
Bug reports lack important information
●
Difficul...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27
Lesson
●
Automate bugs reporting as much as possibl...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29
Manual testing:
●
not suitable for complex matrix o...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30
Lesson
●
Find the right balance between manual and ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31
NetBeans 3.5 NetBeans 7.3
Sources are compilable
Ba...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32
Typical problems of test specifications:
●
out­of­d...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33
Lesson
●
Assign each test specification its owner
●...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34
NetBeans 6.9:
 testing of some areas behind
 exec...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35
●
How to find not tested code?
●
Do we have some de...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36
Lesson
●
Evaluation of code coverage results is ver...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37
NetBeans 6.0
 measurement and evaluation
of action...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38
●
Staff is a very expensive tool
●
They are lazy to...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39
Lesson
●
Don’t reinvent the wheel
●
Evaluate existi...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40
NetBeans 3.5
 own tool but extremely expensive
 n...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.41
Destabilized codebase after merge with
major featur...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42
Lesson
●
Pre­test big changes before their integrat...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.43
# 12 – Pre-integration testing
NetBeans 3.5
 first...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.44
●
Lack of intensive communication of
development to...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.45
Lesson
●
Close communication between development an...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.46
Java Projects
 54.000 lines of code
 developed an...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.47
●
You have never enough resources,
have you?
●
So, ...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.48
Lesson
●
No! Testing of new or heavily used feature...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.49
NetBeans 6.8
 1 tester for 2 developers
NetBeans 6...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.50
●
Serious bugs found in product
shortly after its r...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.51
Lesson
●
Engage end users to decision process wheth...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.52
Community Acceptance
 two weeks long survey
 to g...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.53
Questions & Answers
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.54
●
How to overcome operational
blindness?
●
How to c...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.55
Lesson – Operational Blindness
Make the work more d...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.56
Lesson – Quality before Feature Freeze
What did not...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.57
Insert Picture Here
Thank you!
Jiří Kovalský
NetBea...
Upcoming SlideShare
Loading in...5
×

Bof4162 kovalsky

5,299

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,299
On Slideshare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
13
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Bof4162 kovalsky

  1. 1. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1 Insert Picture Here 15 Lessons Learned from 15 Years of Quality Control Jiří Kovalský NetBeans Community Manager, Oracle, Inc. Glenn Holmer Web Development Lead, Weyco Group, Inc. Thomas Kruse Java Developer, JUG Muenster
  2. 2. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.2 Program Agenda  NetBeans > About  15 Lessons Learned ● Problem ● Solution ● Proof  Q & A
  3. 3. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3 Graphic Section Divider
  4. 4. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4 NetBeans > About ● Integrated Development Environment ● Rich Client Platform ● Plugins ● Community
  5. 5. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5 NetBeans > About ● almost 35 000 000 downloads ● ~4 000 000 lines of code ● 680 code contributors ● since 1998 Number of code contributors ● last issue ID: #236132
  6. 6. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6 NetBeans Experience with Quality Control You are a lucky bug. I'm seeing that you'll be shipped with the next five releases.
  7. 7. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7 ● Recurring problems with lack of time for necessary stabilization at the end of release cycles ● Unplanned slips # 1 – Quality is feature Problem Fig 1. Preview of new java.awt.RealStuffLayout in NetBeans 8.0 GUI Builder
  8. 8. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8 Lesson ● Add quality into feature set and keep that in mind when planning the release ● Get support from developers # 1 – Quality is feature
  9. 9. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9 NetBeans 7.0  quality didn't get enough attention right from the start  release date twice moved  longest release ever (10 months) # 1 – Quality is feature July 2010: “We're smart. Let's release on 2010­12­13!” October 2010: “Well, if we send QA to Hawaii for a month we could release on 2011­02­08!” November 2010: “Hm, 2011­03­08 is a nice release date too, isn't it?” NB 4.0 NB 5.5 NB 6.1 NB 6.8 NB 7.1 0 10 20 30 40 50 releaselengthinweeks
  10. 10. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10 ● How to set the right Release Criteria? ● How to determine whether the product is ready for shipment? # 2 - Release Criteria Problem Fig 2. NetBeans engineers shortly before going on vacation
  11. 11. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11 Lesson ● Listen to your customer ● Find a metric matching satisfaction of the customer ● Base Release Criteria on the metric ● Re­evaluate Release Criteria periodically # 2 - Release Criteria
  12. 12. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12 NetBeans 3.0 – 7.0 ● guessed overall bugs target NetBeans 7.1 – 7.3 ● calculated bugs target considering: ● historical data ● expected size of changes ● planned release dateNetBeans 7.4 ● set number of bugs for 10k LOC per areas ● agreement from development # 2 - Release Criteria
  13. 13. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13 Notice:  promising trend  decreasing difference  6.9 goal changed on the fly  7.0 goal too aggressive Recommendation index: NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3 NB 7.4 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Bugs target Opened bugs NetBeans 7.0 86 % NetBeans 7.1 91 % NetBeans 7.2 94 % NetBeans 7.3 95 % # 2 - Release Criteria numberofbugsatCodeFreeze
  14. 14. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14 Lack of structured view on complex data in issue tracking system: ● bug trends ● quality status of individual areas ● actual distance from Release Criteria ● contribution of individuals # 3 – Metrics and dashboards Problem Fig 3. NetBeans managers in the NetBeans Release Control Center
  15. 15. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15 Lesson Create presentation layer on raw data from your issue tracking system and use it to ● predict progress of stabilization ● estimate whether Release Criteria is achievable # 3 – Metrics and dashboards
  16. 16. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16 # 3 – Metrics and dashboards
  17. 17. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17 ● Relying only on absolute bug count is like planning a bike trip on 2D map. If you ignore hills, you will never reach destination on time. ● Meeting Release Criteria before FCS does not mean anything! # 4 – Income vs. total bug count Problem Fig 4. NetBeans bugs desperately trying to escape attention of NetBeans testers
  18. 18. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18 Lesson ● Income of bugs for brand new features is equally important as actual bug count ● Having 10 open bugs at the end of every week however with income of 10 new bugs daily means that product is far from stable ● Release only if you get bug income under control (i.e. 50 % bug count = bug income) # 4 – Income vs. total bug count
  19. 19. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19 JavaScript Editor in NetBeans 7.3:  week #2: high income → Code Freeze to slip by 2 weeks  week #4: income below 40% → desired quality reached  week #7: income still under control (33%) 1 2 3 4 5 6 7 0 20 40 60 80 100 120 140 Total Income week # 4 – Income vs. total bug count #ofbugsinJavaScriptEditor
  20. 20. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20 ● Testing on artificial projects & use cases ● Serious problems found after release ● Not covered features and peaks ● Subjective view on new features # 5 – Engagement of end users Problem Fig 5. NetBeans developer fixing serious memory leak in High Resistance mode
  21. 21. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21 Lesson ● Engage real users of your product during its stabilization ● Ask for concrete help and provide your support ● Give appropriate acknowledgement # 5 – Engagement of end users
  22. 22. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22 NetCAT 7.3:  71 tested configurations  116 reviewed tutorials  456 bugs fixed in NetBeans 7.3 Time investment: Members Reported defects Productivity Fixed NetCAT 7.3 112 1363 12,2 43 % Community 1686 3273 1,9 32 % 147 1490 Invested hours Gained hours # 5 – Engagement of end users
  23. 23. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23 Low quality bug reports ● have low fix rate ● consume more time # 6 – Quality of bug reports Problem Fig 6. Main telescopes near Prague for communication on NetBeans mailing lists
  24. 24. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24 Lesson ● Encourage users to follow bug reporting guidelines such as: ● one problem = one bug ● meaningful summary ● detailed description ● steps to reproduce ● use attachments ● Try to automate reporting of suitable kind of bugs # 6 – Quality of bug reports
  25. 25. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25 #151578 #213130 34months Day #1: bug reportedDay #1: bug reported Day #2: not reproducible, more information required Day #1: not reproducible, more information required Day #36: silence, bug closed as invalid Day #2: exact steps to reproduce bug provided Day #1073: reopened, exact steps to reproduce provided Day #2: reproduced & reassigned Day #3: bug fixed Day #4: bug verifiedDay #1081: reproduced & bug fixed # 6 – Quality of bug reports
  26. 26. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26 ● Bug reports lack important information ● Difficult & annoying bug submission ● Subjectivity of reporters ● Duplicated bugs slow down stabilization # 7 – Automated bugs reporting Problem Fig 7. Two NetBeans engineers parking their vehicles in NetBeans garage
  27. 27. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27 Lesson ● Automate bugs reporting as much as possible ● Send all relevant information as part of the automatic report ● Automate post­processing of the reports # 7 – Automated bugs reporting
  28. 28. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 0 5 10 15 20 25 30 35 Automated exception reports All exception reports Notice:  50 % increase in exception reports since 2009 Comments:  2006 – automated exception reporting introduced  2011 – automated exception reporting further improved # 7 – Automated bugs reporting %ofexceptionreportsinalldefects
  29. 29. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29 Manual testing: ● not suitable for complex matrix of supported configurations ● boring in long­term projects Automated testing: ● expensive maintenance ● stability is never 100% # 8 – Automated vs. manual testing Problem Fig 8. Building machines in full swing trying to compile NetBeans sources
  30. 30. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30 Lesson ● Find the right balance between manual and automated testing ● NetBeans experience: ● Define basic functionality as features used almost by everyone ● Develop automated tests only for such basic functionality # 8 – Automated vs. manual testing
  31. 31. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31 NetBeans 3.5 NetBeans 7.3 Sources are compilable Basic functionality always works Length of full certification 13 weeks 5 weeks # 8 – Automated vs. manual testing NetBeans 3.6  decision to invest into test automation  basic functionality in daily builds guaranteed  full certification shortened by 8 weeks
  32. 32. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32 Typical problems of test specifications: ● out­of­date instructions ● vague or not descriptive statements ● wrong granularity/formatting ● unknown execution time # 9 – Quality of test specifications Problem Fig 9. Proposal of NetBeans 8.0 splash screen
  33. 33. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33 Lesson ● Assign each test specification its owner ● Review and update test specifications on a release basis by other volunteers ● Measure execution time regularly # 9 – Quality of test specifications
  34. 34. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34 NetBeans 6.9:  testing of some areas behind  execution time measured NetBeans 7.0:  priorities changed to cover important areas primarily  invest into test specifications NetBeans 7.1 NetBeans 7.2 NetBeans 7.3 NetBeans 7.4 0 50 100 150 200 250 300 34 51 71 129 NetCAT team NetBeans QE team Having quality test specifications pays off greatly # 9 – Quality of test specifications #oftestedconfigurations
  35. 35. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35 ● How to find not tested code? ● Do we have some dead code? # 10 – Code coverage Problem Fig 10. Close­up on broken chip responsible for outages of NetBeans 7.0 daily builds
  36. 36. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36 Lesson ● Evaluation of code coverage results is very expensive ● Use wisely or not at all ● Cheaper way for finding not tested code is to review and update test specifications by experienced users # 10 – Code coverage
  37. 37. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37 NetBeans 6.0  measurement and evaluation of actions code coverage took 5 months  results too late for NetBeans 6.1 Tools to find dead code:  FindBugs  PMD NetBeans 6.0 released NetBeans 6.1 released Actions code coverage # 10 – Code coverage
  38. 38. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38 ● Staff is a very expensive tool ● They are lazy to wake up at 2:30 am and test at least basic features in that nice cold lab! # 11 – Testing infrastructure and tools Problem Fig 11. Survival knife every NetBeans engineer is equipped with when leaving office
  39. 39. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39 Lesson ● Don’t reinvent the wheel ● Evaluate existing frameworks and tools, do a cost analysis and implement what suits your needs ● Prefer open source solutions # 11 – Testing infrastructure and tools
  40. 40. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40 NetBeans 3.5  own tool but extremely expensive  not maintainable after departure of its author NetBeans 5.0  attempt to stabilize the tool  expensive, failed NetBeans 5.5  switch to a different tool  worked for several years NetBeans 6.0  tool slowly on decline NetBeans 7.3  switch to open source tool NetBeans 7.4  satisfaction with the move lasts # 11 – Testing infrastructure and tools
  41. 41. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.41 Destabilized codebase after merge with major feature developed in parallel on a branch # 12 – Pre-integration testing Problem Fig 12. Dedication of NetBeans test engineers is often admirable
  42. 42. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42 Lesson ● Pre­test big changes before their integration to main repository # 12 – Pre-integration testing
  43. 43. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.43 # 12 – Pre-integration testing NetBeans 3.5  first attempt to modernize project infrastructure totally failed  codebase not compilable for 8 weeks NetBeans 4.0  new Ant based project system pre­integration tested  no broken builds at all NetBeans 3.5 NetBeans 4.0 0 1000 2000 3000 4000 5000 6000 Projects bugs All bugs Impact of 8 weeks of missing build on bugs income
  44. 44. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.44 ● Lack of intensive communication of development towards quality assurance ● Even relatively small undocumented change can have a big impact on quality and influence testing process # 13 – Developers proximity Problem Fig 13. NetBeans test engineer surrounded by two NetBeans developers
  45. 45. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.45 Lesson ● Close communication between development and quality assurance teams especially about: ● future plans ● expected code changes ● planned integrations into main branch ● Frequency and intensity can vary case by case however Always­Inform­QE­And­ Let­Them­Decide approach is a good start which can be reduced later. # 13 – Developers proximity
  46. 46. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.46 Java Projects  54.000 lines of code  developed and tested by colleagues in the same office  very stable functionality GlassFish support  56.000 lines of code  always developed by distant developer in different location  problems graduated during NetBeans 7.4 release # 13 – Developers proximity Java Projects GlassFish support Lines of code 54 000 56 000 Changed lines of code 17 % 15 % Income of bugs 68 121 Income of P1/P2 bugs 7 51 Normalized income of P1/P2 1.3 9.1 Income of bugs in NetBeans 7.4 (6 months)
  47. 47. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.47 ● You have never enough resources, have you? ● So, should highly exposed features be tested more intensively or do all features deserve equal attention? ● Same question applies to new vs. existing functionality # 14 – Resource balancing Problem Fig 14. Wife of NetBeans web master explaining new family model to her husband
  48. 48. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.48 Lesson ● No! Testing of new or heavily used features is a must of course. Surprised? :) ● Rule of 3 steps: ● test new or changed code ● test unchanged but highly used code ● test the rest of the code if you still have resources left or let it be # 14 – Resource balancing
  49. 49. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.49 NetBeans 6.8  1 tester for 2 developers NetBeans 6.9  1 tester for 8 developers  still tried to full test everything NetBeans 7.0  the same approach didn't work  gave up to test everything  employed community to help cover the missing resources # 14 – Resource balancing NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3 NB 7.4 0 1000 2000 3000 4000 5000 NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3 40 50 60 70 80 satisfactionwithqualityin%#ofbugsatCodeFreeze
  50. 50. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.50 ● Serious bugs found in product shortly after its release ● How to eliminate them? # 15 – Go/NoGo includes end users Problem Fig 15. NetBeans engineer eagerly completing appraisal of his manager
  51. 51. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.51 Lesson ● Engage end users to decision process whether product is ready for release or not ● Side effect: end users will then consider the product as “theirs” and help advertise it # 15 – Go/NoGo includes end users
  52. 52. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.52 Community Acceptance  two weeks long survey  to gather feedback on RC  to identify potential stoppers  to approve FCS RC – Release Candidate FCS – First Customer Ship # 15 – Go/NoGo includes end users NB 6.5 NB 6.7 NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3 0 5 10 15 20 25 Bugs nominated as stoppers in CA survey numberofpotentialstoppers
  53. 53. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.53 Questions & Answers
  54. 54. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.54 ● How to overcome operational blindness? ● How to control quality before Feature Freeze? # 16 – Bonus lessons Problem Fig 16. Incident which convinced NetBeans team to change their parking system
  55. 55. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.55 Lesson – Operational Blindness Make the work more diverse ● different types of tests ● automated testing ● tools development / maintenance ● shift responsibilities # 16 – Bonus lessons Motivate testers suitably ● give them responsibility ● reward best contributors (gifts, publicity, credit)
  56. 56. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.56 Lesson – Quality before Feature Freeze What did not work: ● milestones driven development ● 4+1 cycles # 16 – Bonus lessons What worked: ● have a good buffer in your plan ● quality (not date) driven releases 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 0 100 200 300 400 500 600 700 800 NetBeans 6.7 NetBeans 7.3 M1 M2 M3 numberofreportedbugs No effect of 4+1 development on overall quality
  57. 57. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.57 Insert Picture Here Thank you! Jiří Kovalský NetBeans Community Manager at Oracle, Inc. Glenn Holmer Web Development Lead at Weyco Group, Inc. Thomas Kruse Java Developer, JUG Muenster
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×