Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Understand release engineering


Published on

Published in: Technology
  • Be the first to comment

Understand release engineering

  1. 1. Understand Release Engineering Quality Software Release Liang Gao
  2. 2. We’ll cover • What is release engineering. • Why do we need release engineering • Release version planning • Branch management • Build process • Release process • Source code control • Release engineering best practice
  3. 3. What is Release Engineering Delive r Delive r
  4. 4. Why do We need Release Engineering • Do you know how many releases Cisco IOS has?
  5. 5. Why do We need Release Engineering • Do you know how many platforms Cisco IOS runs? 1000 Series 1600 Series 1700 Series 2500 Series MC 3810 4500 Series 7000 Series 7400 Series 7500 & 7000 RSP Series 7600 OSR Series 10000 Series 12000 GSR Series Catalyst 4000 Catalyst 5000 Catalyst 6000 & 6500
  6. 6. What is Release Engineering • build a good quality binary • Source + tools = product
  7. 7. What is Release Engineering • Compile • Verifying that the software on the (uncharacterized) end user system continues to execute properly • Source code quality control • branching and merging separate but parallel codelines of development • Make it a predictable, quality release
  8. 8. Overall Picture Customer Marketing/Product Release Planning Release Engineering QC DE Customer
  9. 9. Types of Releases • Major: Introduce major functionalities, major infrastructure changes unaware by users • Minor: Introduce minor enhancements and bug fixes • Maintenance/service: Bug fixes, mostly critical bugs found by customers • Patch: Bug fixes targeted toward on customer • Platform: Release only one selected platforms, mostly introduce new hardware platforms (ASIC, new hardware revision, component changes
  10. 10. Sample Release Process IntroductionQC lead identifies an official build as release candidate Build master rebuilds the candidate with a release name using build tool Build master puts the signed image to release directory Auto Regression QC Lead to announce Exit QC Technical Writer and QC provide Release Note Store image, Go Live, Sales notify customer
  11. 11. Release Manager's question • Which release contains this defect repair? • Which releases have this defect fixed? • Which codelines contain this defect repair? • What files have changed between these two releases? • What defects have been added/dropped between this release and that release? • What changes have been added/dropped between this release and that release?
  12. 12. Introducing Branch and its Management
  13. 13. Branch Management
  14. 14. Release Management • Customers may use different versions • How many branch you need to support • XP and Vista will be co-exist.
  15. 15. Release Management1.1 1.2 1.3 1.4 2.0 2.1 1.1r1 1.1r2 1.1r3 1.1r4 1.1r5 1.1Throttle 1.2FCS 1.2r1 1.2r2 1.2r3 2.0 3.0 3.0FCS 1.2Beta Bug Fix Bug Fix Bug Fix 2 month
  16. 16. Maintain Multiple Releases 1.1 1.2 1.1r1 1.1r2 1.2r1 1.2r2 One Testing Group
  17. 17. Maintain Multiple Releases P4 – Beta3 1 week 2008-09-17 2008-09-23 P4 - Golden Week 10 days  2008-09-24  2008-10-9 Extended time period 4 days 2008-10-10 2008-10-15 P4 - FCS - Internal 5 days  2008-10-16  2008-10-22 Sanity on CTU0z0r1, all New Features need to be run. 10/16-10/17 FCS regression Suites : Security Scan flow Sanity Release job All Featuress sanity (Besides private script, please also do manual sanity on al lFeaturess) System level testing automation jobs Upgrade testing Checksum testing 10/17-10/23 Official images Confirm files are on staging srv 10/23 FCS
  18. 18. Merge Process
  19. 19. Merge Process • Get the Diff • Use CVS or Subversion to find code conflict • Get developers to result • Compile binary • Auto regression • Code commits
  20. 20. Branch Process • Copy over
  21. 21. Bug Fix Checkin Process • A field in bug report: To be fixed in • It is filled by DE manager case by case • Forward Fix – older version found bug, that will need to be fixed in the later version • Backward fix – Current version’s old code found bug, need to be fixed in the previous version as well • Both forward and backward fix • A tool can be used to auto propagate the fix
  22. 22. CM Tools For Source Code Control • CVS • Subversion • Rational Clearcase
  23. 23. Stable Branch • After a full regression, no P1 bugs, high percentage of the regression test cases passed. No many regression bugs. • Stable branch take a great effort to get. – Regression and bug filling – Bugs fix – More regression – ….
  24. 24. Stable Branch Strategy • Main branch is stable – Release branch is for new code development – Strictly code merge criteria • Release branch is stable – New branch contain most of the new code (every one can check in – Extensive testing on the release branch
  25. 25. Branch Management in Real Life
  26. 26. Some of The Best Practice in Process • New feature Code Checkin Process • Bug Fix code checkin process • Branch open and close time process • QC on Branch Control • Code Freeze • Nightly build and smoke test • Customer Patch • Testing Coverage on different release • Release Criteria Sample
  27. 27. New code check in process • Developer checkout the private branch and develop the code. • Developer drop the private branch build image for tester to test • Test file bugs and developer fix bugs on the private branch • Code check in criteria – No P1 bugs – P2 bugs no more than 2 – Code coverage 75% – Automation script is developed. – Tester has authority to veto the commit – `
  28. 28. Bug fix process. • Developer checkout a private branch for bug fix • Developer done code review • Developer provide code diff in the bug report • Developer provide the simple test to show bug is fixed • Developer run small scale regression on the core feature that might be affected by this bug fix
  29. 29. Branch Open and Close Windows Timing • For new features, open the branch (that code can be checked in) on a specific time. Accept commit only it meets the commit criteria. • If the feature missed the branch open time, it has to wait for the next round (performance impact on the developer) • Customer found bug, regression bug code fix can be checked in through out the cycle. Use nightly build and daily smoke test
  30. 30. Branch Open and Close Windows Timing Close CloseCloseOpen Open
  31. 31. QC on Branch Control • The firmware is always built by QC to ensure the source code is in repository and the quality of the firmware. Every firmware built including the source code is kept and backup for reference. • QC will not accept firmware from developer for testing. If developer request to test his/her firmware for bug fixes, QC will re- test again using QC built firmware. • The firmware build can be done by demand
  32. 32. Code Freeze • Feature Freeze • Code Freeze – When the code is ready for beta and freezed, all source code checked in require bug id, no new feature check in will be allowed and source tree is locked. The code will be reviewed by code reviewer. Once it is reviewed, the QC project lead will open the permission for the developer to check in. – Any last minutes bug fixes require reviewed by two chief architects. QC Project lead will
  33. 33. Development and Testing At a Glance MRD/PRD Functional Specification Code Development Test Plan Test Case Code Release to Test Code to Test Alpha Start Scripting Dev Test Code Control Opened Bug ID Feature Freeze Code Freeze Beta Start FCS Various Testing Code Review Bug Fixes Form Golden Week Daily/Frequent Code Drops
  34. 34. Nightly build and smoke test • The Nightly Build and Smoke Test is a process in which a software product is completely built every night and then put through a series of tests to verify its basic operations. This process is a coding-stage process, and it gets initiated at the start of the coding phase of the project. The process produces its savings by reducing the likelihood of several common, time- consuming risks – unsuccessful integration, low quality, and poor progress visibility. • The Nightly Build and Smoke Test can be used effectively on projects of virtually any size and complexity. • It is the responsibility of the Build Group to create the required environment for running the nightly builds and monitoring it on a daily basis.
  35. 35. Customer Patch • For customer patch release, a bug id is required. • Test Engineer will try to reproduce the same problem in test environment. • Developer will determine the cause and fix accordingly. • Since this is patch release, we will normally make a copy of release source directory, changed the version to identify the customer, check in only the fixes, and build.
  36. 36. Types of Testing Test Type Description New Feature Test Perform new feature tests according to MRD, RFE and functional specification. The new feature test cases will become part of regression test once the product is released. Regression Test Perform feature tests that is already available in the past releases, this is to ensure the existing working features do not break in the new release. System Test Ensure device/software works as a whole by exercising all major components in hardware and software. Interoperability Test Software interoperability ensures the implementation works with our products and other security products. Hardware interoperability ensures our hardware devices work with other hardware vendors to test such as interface link, NSRP, etc. Performance Test For NetScreen devices, measure the throughput of firewall and IPSec and ramp up rate. For Global PRO, we measure the number of devices Global PRO can support. Capacity Test Measure the maximum supported capacity listed in product specifications such as MRD/PRD/RFE or datasheet. Security Test Run through list of well-known security scanners and attack software to assure our devices are not vulnerable to these well-known attack software. Automation Test Automation does not provide new test cases, we develop automation software to facilitate manual regression test, we have more than 30% of ScreenOS regression test cases run in automation environment.
  37. 37. Types of Testing Performed Major Release Minor Release Patch Release Platform Release SFR CSP New Feature Test Full Full TBD Full Regression Test Full Partial Partial Partial Partial System Test Full Full Full Full Interoperability Test Full TBD TBD TBD Performance Test Full TBD Full TBD Capacity Test Full Full TBD Security Test Full Full Full Full Full Automation Test (partial regression) Full Full Full Full Full Full SFR – Special Feature Release CSP – Customer Specific Patch
  38. 38. Sample Release Criteria • Long run tests should be executed for 5 days without QC showstopper being found. • There are no open critical bugs characterized by data corruption or card or machine freeze. • There are no open ship stopper bugs characterized by bad customer experience such as failure to install and configure. • Less than 1 incoming critical bugs per week Incoming bug trend should be checked for decline over time.