Integration Golden Rules


Published on

Integration Golden Rules

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Integration Golden Rules

  1. 1. Integration Golden Rules By Thyagarajan Krishnan Bringing Past Experience Together
  2. 2. The Age of Integration <ul><li>In classical Software Development: </li></ul><ul><ul><li>the Designer/Developer is the key to success </li></ul></ul><ul><ul><li>the Integration & Test Department was considered the ‘B’ team </li></ul></ul><ul><li>Today, software is getting large and complex, it has to be bought in from many vendors and assembled to reach the market faster and to avoid in-house development & maintenance </li></ul><ul><ul><li>Development teams are mostly modifying existing code </li></ul></ul><ul><ul><li>and integrating existing components from various sources </li></ul></ul><ul><ul><li>to make a high quality product software system </li></ul></ul><ul><li>The Integration Team holds the key to success </li></ul>
  3. 3. Successful Integration Disciplines <ul><li>Integration strategies focus on Discipline </li></ul><ul><ul><li>avoiding regression of already working functionality </li></ul></ul><ul><ul><li>coping with new releases of incoming software </li></ul></ul><ul><ul><ul><li>particularly when it has been modified locally </li></ul></ul></ul><ul><ul><li>changing working code as little as possible </li></ul></ul><ul><ul><li>solving complex problems by changing the right part, not the easiest part </li></ul></ul><ul><ul><li>Strong Configuration Management </li></ul></ul><ul><ul><ul><li>understanding exactly what has changed </li></ul></ul></ul><ul><ul><ul><li>making sure all members of the distributed team are working on the same thing (configuration) - the Mainline Baseline </li></ul></ul></ul>
  4. 4. The Mainline/Baseline Concept = Baseline , snapshot of the Mainline – “Last Known Good Configuration” New versions of components from vendors/internal dev teams Development branch A (takes in each Baseline) Development branch B (takes in each Baseline) Submissions to Mainline made at beginning of each Baseline stabilisation period Mainline/Master Stream/Branch
  5. 5. Keep the code Mainline 'sacred' <ul><li>Mainline should be at ' always ready to ship ' quality, even if functionally incomplete </li></ul><ul><ul><li>do not tolerate ‘instability' in the Mainline to be fixed later- it's always harder later </li></ul></ul><ul><ul><li>don't wait to the end to fix stability or system problems – will lead to more rush & chaos later </li></ul></ul><ul><ul><li>don't accept a culture where teams/individuals ignore problems because they are not 'theirs' - record all defects noticed </li></ul></ul><ul><li>All problems should have defects raised to track them </li></ul><ul><ul><li>informal system is ok if a formal system is not appropriate yet </li></ul></ul><ul><ul><li>don't assume that suppliers/vendors know about problems we see, and will fix them of their own accord – inform them & put a system to track them </li></ul></ul><ul><ul><li>have a team responsible for owning un-owned problems - i.e. diagnosing them and assigning ownership correctly </li></ul></ul>
  6. 6. Regular, Frequent Baselines of the Mainline <ul><li>Weekly (or Bi-Weekly) Project Baseline of the Mainline to be distributed to all project members and suppliers/vendors </li></ul><ul><ul><li>do not tolerate teams developing/testing their code against out-of-date Baselines on their local machines. </li></ul></ul><ul><ul><li>this will drive the requirement that Baseline is never allowed to drop in quality, as people will expect/need a new Baseline regularly </li></ul></ul><ul><li>'Keep the big wheel rolling' - once it stops, it's very hard to start again </li></ul><ul><ul><li>(Bi-)weekly project heartbeat monitors project health </li></ul></ul>
  7. 7. Integrate ‘Little and Often’ <ul><li>Integrate 'little and often' - </li></ul><ul><ul><li>Integrating too much at once makes quality dip so far it takes a month or so to recover </li></ul></ul><ul><ul><li>so that no-one has a recent build to work with because recent builds are worse quality than earlier ones </li></ul></ul><ul><ul><li>so teams are forced/encouraged to develop/debug against out-of-date software </li></ul></ul><ul><ul><li>so their work will not integrate well into the Mainline </li></ul></ul><ul><li>Distribute frequent Baselines of the Mainline </li></ul><ul><ul><li>Source/binary whichever is necessary </li></ul></ul>Quality Month to recover Month to recover Quality Week or 2-week to recover
  8. 8. Ever-increasing Quality of Baseline <ul><li>(Bi-)weekly Baseline MUST always increase in stability and quality, and never fall back </li></ul><ul><ul><li>no regressions can be tolerated in the sequence of Baselines </li></ul></ul><ul><ul><li>to achieve this, take in all qualified submissions on fixed schedule, then throw out submissions or correct them until it's ok again </li></ul></ul><ul><ul><li>use daily ‘smoke tests’ to check whether the quality has dropped back or not, i.e. whether new Baseline can be declared yet </li></ul></ul><ul><ul><li>at the end of the (two-)week period, the quality is then as high as the previous Baseline (or higher), but with more functionality </li></ul></ul><ul><ul><li>Best not to allow bad code into the Baseline than throw it out after smoke-test - This is “Gate-Keeping” </li></ul></ul><ul><li>Strong, clear S/W Configuration Management is essential to achieve this tracking of submissions </li></ul>
  9. 9. Integration Function vs Gate-Keeping <ul><li>Integration Function's job is to identify where a problem has arisen, and direct who should change what, to fix it </li></ul><ul><ul><li>this needs the most qualified and experienced 'top' engineers to diagnose complex scenarios quickly </li></ul></ul><ul><ul><li>and understand how the system was 'supposed' to work </li></ul></ul><ul><li>The Gate-keeping team must have the authority to reject submissions that may risk Mainline stability </li></ul><ul><ul><li>The gate-keeper must have strong backing from the overall project manager to resist pressure to force in too many changes in to the build at one time </li></ul></ul><ul><ul><li>as project end nears, the pressure to fix everything at once will become huge, </li></ul></ul><ul><ul><li>but must be resisted to avoid regression with resulting loss of weeks of time - the stability of the Baseline is ‘sacred’ </li></ul></ul>
  10. 10. Gate-keeping (continued) <ul><li>Gate keeper should check: </li></ul><ul><ul><li>evidence that submission has been tested against Last Known Good Configuration/Baseline </li></ul></ul><ul><ul><li>evidence that change has been code-reviewed by a peer engineer </li></ul></ul><ul><ul><ul><li>ALL code and changes must be peer-reviewed by at least one person other than the author </li></ul></ul></ul><ul><ul><li>consideration has to be given to dependencies (i.e. what will be affected by this change) - reject if too risky or un-coordinated </li></ul></ul>
  11. 11. Baseline vs Daily builds vs Continuous Integration Builds <ul><li>Everyone has the duty to report defects observed on the (Bi-)Weekly Baseline build </li></ul><ul><ul><li>don't report defects on non-Baseline builds - make Baselines significant, important and respected </li></ul></ul><ul><li>Daily (overnight) build should get a smoke-test run on it. </li></ul><ul><ul><li>even from very early on in the project </li></ul></ul><ul><ul><li>Plot a trend of Smoke Test Percent Run, Percent Passed, from very early stage in the project </li></ul></ul><ul><ul><li>this shows a confidence-building trend to extrapolate to end-of-project, or else shows cause for concern and corrective action </li></ul></ul><ul><ul><li>It's unacceptable that this trend ever goes down for Baselines of the Mainline </li></ul></ul><ul><li>Put a Continuous Integration System to automate the whole process </li></ul><ul><ul><li>Hudson is a free open source tool that helps with the automation </li></ul></ul><ul><ul><li>Avoid manual processes or the team will break shortly </li></ul></ul>
  12. 12. Daily Build Smoke-testing <ul><li>Daily build gets smoke-tested daily </li></ul><ul><ul><li>it’s acceptable that quality (pass rate) dips during stabilisation period </li></ul></ul><ul><ul><li>but this must be corrected before end-of-week Baseline </li></ul></ul><ul><ul><li>don't accept mid-week submissions except to solve integration problems for the upcoming Baseline </li></ul></ul><ul><ul><ul><li>no functional changes accepted - (however important) </li></ul></ul></ul><ul><li>Daily builds are private to the integration team - they are not distributed to the wider development community </li></ul><ul><ul><li>this stresses the importance of the (Bi-)Weekly Baseline </li></ul></ul><ul><ul><li>means that defects are only reported against Baselines - which always come out regularly </li></ul></ul><ul><li>Baselines to have Strong Configuration Management and project accountability </li></ul>
  13. 13. Design Authority/Ownership <ul><li>Every component in the system (including those from suppliers/vendors that your team intends to modify) must have an identified 'owner' (design authority) </li></ul><ul><ul><li>This owner is responsible for the correctness of the files, particularly all changes made to it locally </li></ul></ul><ul><ul><li>And should be responsible for merging in new versions from supplier/vendors with the branched version (containing local changes) </li></ul></ul><ul><li>This does not prohibit multiple people changing a single file, but ensures a single design/approval authority for all these changes </li></ul><ul><li>This leads to strong motivation to minimise number of incoming files/components that gets modified </li></ul><ul><ul><li>every modification creates a branched component, and needs a local ‘owner’ </li></ul></ul>
  14. 14. Minimise Branched Components <ul><li>Branching of a component should be a last resort </li></ul><ul><ul><li>Branching a component (making a locally modified version) is very tempting </li></ul></ul><ul><ul><li>the ‘fix’ seems simple for the developer to do, but creates an ongoing burden for the project </li></ul></ul><ul><ul><li>If uncontrolled, the number of branched components can quickly increase the time it takes to integrate a new release from supplier/vendor or rollout to a new country/project. </li></ul></ul><ul><li>Branching must be under project-level control </li></ul><ul><ul><li>If a component is not fit for purpose - raise a Change Request or Defect Report on the supplier/vendor </li></ul></ul><ul><ul><li>If a branch is approved as necessary, assign a clear owner </li></ul></ul><ul><ul><li>If an interim fix is made (waiting for supplier/vendor fix), branch must be eliminated as soon as supplier’s fix is available </li></ul></ul>
  15. 15. Baseline Frequency <ul><li>Weekly Baselines may seem too frequent </li></ul><ul><ul><li>have a formal release overhead that may too expensive </li></ul></ul><ul><ul><li>new team can’t always recover stability in a week </li></ul></ul><ul><ul><li>at the start, team may set frequency of three or four weeks </li></ul></ul><ul><li>Once team is confident, move to Bi-Weekly pulse </li></ul><ul><li>After functional-complete, in final phase, should move to weekly Baseline </li></ul><ul><ul><li>this makes fixes available to the whole team more quickly </li></ul></ul><ul><ul><li>should be smaller changes going in, large in number, low in risk, quicker recovery to full quality on smoke test </li></ul></ul><ul><li>Big Wheel turns more quickly as project accelerates to Ship date at Product Quality. </li></ul>
  16. 16. Key principles <ul><li>Every week a new Baseline of the Mainline </li></ul><ul><ul><li>integrate little and often - regular heartbeat </li></ul></ul><ul><ul><li>strong gate-keeping to protect Baseline from poor or risky submissions </li></ul></ul><ul><ul><li>Baselines are ‘always’ successful and better than last week </li></ul></ul><ul><ul><li>internal developers weekly re-sync their development environment with the new Baseline </li></ul></ul><ul><ul><li>external developers test their submissions against latest available Baseline before submission </li></ul></ul><ul><li>Project Pulse keeps on ticking </li></ul><ul><ul><li>Relentless progress towards ship quality </li></ul></ul>
  17. 17. <ul><li>Thank You </li></ul>