CompartmentalizedContinuous IntegrationDavid Neto            Devin SundaramSenior MTS            Senior MTS             Al...
THAT SPECIAL THING 2000          That special thing… 2007          p4 vs. svn 2009          Collaboration++
THREE TAKEAWAYS•  Continuous Integration is tough with a complex build•  Compartmentalize   = Classify + filter the change...
Continuous Integration
BROADCAST FEATURES / BUGFIX
FIND AND FIX DEFECTS EARLY                               Release                                dateDefect costRisk to fix...
SYSTEMS FAIL AT THE SEAMS     No substitute for end-to-end test
INTEGRATION BUILD IS YOUR PRODUCT•  Integration build = put all pieces together•  It’s what you deliver. Everything else i...
CONTINUOUS INTEGRATION•  Make an Integration Build as often as possible•  It’s the heartbeat of your project
SHAPES ALL PROCESS AND INFRASTUCTURE •  Supporting practices [Fowler]:    –    Maintain a code repository    –    Automate...
ALTERA’S SOFTWARE BUILD•  Altera makes Field Programmable Gate Arrays (FPGA)   –  Programming = Rewiring   –  3.9 billion ...
MULTI LAYER SYSTEM CHALLENGE•  Long time to build Device data•  Rapid development within and across layers   –  E.g. Roll ...
TOO MANY CHANGES à BUILD RISK•  Probability of a clean build drops quickly with number of   new changes                 P...
STALE BASELINE à COMPOUNDED RISK•  The longer you go without an integration build, the   higher the risk   –  Blind to re...
SOLUTION: COMPARTMENTALIZATION•  Must keep integration build stable•  Limit the damage to the whole by separating the part...
Compartmentalization:Previous approaches
STAGED BUILDS [Fowler]•  Full build = pipeline of smaller builds•  Most developers work with output of earlier stage•  In ...
INCREMENTAL REBUILD BOT•  Each change automatically built on latest stable base   + changes since stable base   –  Tell de...
MULTIPLE CODELINES: Strategy•    Partition active work into different codelines•    Qualify separately, with module build•...
MULTIPLE CODELINES: Variations•  Development codelines [Wingerd]•  Microsoft’s Virtual Build Labs [Maraia]•  Inside/Outsid...
MULTIPLE CODELINES: Issues•  Integration is manual   –  Requires superhero to integrate. Painful.   –  Manual implies infr...
MULTIPLE CODELINES: Verdict•    Ok if perfect modularity•    Manual•    Infrequent•    Inflexible: Can’t develop across co...
Compartmentalization:  Altera’s solution
REQUIREMENTS•  One codeline•  No client customization: Server side only•  Transparent to most users, most of the time•  Su...
GATEKEEPER STRATEGY•  Limit the amount of untested code accepted into the   integration build•  All code is guilty until p...
COMPARTMENTALIZE = CLASSIFY + FILTER       Classify         into     Gatekeepers       Domains                            ...
CLASSIFICATION: ZONES, DOMAINS•  Classify each submitted change into a domain•  Site = Dev location that makes an integrat...
GATEKEEPER RESPONSIBILITY•  Each Gatekeeper is responsible for a Domain   –  Validates Fresh changes in that Domain•  Run ...
EXAMPLE GATEKEEPER     N         Gatekeeper     N+1                                          Runs part of the             ...
OTHER GATEKEEPER: SPREAD + LIMIT RISK      N         Gatekeeper     N+1                                           In gener...
GATEKEEPER CAN RUN WHOLE BUILD           Gatekeeper     N+1                                      But responsible          ...
EXCLUSION RULE•  Should avoid “broken by construction” gatekeepers•  Rule: Each file may have fresh revisions from at   mo...
E.g. Alice (site TO) submits foo.c, foo.h                               q:SJ         q:TO           Alice changed    Gatek...
Bob (site SJ) develops update to foo.c …        Bob does not know about Alice’s change             #5?foo.c   #4          ...
Bob resolves to Alice’s change              #5    #6?foo.c   #4             q:TO   q:SJ              #2foo.h   #1   q:TO
What if we allow Bob to submit?                              q:SJ                           Gatekeeper                    ...
Exclusion Rule avoids broken-by-construction              #5     #6?                     #6    Exclusion rule detectsfoo.c...
Bob waits until Alice’s change is verified                    #6                    #6      Now Bob’s change isfoo.c   #4 ...
NOMADIC OWNERSHIP•  Exclusion Rule creates temporary “ownership” of a file   –  Delays updates destined to other domains  ...
SOMETIMES BYPASS GATEKEEPERS                   •  Rare, long build time                      –  E.g. COMBO                ...
TURN OFF GATEKEEPERS                       •  Late in release cycle                          –  Development has           ...
Mechanics
INTEGRATION STATUS TRACKING: WHAT•  For each file revision keep:   –  State:   Fresh, or Verified   –  Domain   –  User, c...
INTEGRATION STATUS TRACKING: HOW•  Integration status must always be up-to-date   –  Needed for Exclusion Rule, checked in...
USING A SECOND PERFORCE REPOSITORY            Primary P4D                        TriggersUserssubmit                      ...
CONFIGURATION•  Map developers to sites via P4 Group membership   –  E.g. p4sip.to.users, p4sip.sj.users•  Codeline policy...
LIFE CYCLE OF A CHANGE SUBMISSION (1/2) User               Primary P4D             Sister P4D p4 submit               Form...
LIFE CYCLE OF A CHANGE SUBMISSION (2/2) User                  Primary P4D                  Sister P4DSend list of       Ch...
MAKING A BUILD1.  Produce “build label”: revisions to use    –     Select: Base build label + base domains, Verified domai...
Summary
WHY DOES IT WORK?•  Climb the reliability curve•  Gatekeepers catch most breaks•  Run Gatekeepers often: usually fast•  Re...
USABILITY•  Most people never notice•  Exclusion rule can annoy•  “Build temporal mechanics”   –  Uncertain delay between ...
EFFECTIVENESS•  Integration build each day x 3 sites•  Changes from site X   –  In site X’s integration build next day   –...
OTHER USES FOR YOUR OWN METADATA?•  Can track whatever metadata you want   –  Within a “recent” time window•  Change revie...
ANOTHER TOOL FOR YOUR TOOLBOX                                                  s                                  Sta ged ...
Acknowledgements•    Rob Romano•    Jeff Da Silva•    David Karchmer•    Mun Soon Yap•    Addy Yeow•    Alan Herrmann•  An...
Thank You!
Upcoming SlideShare
Loading in …5
×

Compartmentalized Continuous Integration: Enabling Rapid, Flexible Collaboration for Complex Software Projects

946 views

Published on

Discover how to make continuous integration practical for complex projects and hear how Altera uses a set of gatekeeper builds to validate new code changes before they are accepted into the project-wide integration build and how it uses a second "sister" Perforce repository to track each change's integration status, allowing Altera to fully automate submission-time classification, selection of code for builds and validation marking. Extending the standard mainline model of software development, this lightweight scheme has processed over 175,000 changes in three years, enabling a stable but rapid evolution.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
946
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
56
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Compartmentalized Continuous Integration: Enabling Rapid, Flexible Collaboration for Complex Software Projects

  1. 1. CompartmentalizedContinuous IntegrationDavid Neto Devin SundaramSenior MTS Senior MTS Altera Corp.
  2. 2. THAT SPECIAL THING 2000 That special thing… 2007 p4 vs. svn 2009 Collaboration++
  3. 3. THREE TAKEAWAYS•  Continuous Integration is tough with a complex build•  Compartmentalize = Classify + filter the change going into your integration build•  Track your own metadata for a codeline –  With triggers and a second Perforce repository
  4. 4. Continuous Integration
  5. 5. BROADCAST FEATURES / BUGFIX
  6. 6. FIND AND FIX DEFECTS EARLY Release dateDefect costRisk to fix Time
  7. 7. SYSTEMS FAIL AT THE SEAMS No substitute for end-to-end test
  8. 8. INTEGRATION BUILD IS YOUR PRODUCT•  Integration build = put all pieces together•  It’s what you deliver. Everything else is just pretend.•  Communicate functionality across your team –  Broadcast new feature / bugfix•  Complex systems fail at the seams –  Feedback for developers
  9. 9. CONTINUOUS INTEGRATION•  Make an Integration Build as often as possible•  It’s the heartbeat of your project
  10. 10. SHAPES ALL PROCESS AND INFRASTUCTURE •  Supporting practices [Fowler]: –  Maintain a code repository –  Automate the build –  Make the build self-testing –  Commit as often as possible –  Every commit to mainline should be built – Keep the build fast –  Test in a clone of production environment –  Make it easy to get latest deliverables –  Everyone can see result of latest build –  Automate deployment
  11. 11. ALTERA’S SOFTWARE BUILD•  Altera makes Field Programmable Gate Arrays (FPGA) –  Programming = Rewiring –  3.9 billion transistors!•  Altera Complete Design Suite (ACDS) = Development tools•  ACDS Build: –  255K source files, 45GB –  ~400 developers, 5 locations worldwide –  14 hour build, multiprocessor, multiplatform –  Hundreds of source changes per day
  12. 12. MULTI LAYER SYSTEM CHALLENGE•  Long time to build Device data•  Rapid development within and across layers –  E.g. Roll out new device family –  E.g. DDR memory interface support crosses 5 layers Domain specific IP cores System integration tools Debug and analysis Low level compiler Physical models: logic, timing, power Device data
  13. 13. TOO MANY CHANGES à BUILD RISK•  Probability of a clean build drops quickly with number of new changes Probability of a clean build 1 0.9 37% 0.8 0.7 0.6 0.5 13% 99% per change 0.4 reliability 0.3 95% per change 0.2 reliability 0.1 0 1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 191 201 Number of changes•  When it breaks, hard to tell whose fault
  14. 14. STALE BASELINE à COMPOUNDED RISK•  The longer you go without an integration build, the higher the risk –  Blind to recent data, API, code•  Skip too many heartbeats à PROJECT DIES
  15. 15. SOLUTION: COMPARTMENTALIZATION•  Must keep integration build stable•  Limit the damage to the whole by separating the parts•  But how?
  16. 16. Compartmentalization:Previous approaches
  17. 17. STAGED BUILDS [Fowler]•  Full build = pipeline of smaller builds•  Most developers work with output of earlier stage•  In our case: Too slow –  Device data build = 4 hours –  Most layers built later: need device info•  Verdict: Does not solve our problem
  18. 18. INCREMENTAL REBUILD BOT•  Each change automatically built on latest stable base + changes since stable base –  Tell developer if it passed or broke the bot•  Tricky policy: –  If a new change breaks the bot: Keep or Eject?•  In our case: –  Can’t rely on perfect dependencies –  Device change à full integration build –  Apparent developer reliability improves•  Verdict: We use it, but does not solve whole problem
  19. 19. MULTIPLE CODELINES: Strategy•  Partition active work into different codelines•  Qualify separately, with module build•  Frequently integrate “main” à private•  Occasionally integrate private à “main” Private1Main Private2
  20. 20. MULTIPLE CODELINES: Variations•  Development codelines [Wingerd]•  Microsoft’s Virtual Build Labs [Maraia]•  Inside/Outside codelines, Remote development lines, Change Propagation Queues, … [Appleton et. al.]•  Virtual Codelines (one codeline + just-in-time branching) [Appleton et.al.]
  21. 21. MULTIPLE CODELINES: Issues•  Integration is manual –  Requires superhero to integrate. Painful. –  Manual implies infrequent. Delays integration. Private1Main Private2 Painful?!•  Hard / impossible to develop a change across components
  22. 22. MULTIPLE CODELINES: Verdict•  Ok if perfect modularity•  Manual•  Infrequent•  Inflexible: Can’t develop across components•  “Occasional” integration, not Continuous!“90% of SCM "process" is enforcing codeline promotion to compensate for the lack of a mainline” -- Wingerd
  23. 23. Compartmentalization: Altera’s solution
  24. 24. REQUIREMENTS•  One codeline•  No client customization: Server side only•  Transparent to most users, most of the time•  Support ad hoc cross-component development•  Automatic: Hands off operation
  25. 25. GATEKEEPER STRATEGY•  Limit the amount of untested code accepted into the integration build•  All code is guilty until proven innocent•  Integration build uses only innocent (verified) code*•  Each file revision in one of two integration states: Fresh Verified•  Upgrade from Fresh to Verified when used in a successful Gatekeeper build *Some exceptions
  26. 26. COMPARTMENTALIZE = CLASSIFY + FILTER Classify into Gatekeepers Domains Integration
  27. 27. CLASSIFICATION: ZONES, DOMAINS•  Classify each submitted change into a domain•  Site = Dev location that makes an integration build•  Zone = named set of depot paths –  One zone for each major component –  Zone can be “site specific” •  When lots of activity in that zone, and want to protect a site from bad changes from other sites•  Domains = –  Zone –  { Zone:Site | for each Site, each site-specific Zone } –  COMBO •  If a change touches files in more than one Zone
  28. 28. GATEKEEPER RESPONSIBILITY•  Each Gatekeeper is responsible for a Domain –  Validates Fresh changes in that Domain•  Run part or all of the build –  Uses Fresh revisions from its own Domain –  Verified code otherwise•  If ok, update integration state: foo.c #1 #2 #3 #4 #5 foo.c #1 #2 #3 #4 #5
  29. 29. EXAMPLE GATEKEEPER N Gatekeeper N+1 Runs part of the build, on top of previous full build. Responsible for one domain, uses verified source from two others. Integration Integration
  30. 30. OTHER GATEKEEPER: SPREAD + LIMIT RISK N Gatekeeper N+1 In general, limited amount of Fresh change going into any one build. Climb the reliability curve! Integration Integration
  31. 31. GATEKEEPER CAN RUN WHOLE BUILD Gatekeeper N+1 But responsible for just one domain. COMBO builds do this Integration
  32. 32. EXCLUSION RULE•  Should avoid “broken by construction” gatekeepers•  Rule: Each file may have fresh revisions from at most one domain –  Conflicts from: Site-specific zones; COMBO•  Allow many fresh revisions from same domain –  Enable rapid development #3 #4 #5 foo.c #1 #2 A A A #3 #4 #5 foo.c #1 #2 A A B
  33. 33. E.g. Alice (site TO) submits foo.c, foo.h q:SJ q:TO Alice changed Gatekeeper Gatekeeper param type uses uses #5 #4 #5 foo.c #4 q:TO q:TO #2 #2 foo.h #1 q:TO #1 q:TOZone “q” is site-specific TO, SJ are sites
  34. 34. Bob (site SJ) develops update to foo.c … Bob does not know about Alice’s change #5?foo.c #4 q:SJfoo.h #1
  35. 35. Bob resolves to Alice’s change #5 #6?foo.c #4 q:TO q:SJ #2foo.h #1 q:TO
  36. 36. What if we allow Bob to submit? q:SJ Gatekeeper uses #5 #6 #6foo.c #4 q:TO q:SJ q:SJ Sees only half of #2 Alice’s change!foo.h #1 q:TO #1 BROKEN BY CONSTRUCTION
  37. 37. Exclusion Rule avoids broken-by-construction #5 #6? #6 Exclusion rule detectsfoo.c #4 this conflict, q:TO q:SJ q:SJ Rejects Bob’s change #2foo.h #1 q:TO
  38. 38. Bob waits until Alice’s change is verified #6 #6 Now Bob’s change isfoo.c #4 #5 q:SJ q:TO acceptedfoo.h #1 #2
  39. 39. NOMADIC OWNERSHIP•  Exclusion Rule creates temporary “ownership” of a file –  Delays updates destined to other domains –  Especially within site-specific zones•  Sometimes annoying –  Willing to pay the price –  Better than the alternatives!•  Minimized by refactoring: break up files•  But it’s flexible and automatic –  Temporary ownership migrates according to update patterns
  40. 40. SOMETIMES BYPASS GATEKEEPERS •  Rare, long build time –  E.g. COMBO •  Site-protected, long build time •  Acceptable integration risk
  41. 41. TURN OFF GATEKEEPERS •  Late in release cycle –  Development has slowed –  Each change carefully reviewed •  Low integration risk •  Avoid annoyance of Exclusion Rule
  42. 42. Mechanics
  43. 43. INTEGRATION STATUS TRACKING: WHAT•  For each file revision keep: –  State: Fresh, or Verified –  Domain –  User, change#, depot path•  Store in log-structured control file•  But only need this for recent revisions –  Only Fresh revisions can conflict –  Each revision eventually Verified –  Need only from oldest Fresh until #head •  Purge older records
  44. 44. INTEGRATION STATUS TRACKING: HOW•  Integration status must always be up-to-date –  Needed for Exclusion Rule, checked in change-submit trigger•  Can’t just use floating labels: –  P4 triggers can’t update P4 metadata !•  Need: –  Fast atomic updates to control file –  Only need latest version –  Compact storage•  Store in a second Perforce repository! –  Filetype text+S512: Purges old contents
  45. 45. USING A SECOND PERFORCE REPOSITORY Primary P4D TriggersUserssubmit Sister P4D One control file per tracked codeline Build scripts: Code selection. Mark-as-Verified
  46. 46. CONFIGURATION•  Map developers to sites via P4 Group membership –  E.g. p4sip.to.users, p4sip.sj.users•  Codeline policy defined in “p4sip.zone” file –  Stored in root of codeline: Carried into branches –  Sites –  Named zones •  Mapping to depot paths relative to the codeline •  Which zones are site-specific –  Parse output of “p4 print”: Safe in triggers•  Triggers –  From-out, form-in: “change” forms –  Per codeline: change-submit, change-commit
  47. 47. LIFE CYCLE OF A CHANGE SUBMISSION (1/2) User Primary P4D Sister P4D p4 submit Form-out: Lookup user site, insert into change descriptionEdit change Form-in: form* Validate site in change description OkSend list of files * Can change site string: Masquerade as other site, or force COMBO
  48. 48. LIFE CYCLE OF A CHANGE SUBMISSION (2/2) User Primary P4D Sister P4DSend list of Change-submit files Assign to domain p4 print Read control file Check Exclusion Rule Send file Ok, or Error with list of conflicts contents Point of no return Change-commit Assign to domain p4 revert, sync, Edit control file edit Add revision records p4 submit Ok Submit
  49. 49. MAKING A BUILD1.  Produce “build label”: revisions to use –  Select: Base build label + base domains, Verified domains, Fresh domains –  Use “painter algorithm”: •  For each file, take latest revision in any category2.  Compile + smoketest (subset of) code3.  If not ok: –  Notify errant developer –  If integration build: Repair build, update label4.  If ok: –  Publish build label along with binary –  Mark revisions in label as Verified •  Use same revert/sync/edit/submit logic as the commit trigger
  50. 50. Summary
  51. 51. WHY DOES IT WORK?•  Climb the reliability curve•  Gatekeepers catch most breaks•  Run Gatekeepers often: usually fast•  Reduce “fog of war” in Integration build
  52. 52. USABILITY•  Most people never notice•  Exclusion rule can annoy•  “Build temporal mechanics” –  Uncertain delay between submit and getting into the build –  Changes appear out of order in different sites
  53. 53. EFFECTIVENESS•  Integration build each day x 3 sites•  Changes from site X –  In site X’s integration build next day –  In site Y’s integration build 36-48 hours later•  No performance loss•  Control file typically ~ 2K revision records•  175,000 changes in 3 years
  54. 54. OTHER USES FOR YOUR OWN METADATA?•  Can track whatever metadata you want –  Within a “recent” time window•  Change review and approval•  Nested transactions? –  “Group commit” emerges from Exclusion rule •  Alice, Andrew, Amy commit changes in TO •  SJ build sees all or none of them –  Could generalize this…?
  55. 55. ANOTHER TOOL FOR YOUR TOOLBOX s Sta ged build Unit tests
  56. 56. Acknowledgements•  Rob Romano•  Jeff Da Silva•  David Karchmer•  Mun Soon Yap•  Addy Yeow•  Alan Herrmann•  And all the developers and builders at Altera
  57. 57. Thank You!

×