Continuous Integration using Hudson and Fitnesse at Ingenuity Systems (Silicon Valley code camp 2011)


Published on

Continuous Integration using Hudson and Fitnesse

Speaker: Vasu Durgavarjhula , Jennifer Wong , Norman Boccone
Level: Intermediate | Room: 4221 | 11:15 AM Saturday

Learn about Continuous Integration (CI) and Continuous Deployment(CD) and how Ingenuity Systems moved from a traditional release process to a more agile frequent release model. In this talk we will discuss specifics and show demos on:

using Hudson as a framework for continuous integration, deployment, and build promotion
deployment and configuration management
changes we made to make our architecture more service-oriented
our automated test strategy using JUnit, FitNesse, and Selenium
migrating our build and deployment process from Ant to Maven
challenges to overcome and lessons learned in implementing a successful CI system

Published in: Business, Technology
  • Our main ci hudson server has a number of slaves and administers the process from the point of building the code to the point of deployment to Stable (ie, it covers the promotion levels dev, ci, stable, beta, rc). The same hudson server/cluster is used for all software products and services that participate in the continuous integration process.
    Are you sure you want to  Yes  No
    Your message goes here
  • So where is/are the Hudson servers? Does each quality gate server has its own, or is there a master one, moving tested software to the next quality gate? Also, the Fitness server/s?
    Are you sure you want to  Yes  No
    Your message goes here
  • @Dennis: Our Beta system is used for beta testing of new features. Sometimes the same builds of services or products might get deployed to both Beta and production systems, but when it's a customer-facing app (as opposed to a backend service), for the most part this system is used to get feedback from early access users. We then use that feedback to make final changes and polish up the features that we push to production. So, if you're talking about the actual builds, often the Beta flow path ends in Beta.
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi Martin,
    The horizontal view on slide 62 is an in-house plugin. I'm currently working on management and the developer who wrote it to get them to make it publicly available, I'll do my best to hurry up its release :)
    Are you sure you want to  Yes  No
    Your message goes here
  • 2 things.
    A/ I read where Jenkins is becoming the replacement for Hudson, by community interest and investment. So Installed it, and it literally REPLACED AND UNINSTALLED Hudson with no wizard questions or anything. Guess it *IS* the replacement, LOL!
    B/ Where does the 'Beta' flow path go? It branches out of 'Stable' and then what?
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Outline for this deck:Who we areWhat challenge we are addressing (high level)Our platform = Ingenuity Knowledge Base Content (3 slides) Ontology (1 slide)Products and Solutions Overview Research and Analysis Solutions The challenge IPA addresses IPA overview The challenge Ingenuity Answers addresses Additional Solutions eCommerce EnterpriseWhat Sets Ingenuity Apart (USPs)
  • What is the challenge Ingenuity is addressing? Speaking points:Helping researchers access detailed and high-quality knowledge and then USING it to design experiments, generate hypothesis, answer targeted questions, and analyze dataAll of Ingenuity's products and services have one common goal — to help life science researchers generate maximum value from all types of biological and chemical knowledge.Ingenuity offers a broad and flexible range of solutions that can be tailored to needs of various clients, including academic and therapeutic area researchers, computational biologists and informatics departments, and suppliers in the life sciences industry.
  • Speaking Points:What is it? [high level description]The Ingenuity Knowledge Base is the core technology of Ingenuity Systems Is the engine behind all of Ingenuity’s products – (IPA) It is a repository of biological interactions and functional annotations created from millions of individually modeled relationships between proteins, genes, complexes, cells, tissues, drugs, and diseasesThe IKB enables scientific understanding – through IPA and other services, researchers canDiscover biological relationships and functionsBuild testable hypothesesLastly, they can answer targeted biological questions – and get back relevant answersSo, what is unique about the IKB – what have we done differently - next slideInference example:For example – can I discover a molecular path connecting a compound known to treat mammary tumors to angiogenesis? Types of contextual detail:SpeciesSynonymsExperimental methodSite of post-translational modificationDirection of changeTissue contextCell line contextOriginal source
  • Speaking Points:How is our approach different? With the information we capture (publications, public knowledge, etc. - not JUST publications) we do THREE important things:We STRUCUTURE the information using the Ingenuity Ontology – statements are modeled into Findings. Findings are not just EXPERT REVIEW PROCESS We capture a lot of detail and ensure it is correct.The information we cover is COMPREHENSIVE – IKB covers and extensive body of biological and chemical info, including publications and third party sources.The information is TIMELY – weekly updates as of IPA 8 and IPA 8.5
  • Small teams empowered to release to production without management approvalBetter iteration and release planning. Learning to build and release features incrementally; show/hide features in betaFocus on Unit testing first and then integration testingStreamline the build environment to do automated builds, deployments and build promotionRefactor the code base to create more modular and service oriented code base
  • --title: --list items
  • --only one repository
  • --mention fingerprinting--all plugins in parent
  • --artifactsinhudson repo--nodpeloyment/test pohase other than junit--just build, not CI
  • --release master is the CI one--remove jenkinsrefernces
  • --cconfigurabel, difficult to maintain
  • --describe what puppet is--erb use--cofnigmgt standardized--moreenforcable
  • --standard vmconfig
  • --hudson build succesnoyl
  • --status app page --hudson
  • --service discovery mgr, router for services
  • Continuous Integration using Hudson and Fitnesse at Ingenuity Systems (Silicon Valley code camp 2011)

    1. 1. Proprietary and Confidential<br />Continuous Integration using Hudson and Fitnesse<br />Norman Boccone | Lead Engineer | twitter: @dropslowly<br />Jennifer Wong | Lead QE Engineer | twitter: @jenlwong<br />Vasu Durgavarjhula | Director of Development | twitter: @d_vasu<br />
    2. 2. Proprietary and Confidential<br />2<br />Overview<br />► Who we are ?<br />► The Challenge<br />► Tools and Processes<br />► Test Strategy<br />► Conclusion<br />
    3. 3. Ingenuity Systems <br />We are a leading provider of information and analytics solutions for life science researchers <br />Our goal is to help life science researchers get biological insights from their data quickly and reliably<br />Our products are used by thousands of researchers worldwide<br />Partial Customer List<br />Proprietary and Confidential<br />3<br />
    4. 4. ?<br />Scientists Need a Tool to Make Good Decisions From Complex Biological Data<br />Toxicology<br />Biomarkers<br />Pharmacogenomics<br />Discovery<br />Use understanding of disease mechanisms to identify and validate targets<br />Elucidate biological mechanisms of drug action and toxicity<br />Identify and prioritize novel biomarkers by understanding role in disease pathways<br />Understand mechanisms behind differential response to R(x)<br />Hard to synthesize the picture of what that data means in a broader biological context, and how the pieces work together… <br />Traditional publishing models are good for learning but not applying knowledge …<br />Enormous volume and complexity of biological and chemical data…<br />Cancer<br />Publishing and Collaboration<br />Experimental Data<br />Search<br />Proprietary and Confidential<br />4<br />
    5. 5. The Ingenuity® Knowledge Base<br /> Is the core technology behind all of Ingenuity’s products and services<br />THE INGENUITY KNOWLEDGE BASE<br />ANALYSIS<br />HYPOTHESIS GENERATION<br />VISUALIZATION<br />ECOMMERCE ENABLEMENT<br />PATHWAY, REAGENT & GENE SEARCH<br />ENABLES SCIENTIFIC UNDERSTANDING<br />Discover existing relationships and function, as well as inference of relationships that may not have been studied in a particular context yet<br />Generatetestable hypotheses, build pathways, ability to inference<br />Get answers by asking detailed and in-depth scientific questions <br />
    6. 6. The Ingenuity® Knowledge Base<br /> How is it different?<br />THE INGENUITY KNOWLEDGE BASE<br />► Structured-to capture relevant details<br /> Scientific statements are modeled into Findings using the Ingenuity Ontology<br />► Expert Review Process- for accuracy<br /> Findings go through QC process<br />► Comprehensive- leverage knowledge in one place<br /> Largest scientific knowledge base of its kind with modeled relationships between proteins, genes, complexes, cells, tissues, drugs, pathways and diseases<br />► Timely- for access to up-to-date knowledge<br /> Findings are added weekly<br />
    7. 7. Proprietary and Confidential<br />The Challenge<br />7<br />
    8. 8. Proprietary and Confidential<br />8<br />Single product<br />
    9. 9. Proprietary and Confidential<br />9<br />multiple products<br />Ingenuity Variant Analysis<br />Next Generation Sequencing Analysis Reports<br />Ingenuity Product Search<br />Reagent Portals for BD, Sigma, Thermo Fischer<br />
    10. 10. Proprietary and Confidential<br />10<br />Large monolithic code base<br />
    11. 11. Proprietary and Confidential<br />11<br />Modular Services<br />Single Sign On<br />User<br />Management<br />Views & Reports<br />User Data<br />Content<br />Service<br />iReport<br />Master Graph<br />
    12. 12. Proprietary and Confidential<br />12<br />Long release cycles<br />
    13. 13. Proprietary and Confidential<br />13<br />Faster release cycles<br />
    14. 14. Proprietary and Confidential<br />14<br />Speeding up the feedback loop<br />Developer<br />Stakeholders<br />
    15. 15. Proprietary and Confidential<br />15<br />Do all of this while supporting the existing product<br />
    16. 16. Proprietary and Confidential<br />16<br />Approach<br />Move to a Service Oriented Architecture<br />Don’t miss Jeff’s talk on Sunday 10:45 am “Learning to Love Services: Experiences from the Transition to a Service-oriented Architecture”<br />Develop a test strategy focused on automated testing<br />Build a continuous integration system<br />Adopt agile product management<br />Improve infrastructure to create and deploy services<br />Foster a Dev/Ops culture<br />
    17. 17. Proprietary and Confidential<br />17<br />Continuous Integration Workflow<br />Application<br />Bundle<br />Deploy Application<br />Run Fitnesse Tests<br />(Nightly suite)<br />Nightly Build<br />(Clover)<br />Run JUnit<br />Tests<br />publish summary<br />Fitnesse<br />Bundle<br />Deploy Fitnesse<br />publish<br />publish<br />Hudson Dashboard<br />(JUnit, Fitnesse summary, Code Coverage)<br />Fitnesse Wiki<br />(Test history, Details, Test Case Management)<br />Link<br />SVN<br />Commit<br />(Test Cases)<br />
    18. 18. Proprietary and Confidential<br />18<br />Tools and Processes<br />
    19. 19. Proprietary and Confidential<br />19<br />Service Oriented Architecture<br />
    20. 20. Proprietary and Confidential<br />20<br />SOA: Before<br />lots of modules, separate but dependent<br />easy to develop on but also easy to add circular dependencies<br />hard to keep track of purpose/value of all the modules<br />product consisted of bunches of modules put together<br />reference was typically to latest devcode<br />to ensure "latest," individual modules of a product were rebuilt for every release<br />junittests were run every time as well<br />new product version meant build everything, test everything<br />builds were slow<br />
    21. 21. Proprietary and Confidential<br />21<br />SOA: “Current”<br />components: module grouped together.  Access through api module only<br />less chance of circular dependencies<br />easier to comprehend modules<br />more flexibility to change code: just do not change api<br />components built/tested separately<br />faster builds<br />less repetitive testing<br />some shared modules still exist (they are not part of any component)<br />
    22. 22. Proprietary and Confidential<br />22<br />SOA: Conversion Notes<br />find a logical group of modules<br />Convert groupput group into hierarchical structure, hide things behind an apilayer<br />convert outside modules to use apis<br />see Jeff's talk<br />
    23. 23. Proprietary and Confidential<br />23<br />Build System <br />
    24. 24. Proprietary and Confidential<br />24<br />Build System: Before<br />in-house ant scripts<br />lots of control but it required lots of resources<br />no real industry standard<br />global scripts/property files, overrides allowed<br />offered consistency but (busy) people would override instead of follow pattern<br />scripts got complicated<br />anyone could add a new feature<br />if anyone wanted a new feature, just tell them to add it<br />inefficient additions, different code styles made scripts hard to read<br />very little dependency management<br />multiple repositories for binary files<br />
    25. 25. Proprietary and Confidential<br />25<br />Build System: “Current”<br /> maven<br />not much control, and very rigid  (a good thing, in a way)<br />steep learning curve<br />different from previous way of doing things<br />standard structure; difficult to not follow structure<br />hierarchical structure easily followed<br />versions for all dependencies well defined<br />easy to add new features (after a short burst of intense agony)<br />only one repository<br />
    26. 26. Proprietary and Confidential<br />26<br />Build System: Conversion Notes<br /> get a maven repository manager (nexus? Turn on search!)<br />add antrun plugin to publish artifacts to old system<br />choose a simple component to start, make lots of little submodules(1 module per artifact, plus assembly)<br />manually add needed dependencies to repository<br />build up company parent pom, use hierarchy<br />Put all plugin management into parent<br />Key maven plugins: buildnumber, buildhelper, release, assembly, <br />
    27. 27. Proprietary and Confidential<br />27<br />Build System: TODO<br />cleaner release plugin<br />continue learning maven<br />contribute to plugins<br />Improve internal maven FAQ<br />Look into git<br />
    28. 28. Proprietary and Confidential<br />28<br />Continuous Builds <br />
    29. 29. Proprietary and Confidential<br />29<br />Continuous Builds: Before<br />Hudson (no plugins)<br />auto build start for each module, with artifacts published<br />manual build start for products<br />shell scripts to copy artifacts to release dir<br />
    30. 30. Proprietary and Confidential<br />30<br />Continuous Builds: “Current”<br />Hudson<br />devmaster for auto build of modules/submodules, auto published to repository manager (dev area)<br />release master for components/products, auto published to repository manager (release area)<br />modules/submodulesbuilt on checkin<br />components/products built on schedule or on demand<br />process includes various tests and deployments (Jen)<br />
    31. 31. Proprietary and Confidential<br />31<br />Continuous Builds: Conversion Notes<br />use Jenkins, not Hudson (more activity for Jenkins)<br />startup new, clean servers<br />copy/create job that we want<br />Try to make Jenkins be the portal to everything <br />fingerprinting<br />Plugins we use: clover, fitnesse, findbugs, checkstyle, disk-usage, downstream build view, (custom) dashboard view, promoted builds, ssh slaves<br />
    32. 32. Proprietary and Confidential<br />32<br />Continuous Builds: TODO<br />better job rollup<br />job templates<br />more maven/hudson plugins<br />submit hudson plugin(s) to community<br />
    33. 33. Proprietary and Confidential<br />33<br />Deployment: Before<br />proprietary shell/perl scripts<br />targeted to install app only (not machine set up)<br />multiple proprietary property config scripts<br />
    34. 34. Proprietary and Confidential<br />34<br />Deployment: “Current”<br />Puppet<br />Configures machines and apps<br />Work on our machines and EC2<br />Use of .erb files everywhere for property configuration<br />
    35. 35. Proprietary and Confidential<br />35<br />Deployment: Conversion Notes<br />modify tar format<br />moved to one tomcat<br />used erb files<br />split properties into app <br />
    36. 36. Proprietary and Confidential<br />36<br />Deployment: TODO<br />standard server configuration types (e.g. “build machine”, “app machine”, …)<br />more puppet scripts<br />
    37. 37. Proprietary and Confidential<br />37<br />Visibility<br />
    38. 38. Proprietary and Confidential<br />38<br />Visibility: Before<br />cfmap (IT server diagnostic tool)<br />Hudson status page<br />FitNesse server<br />
    39. 39. Proprietary and Confidential<br />39<br />Visibility: “Current”<br />cfmap<br />Hudson status page<br />Hudson dashboard<br />extra tests built in to Hudson<br />FitNesse server<br />Service status page<br />
    40. 40. Proprietary and Confidential<br />40<br />Visibility: Conversion Notes<br />use Hudson plugins, with drill down<br />get everything to report in a standard way (junit xml)<br />
    41. 41. Proprietary and Confidential<br />41<br />Visibility: TODO<br />more widgets for the dashboard<br />Integrate outside reports into dashboard<br />cfmap<br />status page<br />sdm? (service discovery manager)<br />
    42. 42. Proprietary and Confidential<br />42<br />Test Strategy<br />
    43. 43. Proprietary and Confidential<br />43<br />Testing For CI/CD<br />Test Automation is key<br />How and what to automate<br />Mezaros<br />Cohn<br />Crispin<br />
    44. 44. Proprietary and Confidential<br />44<br />Test Challenges<br />Testability: some products are difficult to test at lower levels<br />Legacy Apps: one of our main products was a ‘Legacy App’. Tests used to look like this:<br />
    45. 45. Proprietary and Confidential<br />45<br />Test Challenges<br />varying levels of coverage<br />some of our newer components have great coverage<br />other components have lower coverage: legacy, proof of concept<br />
    46. 46. Proprietary and Confidential<br />46<br />Solutions<br />Test and code refactoring<br />Legacy Apps: Strangulation approach<br />automate new and refactored features<br />incremental work on tests: reserve time in each iteration for adding to tests<br />Change in culture: team ownership of tests and status<br />Cycle time for ipa was 2.5 weeks<br />Now for most of our services, it is 20-40 minutes<br />
    47. 47. Proprietary and Confidential<br />47<br />Test Infrastructure and Environments<br />
    48. 48. Proprietary and Confidential<br />48<br />Process: Build Promotion<br />Build Promotion through Quality Gates<br />Beta<br /><ul><li>Deployed to Beta</li></li></ul><li>Proprietary and Confidential<br />49<br />Test Environments<br />Stable Environment<br /><ul><li>Multiple nodes
    49. 49. Deployment Tests
    50. 50. System and performance tests
    51. 51. Manual/exploratory</li></ul>CI Environment<br /><ul><li>Single nodes
    52. 52. functional tests</li></ul>Service1<br />Service1<br />Service2<br />Service2<br />Service3<br />Service3<br />Service4<br />Service4<br />Service5<br />Service5<br />Service6<br />
    53. 53. Proprietary and Confidential<br />50<br />Test Types<br />
    54. 54. Proprietary and Confidential<br />51<br />Test Types: Deployment and Health<br />Each of our components has a built-in status page<br />
    55. 55. Proprietary and Confidential<br />52<br />Test Types: Deployment and Health<br />Status page<br />Reflects app status<br />Resource availability: DB Connections, Services<br />
    56. 56. Proprietary and Confidential<br />53<br />Test Types: Deployment and Health<br />Information is used for:<br />Deployment<br />Health Monitoring<br />Service Compatibility and Dependency checking<br />
    57. 57. Proprietary and Confidential<br />54<br />Test Types: FitNesse<br />FitNesse is a wiki-based web server test tool<br />Helps abstract test definition from technical implementation<br />Provides visual reporting and result history tracking<br />
    58. 58. Proprietary and Confidential<br />55<br />Test Types: FitNesse<br />The FitNesse Server <br />http://localhost:8080/FitNesse.UserGuide.TwoMinuteExamplehttp://localhost:8080/FitNesse.UserGuide.TwoMinuteExample?pageHistory<br />Wiki view Editing the wiki Test Results <br />
    59. 59. Proprietary and Confidential<br />56<br />Test Types: FitNesse<br /><ul><li> Integration with Hudson</li></ul>- Views, drilldown, configuration<br /><ul><li> Test results converted to junit format</li></ul>View of fitnesse test job in Hudson<br />
    60. 60. Proprietary and Confidential<br />57<br />Drill downs on fitnesse test results in Hudson<br />
    61. 61. Proprietary and Confidential<br />58<br />FitNessePlugin Configuration<br />Convert results to Junit format<br />
    62. 62. Proprietary and Confidential<br />59<br />Publish FitNesse Results and Coverage<br />
    63. 63. Proprietary and Confidential<br />60<br />Test Types: FitNesse<br /><ul><li>What we use it for:
    64. 64. Functional tests
    65. 65. Integration tests
    66. 66. UI Tests (com.jbergin.HtmlFixture)
    67. 67. DB Tests (dbfit)</li></ul>What we’ve done with it that is different<br />Use as execution framework for more complex tests <br />Extension of fitnesse server for data-driven tests<br />json fixture – pass in javascript<br />Execution of Selenium tests<br />
    68. 68. Proprietary and Confidential<br />61<br />Test Types: FitNesse<br />Lessons learned<br />FitNesse is good for straightforward verification of data<br />To do more, you have to get creative<br />Fixture and test ownership needs to be a shared responsibility<br />
    69. 69. Proprietary and Confidential<br />62<br />Test Types: Selenium<br />We fire off Selenium tests in two ways: via Hudson job or through FitNesse.<br /><ul><li>Test results converted to junit format for display in hudson</li></ul>Hudson job in dashboard:<br /> Hudson job view:<br />
    70. 70. Proprietary and Confidential<br />63<br />Test Types: Selenium<br />Hudson job configuration: trigger execution on Selenium grid using Ant <br />
    71. 71. Proprietary and Confidential<br />64<br />Test Types: Selenium<br />Second way we fire off Selenium tests is using a FitNesse fixture: WebTest<br />
    72. 72. Proprietary and Confidential<br />65<br />Test Types: Backward Compatibility<br />No staging environment<br />How do we know that when we release a new service version to prod that it won’t break?<br />Service version compatibility<br />Leverage Service Discover Manager (SDM) and Status pages to check for availability of services<br />Jar compatibility<br />static code analysis could detect changes in method signature but not underlying object or serialization changes<br />
    73. 73. Proprietary and Confidential<br />66<br />What’s Next: Getting to the End of the Road<br />
    74. 74. Proprietary and Confidential<br />67<br />What’s Next<br />What do we need to do to get to the end of the path to Continuous Integration and Deployment?<br />Continue to invest in standardized tools and processes<br />Maven<br />Puppet<br />Jenkins<br />Metrics and Immune Systems<br /><ul><li> Continue to improve our test coverage and practices
    75. 75. Culture:</li></ul>Continue to support, reinforce, educate<br />
    76. 76. Proprietary and Confidential<br />68<br />The Finish Line<br />Continuous deployment<br />Reduced Cycle Time<br />Improved quality<br />Predictable metrics<br />Reduced risk<br />
    77. 77. Proprietary and Confidential<br />69<br />Slides are posted here:<br />Plugs<br />Learning to Love Services: Experiences from the Transition to a Service-oriented Architecture<br />Speaker: Jeff Green 10:45 AM Sunday<br />Ingenuity is hiring:<br />Q&A<br />References<br />Todd. Test Automation Pyramid – review. Retrieved September 29, 2011 from:<br />Humble, Jez, and Farley, David. Continuous Delivery. Boston: Pearson Education Inc, 2011. Print.<br />Crispin, Lisa, and Gregory, Janet. Agile Testing: A Practical Guide for Testers and Agile Teams. Boston: Pearson Education Inc, 2011. Print.<br />