How Nuxeo uses the open-source continuous integration server Jenkins

  • 4,041 views
Uploaded on

Learn how Nuxeo has been using Jenkins for the past five years.

Learn how Nuxeo has been using Jenkins for the past five years.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,041
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
43
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf Julien Carsique, DevOps https://qa.nuxeo.org/jenkins
  • 2. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The Nuxeo company — www.nuxeo.com ● Founded in 2000 ● ~ 50 employees (mainly France & USA) ● Open Source Enterprise Content Management ● No-charge software licensing with LGPL ● Nuxeo Connect subscription offering for support, maintenance, consulting, training and services: ● Nuxeo Studio ● Nuxeo Marketplace ● Nuxeo Cloud17/04/2012 2/25
  • 3. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The Nuxeo products Document and Case Management17/04/2012 3/25
  • 4. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The Nuxeo products Digital Assets Management17/04/2012 4/25
  • 5. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The Nuxeo products Web sites17/04/2012 5/25
  • 6. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The Nuxeo products Public facing applications, RIA and much more …17/04/2012 6/25
  • 7. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The Nuxeo products Open Source ECM platform, applications and services ● Nuxeo Platform ● Java and standards based ● Component based architecture ● Extensible and configurable ECM services ● UI building blocks and High level frameworks ● A lot of integration levels and APIs ● Testing, benchmarking and packaging tools ● Development tools (Nuxeo Studio, Nuxeo IDE) ● Ready to use pre-packaged applications17/04/2012 7/25
  • 8. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The challenge of The source code ● 200+ SubVersion, Mercurial and Git repositories ● 1M+ LoC, 500+ Maven modules ● 5 application modules and 60+ addons ● 500+ components in the Platform ● 2500+ unit tests, 2000+ Selenium and WebDriver tests ● 10+ languages (Java, XML, JavaScript, HTML, CSS, Python, Shell, SQL, Groovy, …) and tools (Maven, Ant, Shell, …) ● 30~50 active committers and contributors ● ~1000 commits/month ● 1 development branch, 3~5 maintenance branches17/04/2012 8/25
  • 9. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The challenge of The human constraints ● Follow and maintain quality ● Save developers time ● Doing more checks than they can ● Providing a safety net for their changes ● No need to build everything themselves ● Save testers time ● Automate tests in various environments ● Warn on regression ● Easy to download latest deliverables ● Save integrators time ● Work on Nuxeo development branch with more confidence ● Get guarantees on code quality17/04/2012 9/25
  • 10. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The challenge of The deployment constraints ● Multiple deployment target ● WAR, JEE or bare Java ● OS (Linux, Unix, Windows, MacOS) ● DB (PostgreSQL, Oracle, SQLServer, MySQL, …) ● BlobStore backend (FS, NAS, S3, SQLDB, ...) ● Scalability and performance ● Cluster support ● Multi-DB scale-out (several repositories and/or DB) ● Raw performances (100M docs benchmark on a single server; 200 pages/s output on complex business apps) ● Easy maintenance ● Backup, restore, upgrade ● export/import17/04/2012 10/25
  • 11. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of The beginning ● Before 2007: BuildBot ● 2007 — Nuxeo 5 switch to Java ● Trying CruiseControl, TeamCity, Continuum, … Hudson! ● Goals for Nuxeo 5.1 ● Code analysis (Quality Assurance) ● Build and unit test Nuxeo against default environment – Linux, JBoss, Sun JDK 5, H2 – Smoke test at JBoss start – Run Selenium suite ● Automate Nuxeo release process ● Automate generation and test of documentation ● Involve developers17/04/2012 11/25
  • 12. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.1 — CI architectureRELEASES and SNAPSHOTS Maven.nuxeo.org from outside RE UPLOAD RELEASES & SNAPSHOTS RE S LEA LE SE SE S AS EA L ES RE Public Internal Nexus Private Internal Nexus Nexus Preprod DOWNLOAD DOWNLOAD UPLOAD UPLOAD DOWNLOAD UPLOADPublic Jenkins Master Private Jenkins Master17/04/2012 12/25
  • 13. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.1 — CI architecture and usage ● Speed up build time ● Organize teams resources ● Ensure publicly available artifacts quality ● Isolate internal and external build chains ● Enforce rules for developers ● Early use of CI ● Think about what kind of build (Maven, Shell, Ant, …) and what must be checked for validation (unit tests, errors in logs, …)? ● Automated build: nothing specific or contextual to the slave ● Make the build self-testing ● Stay tuned, be aware of CI activity ● Never let a project in an unstable status17/04/2012 13/25
  • 14. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.1 to 5.3 — making heavy use of CI ● Release process improvement ● Automated tasks ● Nightly / On demand ● Build promotion ● More and more tests and environments ● More performance and functional tests ● Tests with Jetty, JBoss and Tomcat ● Matrix testing jobs with OS and databases ● Testing development and administration tools ● Seriously increasing number of builds ● New packages and installers (Debian, VM, Windows, Multi-OS, …) ● New applications (Android, Nuxeo CMF, Nuxeo DAM, …) ● New plugins for Firefox, IE, OpenOffice/LibreOffice ● Accumulating versions to maintain17/04/2012 14/25
  • 15. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.3 to 5.4 (2010-2011) — growing CI issues ● CI performance issues ● CPU, network and disk usage ● Slave usage strategy ● Full cycle duration ● CI usage issues ● Build waiting time ● Feedback delay ● Feedback accuracy ● Jobs dependencies17/04/2012 15/25
  • 16. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.4 (2011) — CI improvements, Jenkins analysis ● Build cycles, workflows and durations ● Use of Jenkviz (homemade tool), Dependency Graph View and (enhanced) Build Timeout plugins17/04/2012 16/25
  • 17. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.4 (2011) — CI improvements, solutions ● Work on source code ● GWT optimizations (reduce build time) ● Review testing suites, prefer use of WebDriver over Selenium ● Work on CI ● Improve slaves usage (using labels), local vs remote ● Give priority to jobs with low duration (earlier feedback) ● New short and permanent cycle ● Create independent jobs for specific purposes ● Create new views to help developers ● Add CPU and/or memory on some slaves ● Improve Nexus infrastructure (Maven build time) ● Fix jobs triggering issue17/04/2012 17/25
  • 18. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.5 — the current status17/04/2012 18/25
  • 19. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.5 — the current status ● Great coverage and good delays ● 4 kinds of CI cycles per Nuxeo version maintained – Triggered by commit (300 jobs) – Full build and tests per day (10 jobs) – Independent cycle per 3 hours (11 jobs) – Nightly build (40 jobs) ● Less than 10 minutes to 3 hours max for main feedback ● Code and applications tested on multi-DB/OS/JDK/servers ● 6 hours for the longest build cycle ● About 450 jobs on 25 servers (2 masters), 200~?K builds/day ● 6 database engines tested on 5 operating systems ● 7 applications tested in all environments ● 50K~100K tests ran including all products and versions under CI17/04/2012 19/25
  • 20. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.5 — the current status, major CI cycles Job trigger Performance tests multi-DB Commit SCM Sources Build & Packaging Functional tests multi-OS/DBMercurial/GitHub trigger checkout Unit tests 7 apps, 2 servers Various addons functional tests Full build & tests multi-JDK daily Sources Packaging multi-OS Smoke tests trigger Full build & tests multi-DB Linux checkout Full build & tests multi-DB Win 3h Sources Merge « stable » trigger Full build (no tests) Packaging all applications Functional tests checkout branch (GitHub) Permanent cycle Nightly Sources Full build (no tests) Functional tests Packaging all applications multi-DB/OS/JDK/servers, cluster Manual checkout Prepare release Batch task Performance tests (référence) Perform release Build installers (Debian, Merge «5.6-SNAPSHOT» Win, multi-OS, ...) branch (GitHub) Push release tag (GitHub) Deploy «5.6-SNAPSHOT» Test and upload installers Deploy release artifacts Maven artifacts17/04/2012 20/25
  • 21. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.5 — main Jenkins plugins used ● Required ● Subversion, Mercurial, MSBuild, GIT, GitHub, Android Emulator ● Useful for Jenkins management ● Priority Sorter, Parameterized Trigger, Retry Failed Builds, Rebuilder, Hudson Post build task, Matrix Reloaded, next-executions, Bulk Builder, Join, Promoted Builds, batch task ● Change Log History, Maven Deployment Linker, SCM Sync Configuration, Multiple SCMs, Static Analysis Utilities, Timestamper, build timeout, Hudson Setenv, java.io.tmpdir cleaner, Hudson global-build-stats, Node and Label parameter, Monitoring, SSH Slaves, Dependency Graph Viewer, Performance, Disk Usage ● Useful reports ● Checkstyle, FindBugs, Task Scanner, PMD, Warnings, Hudson Seleniumhq, Selenium HTML report, Cobertura, TextFinder, JUnit Attachments, Duplicate Code Scanner ● For end-users ● Dashboard View, Claim, Radiator View, Hudson iPhoneView, Continuous Integration game, instant-messaging, Jabber notifier, built-on-column, Compact Columns, Version Column, Email-ext, View Job Filters, Extra Columns, Simple Theme17/04/2012 21/25
  • 22. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.6 — the near future ● Better quality testing with Sonar ● Integration with Jira ● Automated merges from QA dedicated branches to the development branch ● Automated Jenkins infrastructure deployment on cloud ● Custom slaves on demand (Chef / Puppet, chroot / LXC, …) ● Work on Nuxeo Cloud & Compatible One R&D ● CloudBees ● Nuxeo CI (partial or full) migration to the cloud17/04/2012 22/25
  • 23. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf The history of Nuxeo 5.6 — the near future ● Jenkins enhancements (plugins) ● More abstraction to avoid plugins compliance ● Multiple SCM & Git ● SCM change log feeder ● Matrix jobs improvements ● Node Label & Parameterized trigger ● Dependency Graph ● Infrastructure failures (Check job prerequisites++) ● Jobs workflow management ● …17/04/2012 23/25
  • 24. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf Conclusion ● Jenkins ● Easy maintenance ● Good scalability ● Great community and plugins catalog ● Powerful extensibility ● Efficient contribution process ● Cloud capabilities ● Nuxeo ● CI & QA on Nuxeo Platform frameworks ● Stable APIs with backward and forward compatibility ● Stable applications with good performances ● Automated processes with very few manual interventions17/04/2012 24/25
  • 25. 2517/04/2012 Jenkins User Conference Paris 17 April 2012 #jenkinsconf Thank You To Our Sponsors Platinum Sponsor Gold Sponsor Silver Sponsor