Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous integration and delivery

302 views

Published on

A quick guide through the wonders of teamwork with distributed version control systems, dependency management, build automation, and continuous integration and delivery.

Published in: Software
  • This whitepaper explains how we built a continuous testing framework for one of our high value enterprise clients and the challenges we faced along with the solutions we created to overcome those challenges. http://bit.ly/2FTSWT2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Continuous integration and delivery

  1. 1. Continuous integration and delivery A quick guide through the wonders of teamwork with distributed version control systems, dependency management, build automation, and continuous integration and delivery. Danilo Pianini danilo.pianini@unibo.it Alma Mater Studiorum—Universit`a di Bologna a Cesena Paradigmi di Programmazione e Sviluppo May 12, 2017 - Cesena (Italy) Pianini (UniBo) Continuous integration and delivery May 12, 2017 1 / 117
  2. 2. Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 2 / 117
  3. 3. Introduction Why continuous? I Avoid the integration hell Work in parallel Don’t waste developers’ time with repetitive tasks Don’t break stuff Time is money Software development used to take several months for “integrating” a couple of years of development [Fow] Historically introduced by the extreme programming (XP) community Today used by companies that do not adopt XP IMVU [teab] delivers its software up to 50 times per day Google and Mozilla release at least once a day Pianini (UniBo) Continuous integration and delivery May 12, 2017 3 / 117
  4. 4. Introduction Why continuous? II Higher frequency, lower risk [Fab] Pianini (UniBo) Continuous integration and delivery May 12, 2017 4 / 117
  5. 5. Introduction Highway to hell Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 5 / 117
  6. 6. Introduction Highway to hell What could possibly go wrong? Welcome to Hell I Jimmy [teaa] and Elliot develop a software project together. Diverging histories Jimmy must develop a new feature, picks the latest Elliot’s code Jimmy begins, and in the meantime Elliot goes on with the rest Jimmy completes his part Jimmy’s code does not work the latest version Keep your code in sync with the main development line Pianini (UniBo) Continuous integration and delivery May 12, 2017 5 / 117
  7. 7. Introduction Highway to hell What could possibly go wrong? Welcome to Hell II It was working two minutes ago Jimmy develops a new feature and tests it Jimmy shows the feature five minutes later to Elliot, it does not work anymore Jimmy can’t figure out what happened, and needs to spend time debugging Elliot can’t see the feature, and must wait for Jimmy to iron the problem out Make sure you can restore any small change you made Pianini (UniBo) Continuous integration and delivery May 12, 2017 6 / 117
  8. 8. Introduction Highway to hell What could possibly go wrong? Welcome to Hell III Stylish hell Jimmy and Elliot styles are misaligned CRLF vs. LF Egyptian parentheses [teaa] vs. Allman Tabs vs. spaces Basically anything you can start a religion war upon Jimmy and Elliot IDEs are configured differently The project continuously flips from one style to the other, inconsistently Becomes impossible to localize changes Agree on style (possibly using existing shared conventions), and enforce its adoption Pianini (UniBo) Continuous integration and delivery May 12, 2017 7 / 117
  9. 9. Introduction Highway to hell What could possibly go wrong? Welcome to Hell IV Tests as decorations Jimmy prepared a lot of nice tests Jimmy rarely executes them Jimmy code ends up being broken Execute your tests for every change you make Works on my PC Tests run correctly on Jimmy computer They don’t on Elliot’s Time gets wasted to debug the tests to decide whether or not the code should get debugged Execute your tests in a fresh environment resembling the production environment Pianini (UniBo) Continuous integration and delivery May 12, 2017 8 / 117
  10. 10. Introduction Highway to hell What could possibly go wrong? Welcome to Hell V Unexpected merge failure Jimmy develops a new feature, everything seems to work The feature gets merged with the mainline Tests fail Test the integration before performing it Pianini (UniBo) Continuous integration and delivery May 12, 2017 9 / 117
  11. 11. Introduction Highway to hell What could possibly go wrong? Welcome to Hell VI Dependency hell Jimmy needs a library Imports it directly in the code base This dependency now must be dealt with manually: Updates Bugfixes Dependency’s dependencies Big projects can very easily require hundreds of libraries Automatize the dependency management Pianini (UniBo) Continuous integration and delivery May 12, 2017 10 / 117
  12. 12. Introduction Highway to hell What could possibly go wrong? Welcome to Hell VII Fermented build The project builds correctly and passes al tests Nobody touches it, nobody builds it again for weeks Once a rebuild is required, the project no longer builds or passes tests Rebuild often Pianini (UniBo) Continuous integration and delivery May 12, 2017 11 / 117
  13. 13. Introduction Highway to hell What could possibly get improved? Road to paradise I Protoduction [teaa] When prototype code ends up in production Classically used with a negative meaning It’s time to rehabilitate it Make it easy to access and use the latest prototype Pianini (UniBo) Continuous integration and delivery May 12, 2017 12 / 117
  14. 14. Introduction Highway to hell What could possibly get improved? Road to paradise II It’s compiling [Mun] Make the building process fast! Pianini (UniBo) Continuous integration and delivery May 12, 2017 13 / 117
  15. 15. Introduction Highway to hell What could possibly get improved? Road to paradise III It’s delivering Business Analysis often needs the latest version of the software to demonstrate upcoming features Jimmy’s time gets is spent several times per week just for preparing alpha releases Automatize the delivery Pianini (UniBo) Continuous integration and delivery May 12, 2017 14 / 117
  16. 16. Introduction Stairway to heaven Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 15 / 117
  17. 17. Introduction Stairway to heaven Summary Keep feature code in sync with the mainline Make sure you can restore any small change you make Agree and enforce a style (and report problems) Execute all tests for every change Execute your tests on a freshly prepared machine resembling production Test the integration before integrating Automatize the dependency management Rebuild often Automatize the delivery Make the development artifacts available All the above, make it fast (non functional!) Pianini (UniBo) Continuous integration and delivery May 12, 2017 15 / 117
  18. 18. Introduction Stairway to heaven No silver bullet No single magic solution to all these problems is available But they can all be tackled with the right combination of: Development tools Development methodologies Automatization tools Pianini (UniBo) Continuous integration and delivery May 12, 2017 16 / 117
  19. 19. Introduction Stairway to heaven Our bullets I git A distributed version control system. Will help with: Keep feature code in sync with the mainline Make sure you can restore any small change you make All the above, make it fast! Pianini (UniBo) Continuous integration and delivery May 12, 2017 17 / 117
  20. 20. Introduction Stairway to heaven Our bullets II git-flow A methodology for team development using git. Will help with: Keep feature code in sync with the mainline Test the integration before integrating Pianini (UniBo) Continuous integration and delivery May 12, 2017 18 / 117
  21. 21. Introduction Stairway to heaven Our bullets III GitHub A git repository hosting with support for forking, pull requesting and code reviewing. Will help with: Keep feature code in sync with the mainline Test the integration before integrating Make the development artifacts available Pianini (UniBo) Continuous integration and delivery May 12, 2017 19 / 117
  22. 22. Introduction Stairway to heaven Our bullets IV PMD, CPD, FindBugs, Checkstyle, Scalastyle, Scalameta, Scapegoat... Code quality checkers. Will help with: Agree and enforce a style (and report problems) Pianini (UniBo) Continuous integration and delivery May 12, 2017 20 / 117
  23. 23. Introduction Stairway to heaven Our bullets V Gradle Dependency manager and build automation tool. Will help with: Agree and enforce a style (and report problems) Execute all tests for every change Automatize the delivery All the above, make it fast! Pianini (UniBo) Continuous integration and delivery May 12, 2017 21 / 117
  24. 24. Introduction Stairway to heaven Our bullets VI Travis CI (+ Docker) Continuous integration and delivery system (+ software container). Will help with: Execute all tests for every change Execute your tests on a freshly prepared machine resembling production Test the integration before integrating Rebuild often Automatize the delivery All the above, make it fast! Pianini (UniBo) Continuous integration and delivery May 12, 2017 22 / 117
  25. 25. Introduction Stairway to heaven Our bullets VII Maven Central, Bintray, Surge.sh, Heroku, Azure, S3... Software Distributors. Will help with: Make the development artifacts available Pianini (UniBo) Continuous integration and delivery May 12, 2017 23 / 117
  26. 26. Introduction Stairway to heaven Spoiler: the complete process prom ote clone notify release w eb hook review deploy fork pull request pushpull ... cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 24 / 117
  27. 27. Successful teamwork with distributed control version systems git Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 25 / 117
  28. 28. Successful teamwork with distributed control version systems git How will the world look like shortly prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 25 / 117
  29. 29. Successful teamwork with distributed control version systems git Bits of history In April 2005, BitKeeper, the SCM Linux was developed with, withdrawn the free (as in beer) use No other SCM met the requirements of Torvalds Performance was the real issue with such a code base Torvalds decided to write his own The project was successful, and Torvalds appointed maintenance to Hamano Why the name I’m an egotistical bastard, and I name all my projects after myself. First ’Linux’, now ’git’. a — Linus Torvalds a From the project Wiki. “git” is slang for “pig headed, think they are always correct, argumentative” Pianini (UniBo) Continuous integration and delivery May 12, 2017 26 / 117
  30. 30. Successful teamwork with distributed control version systems git The git README.md file GIT - the stupid content tracker "git" can mean anything, depending on your mood. - random three-letter combination that is pronounceable, and not actually used by any common UNIX command. The fact that it is a mispronounciation of "get" may or may not be relevant. - stupid. contemptible and despicable. simple. Take your pick from the dictionary of slang. - "global information tracker": you're in a good mood, and it actually works for you. Angels sing, and a light suddenly fills the room. - "goddamn idiotic truckload of sh*t": when it breaks Pianini (UniBo) Continuous integration and delivery May 12, 2017 27 / 117
  31. 31. Successful teamwork with distributed control version systems git Features UNIX-oriented: tracks stuff like UNIX file permits. Distributed development (the whole development history is copied locally) Diff-based history tracking Implicit file naming (history preserved on renames) Very strong support to non-linear development Written in C Approximately 10 times faster than Mercurial, 100 times faster than other DVCS (e.g. Bazaar) Uses existing protocols (ssh, http, ftp, rsync...) Pluggable merge strategies (defaults to recursive three-ways merge or octopus for 3+ heads) Pianini (UniBo) Continuous integration and delivery May 12, 2017 28 / 117
  32. 32. Successful teamwork with distributed control version systems git The compact guide to git I Sub-commands in typewritertext, concepts in italic. init Initializes a new repository Stage The place where git stores the files whose changes will be saved next add / reset Moves/Removes files onto/from the stage Pianini (UniBo) Continuous integration and delivery May 12, 2017 29 / 117
  33. 33. Successful teamwork with distributed control version systems git The compact guide to git II .gitignore A file listing the pathspecs that git should ignore even if added commit Create a new “snapshot” (it’s actually a changeset) with the contents of the stage tag Associates a symbolic name and message to a commit. HEAD The pointer to the current changeset (commits) Pianini (UniBo) Continuous integration and delivery May 12, 2017 30 / 117
  34. 34. Successful teamwork with distributed control version systems git The compact guide to git III Branch A named development line. master The name of the default development line. Pianini (UniBo) Continuous integration and delivery May 12, 2017 31 / 117
  35. 35. Successful teamwork with distributed control version systems git The compact guide to git IV checkout Moves the HEAD across commits. Can be used to switch branches or create new ones (checkout -b). If the commit is not the latest (in the current branch), git switches to detached head mode. branch Lists the existing branches. Can delete them (branch -d). merge Unifies a target branch with the current branch, merging the changes and creating a new commit. The merging algorithm is configurable. In case of conflicts, they must be manually solved. Pianini (UniBo) Continuous integration and delivery May 12, 2017 32 / 117
  36. 36. Successful teamwork with distributed control version systems git The compact guide to git V fast forward The operation executed by default when a merge is requested and a branch is simply behind another. remote Configures the (possibly remote) locations where copies of this repository exist. upstream The remote where to execute the remote operation against if not otherwise specified. Pianini (UniBo) Continuous integration and delivery May 12, 2017 33 / 117
  37. 37. Successful teamwork with distributed control version systems git The compact guide to git VI clone Copies a repository from a possibly remote location. Alternative to init. Automatically sets the upstream to the cloned location. fetch Locally copies the changes from an URL or a configured remote. pull Equivalent to fetch+merge. push Copies the local changes to an URL or a configured remote. A configured remote can be set as upstream (push -u). Pianini (UniBo) Continuous integration and delivery May 12, 2017 34 / 117
  38. 38. Successful teamwork with distributed control version systems git Best practices with git I Prefer the terminal Do not use graphical interfaces to the DVCS until you have a complete grasp of it, only rely on the terminal. Ignore list Immediately configure a .gitignore file Pianini (UniBo) Continuous integration and delivery May 12, 2017 35 / 117
  39. 39. Successful teamwork with distributed control version systems git Best practices with git II Choose what to track carefully Track only the files that are not generable (sources, resources, scripts...). Never ever track binaries generated sources (e.g. from ANTLR, Xtend, JavaCC...) generable documentation (e.g. Javadocs) libraries (if you have a dependency manager) Failing at doing so will make your repository explode in size. Pianini (UniBo) Continuous integration and delivery May 12, 2017 36 / 117
  40. 40. Successful teamwork with distributed control version systems git Best practices with git III Properly configure the behaviour with newlines The default behaviour is inconsistent between Windows and UNIX, and you may end up with a mixture Force the end line of your choice (may your choice be LF) Disable the autoclrf conversion, frequently use diff to make sure you’re not messing up Pianini (UniBo) Continuous integration and delivery May 12, 2017 37 / 117
  41. 41. Successful teamwork with distributed control version systems git How the world looks like now prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 38 / 117
  42. 42. Successful teamwork with distributed control version systems git flow Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 39 / 117
  43. 43. Successful teamwork with distributed control version systems git flow How will the world look like shortly prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 39 / 117
  44. 44. Successful teamwork with distributed control version systems git flow Git Flow as a development model Mainline branch: master Always alive Development branch: develop Branches from master Never merges New feature: feature-* Branches from and merges to develop New release: release-* Branches from develop Merges to master and develop Creates a new version Fix a bug on mainline: hotfix-* Branches from master Merges to master and develop Creates a new version Pianini (UniBo) Continuous integration and delivery May 12, 2017 40 / 117
  45. 45. Successful teamwork with distributed control version systems git flow Developing new features When a new feature must be added to the software, the team responsible for it should branch a feature-myfeature branch from develop. git checkout -b feature-myfeature develop Once all the work has been done, the branch should get merged in develop and deleted. Even if fast-forward, the merge should be visible, to prevent history loss. git checkout develop git merge --no-ff feature-myfeature git branch -d feature-myfeature In order to minimize the merging effort, it is a good practice to incorporate changes from develop from time to time (e.g. when another team completed another feature). Pianini (UniBo) Continuous integration and delivery May 12, 2017 41 / 117
  46. 46. Successful teamwork with distributed control version systems git flow Releasing a new version When the status of the develop branch is (besides the very last bits) the status of the next production release, a release-version should branch from develop. git checkout -b release-version develop In this branch, only things like version number changes or last minute bug fixes should get incorporated. Once done, we merge the release-version branch back into develop... git checkout develop git merge --no-ff release-version ...and master. Plus, we tag master, so that we keep a reference to the exact repository status at release time. Then, we delete the release-version branch. git checkout master git merge --no-ff release-version git tag -a version git branch -d release-version Pianini (UniBo) Continuous integration and delivery May 12, 2017 42 / 117
  47. 47. Successful teamwork with distributed control version systems git flow Fix severe bugs affecting the mainline You spot a bug in your current production branch. The fix must be delivered immediately. Start a new hotfix-version branch: git checkout -b hotfix-version master Change the version number, fix the bug (also add a regression test). Once done, repeat the procedure already seen for a normal release. git checkout develop git merge --no-ff hotfix-version git checkout master git merge --no-ff hotfix-version git tag -a version git branch -d hotfix-version Pianini (UniBo) Continuous integration and delivery May 12, 2017 43 / 117
  48. 48. Successful teamwork with distributed control version systems git flow git flow This workflow, first suggested by Vincent Driessen, got very popular. The command sequence is repetitive, and got automated A git extension (not part of the git distribution) is available: Introduces the git flow subcommand Starts and finishes feature, release, hotfix (and support) branches Under the hood, it calls exactly the commands listed previously My suggestion learn git flow as a development model Get acquainted with it using standard git When you are very confident that you know what the tool is doing with your repository, use git flow This is a good approach in general to new tools: understand the idea learn the basics understand what goes on under the hood use the advanced features productively Pianini (UniBo) Continuous integration and delivery May 12, 2017 44 / 117
  49. 49. Successful teamwork with distributed control version systems git flow How the world looks like now prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 45 / 117
  50. 50. Successful teamwork with distributed control version systems GitHub Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 46 / 117
  51. 51. Successful teamwork with distributed control version systems GitHub How will the world look like shortly prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 46 / 117
  52. 52. Successful teamwork with distributed control version systems GitHub GitHub I Main features Support for Git repositories hosting Free for public, open source projects Issue tracker Support for defining teams / organizations Code highlighting History navigation Markdown formatting supported Fork/Pull request support Pianini (UniBo) Continuous integration and delivery May 12, 2017 47 / 117
  53. 53. Successful teamwork with distributed control version systems GitHub GitHub II Static websites One static website per project, one per user, one per organization The gh-pages branch of each repository is implicitly considered a documentation web page Support for Jekyll! a a https://jekyllrb.com/ Pianini (UniBo) Continuous integration and delivery May 12, 2017 48 / 117
  54. 54. Successful teamwork with distributed control version systems GitHub GitHub III Why GitHub By far the largest host of source code in the world Markdown supported also in comments and issue posts Beautiful development statistics De facto standard for open source software Pianini (UniBo) Continuous integration and delivery May 12, 2017 49 / 117
  55. 55. Successful teamwork with distributed control version systems Working with forks Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 50 / 117
  56. 56. Successful teamwork with distributed control version systems Working with forks Back to centralized? There is no inherent concept of “central repository” in git The Git flow branching model considers a “central” repository (truth repo) where every developer insists The methodology focusses on how to maintain the central repo, however: Each developer may have his own copy (fork) of the project There may be a responsible of the central repository, the only one with write permits Other developers request the maintainer to pull from their forks there is specific support for this workflow in hosting platforms The maintainer decides when to actually create releases In git, use remote to easily work with multiple forks Pianini (UniBo) Continuous integration and delivery May 12, 2017 50 / 117
  57. 57. Successful teamwork with distributed control version systems Working with forks How the world looks like now prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 51 / 117
  58. 58. Build automation Dependency management Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 52 / 117
  59. 59. Build automation Dependency management How will the world look like shortly prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 52 / 117
  60. 60. Build automation Dependency management The concept of dependency Any software depends on other software All the runtime base libraries (think of java.lang.* and System.*) All the other core libraries Possibly, external resources (e.g., images, sounds, translation files...) Normally, this software depends on other software That depend on other software That depend on other software, and so on... A normal applications has a tree of dependencies Pianini (UniBo) Continuous integration and delivery May 12, 2017 53 / 117
  61. 61. Build automation Dependency management Example: print titles Requirements Write a program that: Visits TheTVDB.org (public TV Series database) Searches for a series Download the titles of all the episodes Prints them on screen Questions Estimate how much code (in Java) you’d need to write How much code can be just inherited from existing, public libraries? Pianini (UniBo) Continuous integration and delivery May 12, 2017 54 / 117
  62. 62. Build automation Dependency management Maybe less code than you may expect package it.unibo.ci; import ... public final class PrintSeries { private static final String LANG = "it"; private static final String SERIE = "Breaking Bad"; private PrintSeries() { } public static void main(final String... args) throws IOException { final String key = IOUtils.toString(PrintSeries.class .getResourceAsStream("/TheTVDBAPIKey"), Charsets.UTF_8); final TheTVDBApi api = new TheTVDBApi(key); api.searchSeries("Breaking Bad", LANG).stream() .filter(s -> s.getSeriesName().equals(SERIE)) .map(Series::getId) .flatMap(sId -> api.getAllEpisodes(sId, LANG).stream()) .map(Episode::getEpisodeName) .forEach(System.out::println); } } Pianini (UniBo) Continuous integration and delivery May 12, 2017 55 / 117
  63. 63. Build automation Dependency management Trick revealed I used a few existing libraries! Google Guava Used for referencing the UTF-8 Charset without using Strings (less error-prone) Apache Commons I/O Used for converting a resource stream pointing to a String Omertron’s thetvdbapi Queries TheTVDB given a valid API key, hiding all the HTTP communication and XML parsing But wait, there is more! I only needed three libraries to get the job done. But are those libraries using other libraries? Pianini (UniBo) Continuous integration and delivery May 12, 2017 56 / 117
  64. 64. Build automation Dependency management The actual dependency tree +--- com.google.guava:guava:+ -> 19.0-rc2 +--- commons-io:commons-io:+ -> 2.4 --- com.omertron:thetvdbapi:+ -> 1.7 +--- org.slf4j:slf4j-api:1.7.9 --- org.yamj:api-common:1.2 +--- org.apache.commons:commons-lang3:3.3.2 +--- commons-dbcp:commons-dbcp:1.4 | --- commons-pool:commons-pool:1.5.4 -> 1.6 +--- commons-pool:commons-pool:1.6 +--- commons-codec:commons-codec:1.10 +--- org.apache.httpcomponents:httpclient:4.3.6 | +--- org.apache.httpcomponents:httpcore:4.3.3 | +--- commons-logging:commons-logging:1.1.3 | --- commons-codec:commons-codec:1.6 -> 1.10 --- org.slf4j:slf4j-api:1.7.9 Libraries depend on other libraries All the libraries must be in the classpath! Pianini (UniBo) Continuous integration and delivery May 12, 2017 57 / 117
  65. 65. Build automation Dependency management Towards a dependency hell I This was a toy program, consisting of a single Java source file of some twenty lines of code. Regardless, it requires 12 external libraries in order to run. Libraries explosion It is very common for a non-toy project to get past 50 dependencies Alchemist, big but not huge, counts more than 120 dependencies Hard to search, download and verify compatibilities by hand Version conflicts soon arise one of your direct dependencies uses library A at version 1 another uses library A at version 2 you have a so-called transitive dependency conflict on A Upgrading by hand requires, time, effort and tons of testing Pianini (UniBo) Continuous integration and delivery May 12, 2017 58 / 117
  66. 66. Build automation Dependency management Towards a dependency hell II Source import Duplication More library code than product code Extremely difficult to update Style inconsistencies Different quality metrics Duplication Unmaintainable Pianini (UniBo) Continuous integration and delivery May 12, 2017 59 / 117
  67. 67. Build automation Dependency management Towards a dependency hell III Binary import (copy of jars in the repo) At every update, a new jar must be included, along with all its dependencies Being a compressed binary file, its hard to write diffs Git must store a copy of the file for each version (even if very little changed actually) The repository size explodes Take a look at [tt] for an anti-pattern Trust me, you want an automated tool to get you out of this hell. Pianini (UniBo) Continuous integration and delivery May 12, 2017 60 / 117
  68. 68. Build automation Dependency management Desiderata Declarative dependency specification Automatic fetch and retrieve of the required dependencies Automatic and configurable version conflict resolution Multiple dependency scopes: We need JUnit for our tests, but we don’t want any of our production sources to use its APIs (test scope differs from compile scope) also, we don’t want to embed JUnit in our production dependency set (test scope differs from runtime scope) We need Logback [teac] for our runtime, but we want our code to depend only on the SLF4J APIs for reusability (runtime scope larger than compile scope) We want ANTLR4 [Par13] for generating some source code, but we only want its runtime once this phase is concluded (custom scope) Configurable software sources Pianini (UniBo) Continuous integration and delivery May 12, 2017 61 / 117
  69. 69. Build automation Dependency management Possible desired code repositories { mavenCentral() jCenter() } dependencies { sourceGeneration 'org.antlr:antlr4:4.7' compile 'org.slf4j:slf4j-api:1.7.25' compile 'org.antlr:antlr-runtime:3.5.2' testCompile 'junit:junit:4.12' runtime 'ch.qos.logback:logback-classic:1.2.2' testRuntime 'ch.qos.logback:logback-classic:1.2.2' } Pianini (UniBo) Continuous integration and delivery May 12, 2017 62 / 117
  70. 70. Build automation Dependency management Moar automation Dependency management is just the first need that arises. What you really want to automatize Dependency management Software compilation Test execution compilation Documentation generation Reports generation Artefacts assemblage Artefacts signing Everything that goes from declaring what your software needs to having it ready for deployment Except actually developing the project Pianini (UniBo) Continuous integration and delivery May 12, 2017 63 / 117
  71. 71. Build automation Gradle Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 64 / 117
  72. 72. Build automation Gradle Gradle I Idea Pick the best of “declarative” build systems, such as Apache Maven Dependency resolution Rich default configuration Pick the best from “imperative” build systems, such as Apache Ant. Extreme flexibility Pianini (UniBo) Continuous integration and delivery May 12, 2017 64 / 117
  73. 73. Build automation Gradle Gradle II Features The build is written in a Groovy based DSL Inherits from Ant the concept of “task” Automagic resolution of the order in which tasks should be executed Built-in dependency resolution as Maven (SBT and Ant rely on Apache Ivy) Incremental builds Parallel task execution Supports many languages (Java, Scala, Groovy are first class citizens) Maven-style extensibility via plugins Pianini (UniBo) Continuous integration and delivery May 12, 2017 65 / 117
  74. 74. Build automation Gradle Gradle III Diffusion Slowly replacing Ant+Ivy and Maven as reference build system for Java and other JVM based languages Gaining momentum also in the C/C++ community (assembler, c, cpp, objective-c, and objective-cpp are set to be included in the base distribution) Selected by Google as preferred build system for Android Android Studio by default configures a Gradle build under the hood Similar to SBT, arguably less tailored towards Scala Well integrated in Eclipse and (especially) IntelliJ Eclipse recently included the Gradle Buildship in the classic distribution Pianini (UniBo) Continuous integration and delivery May 12, 2017 66 / 117
  75. 75. Build automation Gradle Gradle IV Details Created in 2008 by Gradleware Mostly implemented in Java 5, with an outer layer in Groovy Free - both as in freedom (Apache License 2.0) and as in beer (no fees required) Source code available on GitHub Thorough documentation - though some advanced use requires some good personal initiative... Pianini (UniBo) Continuous integration and delivery May 12, 2017 67 / 117
  76. 76. Build automation Gradle Gradle concepts Project - from the Gradle documentation What a project represents depends on what it is that you are doing with Gradle. For example, a project might represent a library JAR or a web application. It might represent a distribution ZIP assembled from the JARs produced by other projects. A project does not necessarily represent a thing to be built. It might represent a thing to be done, such as deploying your application to staging or production environments. Task - from the Gradle documentation Each project is made up of one or more tasks. A task represents some atomic piece of work which a build performs. This might be compiling some classes, creating a JAR, generating Javadoc, or publishing some archives to a repository. Pianini (UniBo) Continuous integration and delivery May 12, 2017 68 / 117
  77. 77. Build automation Gradle Under the hood The Gradle build script is technically a valid Groovy script (if you consider the Gradle API) Anything that has not a valid Groovy syntax is not a valid Gradle build script Groovy makes it easy to create DSLs, Gradle is built relying on such a feature Everything you write is actually proper Groovy code (method calls, closures, and so on), but (you’ll see) that aspect is very nicely hidden At the high level, the feeling is to just have to configure an existing plugin, much like Maven, for most of the things you normally do When needed, it is easy to configure custom behaviour, or fiddle with the internals, in a functional or imperative fashion Pianini (UniBo) Continuous integration and delivery May 12, 2017 69 / 117
  78. 78. Build automation Gradle Minimal Java build I src/main/java/HelloWorld.java public class HelloWorld { public static void main(String[] args) { System.out.println("Hello, world!"); } } build.gradle apply plugin: 'java' Yes, it’s a one-liner. Pianini (UniBo) Continuous integration and delivery May 12, 2017 70 / 117
  79. 79. Build automation Gradle Minimal Java build II File system before the Gradle execution $ tree -A . |-- build.gradle |-- src |-- main |-- java |-- HelloWorld.java 3 directories, 2 files Pianini (UniBo) Continuous integration and delivery May 12, 2017 71 / 117
  80. 80. Build automation Gradle Minimal Java build III Terminal $ gradle build :compileJava :processResources NO-SOURCE :classes :jar :assemble :compileTestJava NO-SOURCE :processTestResources NO-SOURCE :testClasses UP-TO-DATE :test NO-SOURCE :check UP-TO-DATE :build BUILD SUCCESSFUL Total time: 1.024 secs Pianini (UniBo) Continuous integration and delivery May 12, 2017 72 / 117
  81. 81. Build automation Gradle Minimal Java build IV File system after the execution $ tree -A . |-- build | |-- classes | | |-- main | | |-- HelloWorld.class | |-- libs | | |-- 06 - Minimal Java.jar | |-- tmp | |-- compileJava | |-- jar | |-- MANIFEST.MF |-- build.gradle |-- src |-- main |-- java |-- HelloWorld.java 10 directories, 5 files Pianini (UniBo) Continuous integration and delivery May 12, 2017 73 / 117
  82. 82. Build automation Gradle Minimal Java build V Pretty magic, huh? Pianini (UniBo) Continuous integration and delivery May 12, 2017 74 / 117
  83. 83. Build automation Gradle Let’s build our toy example src/main/java/it/unibo/ci/PrintBreakingBad.java package it.unibo.ci; import java.io.IOException; import org.apache.commons.io.IOUtils; import com.google.common.base.Charsets; import com.omertron.thetvdbapi.TheTVDBApi; import com.omertron.thetvdbapi.model.Episode; import com.omertron.thetvdbapi.model.Series; public final class PrintBreakingBad { private static final String LANG = "it"; private static final String SERIE = "Breaking Bad"; private PrintBreakingBad() { } public static void main(String... args) throws ClassNotFoundException, IOException { final String key = IOUtils.toString(PrintBreakingBad.class.getResourceAsStream("/TheTVDBAPIKey"), Charsets.UTF_8); final TheTVDBApi api = new TheTVDBApi(key); api.searchSeries("Breaking Bad", LANG).stream() .filter(s -> s.getSeriesName().equals(SERIE)) .map(Series::getId) .flatMap(s -> api.getAllEpisodes(s, LANG).stream()) .map(Episode::getEpisodeName) .forEach(System.out::println); } } Pianini (UniBo) Continuous integration and delivery May 12, 2017 75 / 117
  84. 84. Build automation Gradle Let’s build our toy example build.gradle apply plugin: 'java' sourceCompatibility = "$jdkVersion" repositories { mavenCentral() } dependencies { compile "com.google.guava:guava:$guavaVersion" compile "commons-io:commons-io:$commonsIoVersion" compile "com.omertron:thetvdbapi:$tvDbApiVersion" } Pianini (UniBo) Continuous integration and delivery May 12, 2017 76 / 117
  85. 85. Build automation Gradle Let’s build our toy example gradle.properties group = it.unibo.apice artifactId = printbbepisodes version = 0.0.0 projectLongName = Breaking Bad Episode Titles projectDescription = A program that fetches information about Breaking Bad on TheTVDb and prints episodes licenseName = Apache License 2.0 jdkVersion = 1.8 commonsIoVersion = + guavaVersion = 19.0 tvDbApiVersion = [1.6, 1.7] settings.gradle rootProject.name = "$artifactId" Pianini (UniBo) Continuous integration and delivery May 12, 2017 77 / 117
  86. 86. Build automation Gradle Let’s build our toy example Execution $ gradle clean build :clean :compileJava Download https://repo1.maven.org/maven2/com/google/guava/guava/19.0/guava-19.0.pom Download https://repo1.maven.org/maven2/com/google/guava/guava-parent/19.0/guava-parent-19.0.pom Download https://repo1.maven.org/maven2/com/omertron/thetvdbapi/1.7/thetvdbapi-1.7.pom Download https://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom Download https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.9/slf4j-api-1.7.9.pom ...many transitive dependencies after... Download https://repo1.maven.org/maven2/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar Download https://repo1.maven.org/maven2/commons-pool/commons-pool/1.6/commons-pool-1.6.jar Download https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar Download https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar Download https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar :processResources :classes :jar :assemble :compileTestJava UP-TO-DATE :processTestResources UP-TO-DATE :testClasses UP-TO-DATE :test UP-TO-DATE :check UP-TO-DATE :build BUILD SUCCESSFUL Total time: 9.002 secs Pianini (UniBo) Continuous integration and delivery May 12, 2017 78 / 117
  87. 87. Build automation Gradle Dependency management in Gradle Dependencies in Gradle can be specified by defining Where to retrieve software (Maven Central and JCenter are first class citizens) The list of dependencies for each scope Some scopes import other scopes (e.g. runtime by default contains the whole compile scope) Dependencies can be specified with a specific version, a version range, or using + as “latest” With 1.+ meaning “the latest version starting with 1.” Prefer specific versions or ranges (you must test them) Transitive dependencies are resolved automatically (if available in the repository, of course) Specifying the sourceCompatibility is not required, but recommended Pianini (UniBo) Continuous integration and delivery May 12, 2017 79 / 117
  88. 88. Build automation Gradle Tests as part of the build As you can see from the output of the previous case, the tests are executed by default for every build! Tests and Gradle, how to: Add the libraries you need to the appropriate dependency scope Put your test sources in src/test/nameofyourlanguage build.gradle that enables and uses JUnit apply plugin: 'java' repositories { mavenCentral() } dependencies { testCompile 'junit:junit:4.12' } Pianini (UniBo) Continuous integration and delivery May 12, 2017 80 / 117
  89. 89. Build automation Gradle Multi-language build I File system before the execution $ tree -A . |-- build.gradle |-- src |-- main |-- groovy | |-- HelloGroovy.groovy |-- java | |-- HelloWorld.java | |-- TestXtend.xtend |-- scala |-- HelloScala.scala 5 directories, 5 files src/main/groovy/HelloGroovy.groovy println "Hello, Groovy" Pianini (UniBo) Continuous integration and delivery May 12, 2017 81 / 117
  90. 90. Build automation Gradle Multi-language build II src/main/java/HelloWorld.java public class HelloWorld { public static void main(String... args) { System.out.println("Hello, world!"); } } src/main/java/TestXtend.xtend class TestXtend { def static void main(String... args) { println("Hello Xtend") } } Pianini (UniBo) Continuous integration and delivery May 12, 2017 82 / 117
  91. 91. Build automation Gradle Multi-language build III src/main/scala/HelloScala.scala object HelloScala extends App { println("Hello Scala") } Pianini (UniBo) Continuous integration and delivery May 12, 2017 83 / 117
  92. 92. Build automation Gradle Multi-language build IV build.gradle plugins { id "org.xtext.xtend" version "1.0.0" } apply plugin: 'java' apply plugin: 'scala' apply plugin: 'groovy' repositories { mavenCentral() } dependencies { compile 'org.scala-lang:scala-library:2.12.2' compile 'org.codehaus.groovy:groovy:2.3.7' compile 'org.eclipse.xtend:org.eclipse.xtend.lib:2.9.0' } Pianini (UniBo) Continuous integration and delivery May 12, 2017 84 / 117
  93. 93. Build automation Gradle Multi-language build V Execution $ gradle clean build :clean :generateXtext :compileJava :compileGroovy :compileScala :processResources NO-SOURCE :classes :jar :assemble :generateTestXtext NO-SOURCE :compileTestJava NO-SOURCE :compileTestGroovy NO-SOURCE :compileTestScala NO-SOURCE :processTestResources NO-SOURCE :testClasses UP-TO-DATE :test NO-SOURCE :check UP-TO-DATE :build BUILD SUCCESSFUL Total time: 10.627 secs Pianini (UniBo) Continuous integration and delivery May 12, 2017 85 / 117
  94. 94. Build automation Gradle Multi-language build VI Build folder after the execution $ tree -A build build |-- classes | |-- main | |-- HelloGroovy.class | |-- HelloScala.class | |-- HelloScala$.class | |-- HelloScala$delayedInit$body.class | |-- HelloWorld.class | |-- TestXtend.class |-- java | |-- main |-- libs | |-- 02-multilang.jar |-- tmp | |-- compileGroovy | | |-- groovy-java-stubs | |-- compileJava | |-- compileScala | |-- jar | | |-- MANIFEST.MF | |-- scala | |-- compilerAnalysis | |-- compileScala.analysis | |-- compileScala.analysis.relations |-- xtend |-- main |-- TestXtend.java 15 directories, 11 files Pianini (UniBo) Continuous integration and delivery May 12, 2017 86 / 117
  95. 95. Build automation Gradle Build system as dependency If you rely on a build system to build your project, then the build system itself is a dependency! Different versions may produce a different outcome Old versions may not work Newer version may stop working The build system is as any other dependency! The Gradle wrapper Gradle provides a builtin wrapper task: Downloads a gradle core jar and a few settings files Those file must be added to the tracker gradle wrapper --gradle-version 3.1 downloads version 3.1 ./gradlew <task> runs <task> with that exact version Pianini (UniBo) Continuous integration and delivery May 12, 2017 87 / 117
  96. 96. Build automation Gradle IDE integration IntelliJ integration is straightforward, no additional configuration is required. A Gradle integration plugin is included in the Eclipse distribution (Buildship). build.gradle apply plugin: 'eclipse' eclipse { classpath { downloadJavadoc = true downloadSources = true } } Downloads from the repositories also sources and javadocs and feeds them to Eclipse: much easier to develop... Pianini (UniBo) Continuous integration and delivery May 12, 2017 88 / 117
  97. 97. Build automation Gradle How the world looks like now prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 89 / 117
  98. 98. Build automation Code quality control Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 90 / 117
  99. 99. Build automation Code quality control How will the world look like shortly prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 90 / 117
  100. 100. Build automation Code quality control Automatic code quality control Why Immediately spot subtle bugs Early spot sub-optimal code (singular fields, missing finals...) Enforce encapsulation, spot cut/pastes (normally sign of bad design choices) Use a coherent style across your code Prevent style-change commits (“The Leopard commits”) Particularly important if there are many developers! Pianini (UniBo) Continuous integration and delivery May 12, 2017 91 / 117
  101. 101. Build automation Code quality control Automatic code quality control How FindBugs Analyzes the bytecode searching for bugs PMD Source code analyzer for common programming flaws Finds suspect cut/paste (CPD module) Works for Java, Javascript, PLSQL, Velocity, XML, XSL Checkstyle Forces code standard (indentation, formatting, Javadoc...) Can be configured for working with arbitrary files Codacy Web based scala-meta is available for Scala Plugins available for Gradle, Eclipse and IntelliJ Pianini (UniBo) Continuous integration and delivery May 12, 2017 92 / 117
  102. 102. Build automation Code quality control Code quality control advices Ship the code checkers configuration files with your distribution Usually just one or two XML files Make your IDE plugins configuration match exactly your build automation tool configuration Always track the IDE configuration with your SCM New developers will import the configuration automatically if they have plugins installed If a bad developer tries to change the configuration, you can spot it from the commit change set Pick the rule configuration that suits your project Some rules are controversial at least Some rule have default arbitrary limits Alchemist rules come from several years of tuning, fill free to use them Require your developers to adhere Pianini (UniBo) Continuous integration and delivery May 12, 2017 93 / 117
  103. 103. Build automation Code quality control How does the world look like now prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 94 / 117
  104. 104. Continuous integration Travis CI Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 95 / 117
  105. 105. Continuous integration Travis CI How will the world look like shortly prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 95 / 117
  106. 106. Continuous integration Travis CI Summary I We got Teamwork tools Proficient teamwork strategies Project repositories Dependency management Tests integrated in the build Code quality control Automatic production of the artifacts Automatic production of reports Pianini (UniBo) Continuous integration and delivery May 12, 2017 96 / 117
  107. 107. Continuous integration Travis CI Summary II We miss Execute the whole build for every change Doing it locally makes development too slow, defeating the purpose of saving development time Execute the tests in a fresh environment, resembling the production environment We would need to clone a fresh VM, run the build there, then throw it away Automatic deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 97 / 117
  108. 108. Continuous integration Travis CI Continuous Integration software Software that promotes the practice of continuous integration Runs a build for every change in the project Prepares fresh environments where the builds are hosted Notifies the results, e.g. if a failure occurs Provides tools for deploying the produced artifacts Hosted CI with free plans for open source projects are blossoming: Circle CI Codefresh Codeship drone.io Pipelines Travis CI Wercker ... Pianini (UniBo) Continuous integration and delivery May 12, 2017 98 / 117
  109. 109. Continuous integration Travis CI Travis CI Web based Well integrated with GitHub Build results are displayed in the repo without intervention Automatic build of any pull request Free for open source projects Cronjobs Build instances based on Docker Project-local configuration via YAML (in the .travis.yml file) Out of the box support for Gradle Dozens of deployment targets Pianini (UniBo) Continuous integration and delivery May 12, 2017 99 / 117
  110. 110. Continuous integration Travis CI How it works A web-hook can be registered to your GitHub repository that triggers Travis CI at each new commit Travis CI starts a pristine appropriate environment Can be a container or a full virtual machine, depending on whether sudo is required [CIa] The project gets cloned The system is configured as per .travis.yml settings The configured commands are executed The configured deployments are performed If necessary, project managers are informed of the build status Pianini (UniBo) Continuous integration and delivery May 12, 2017 100 / 117
  111. 111. Continuous integration Travis CI Minimal Travis configuration .travis.yml language: java 1 Automatically searches for build.gradle 2 If found and gradlew is present runs 1 ./gradlew assemble 2 ./gradlew check 3 Otherwise runs the same commands with the version of Gradle pre-installed on the instance The command to be executed can be configured manually (recommended) .travis.yml language: java script: - './gradlew clean build' Further documentation on building Java-based projects is available [CIb] Pianini (UniBo) Continuous integration and delivery May 12, 2017 101 / 117
  112. 112. Continuous integration Travis CI How will the world looks like noe prom ote clone notify release w eb hook review fork pull request pushpull ... deploy cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 102 / 117
  113. 113. Continuous integration Deployment Outline 1 Introduction Highway to hell Stairway to heaven 2 Successful teamwork with distributed control version systems git git flow GitHub Working with forks 3 Build automation Dependency management Gradle Code quality control 4 Continuous integration Travis CI Deployment Pianini (UniBo) Continuous integration and delivery May 12, 2017 103 / 117
  114. 114. Continuous integration Deployment How will the world look like shortly prom ote clone notify release w eb hook review deploy fork pull request pushpull ... cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 103 / 117
  115. 115. Continuous integration Deployment Travis deployments Travis CI supports deployment on a variety of targets. Two examples: 1 GitHub releases Creates a new download section on GitHub Great for storing executable versions (e.g. “fat jars”) Nice automatic deployment examples are available in the Alchemist [Piaa] and Protelis [Piab] build configurations 2 surge.sh hosting Uploads a static web page on a surge.sh page Very handy for web based documentation (e.g. Javadoc) Nice automatic deployment examples are available in the Alchemist [Piaa] and Protelis [Piab] build configurations Details on how to configure the deployments are provided in the Travis documentation [CIc]. Pianini (UniBo) Continuous integration and delivery May 12, 2017 104 / 117
  116. 116. Continuous integration Deployment Online repositories The deployment of your software greatly varies depending on who’s destined to. Making the software easy to use Very common by-product (and sometimes product) of academic research Just shipping source code on GitHub and / or publishing jar files on a random server is not enough Especially for libraries or middleware, it is way better to provide a proper repository, where other projects can just point to and import the product as Gradle / Maven / Ant / Ivy dependency Public services exist that offer a free and reliable repository Pianini (UniBo) Continuous integration and delivery May 12, 2017 105 / 117
  117. 117. Continuous integration Deployment OSSRH — aka Maven Central Offered and managed by Sonatype Default software source for Maven builds Trivial to setup with any other build automation tool De-facto standard No-retract policy: if you publish an artifact, you cannot modify it, no exception allowed Gets copied from other repositories, e.g. jCenter Artifact staging and release through an instance of Sonatype Nexus Artifacts have a product name and belong to a group A Sonatype JIRA account and digital signature is required to manage a group Digital signature on artifacts required Strict quality control: sources and documentation must be provided See: http://central.sonatype.org/pages/ossrh-guide.html Pianini (UniBo) Continuous integration and delivery May 12, 2017 106 / 117
  118. 118. Continuous integration Deployment No retract? What if I make a mistake? Short answer Your error will remain in the repositories forever and you will never able to fix it, you will need to create a new official release with a different version number. Anecdotal evidence of why this is good In March 2016, Azer Ko¸clu unpublished more than 250 of his modules from NPM, which is a popular package manager used by Javascript projects to install dependencies, because he was asked to rename the module “Kik”, whose name is the same of an instant messaging app that wanted to publish a homonym module. Unfortunately, one of those dependencies was left-pad, an 11 lines long Javascript module, used by thousands of projects. This brought breackages throughout many Javascript products (most notably Node and Babel). NPM had to un-unpublish left-pad. Pianini (UniBo) Continuous integration and delivery May 12, 2017 107 / 117
  119. 119. Continuous integration Deployment Deployment automatization We have our artifacts, automatically tested and compiled (at least) every day on our nice automatic build / test / integration framework. We also want to have automatic deployment to OSSRH, so we need: 1 Generation of all required artifacts: a sources jar file and a javadoc jar file 2 An OSSRH account 3 A GPG signature of all these artifacts 4 Creation of a pom.xml file 5 Automatic publication in our target repository 6 Release of final versions of our software Can be automated, but I prefer to have a manual check if there is a no-retract policy Pianini (UniBo) Continuous integration and delivery May 12, 2017 108 / 117
  120. 120. Continuous integration Deployment Automatize artifact creation Very easy in Gradle: A task that runs after classes are compiled (to be sure we don’t pack non-compiling stuff) that fits in a jar all the source code A task that runs after the Javadoc has been generated, and compresses it in a jar file Configure Gradle to add those files to the generated artifacts In build.gradle task sourcesJar(type: Jar, dependsOn:classes) { classifier = 'sources' from sourceSets.main.allSource } task javadocJar(type: Jar, dependsOn:javadoc) { classifier = 'javadoc' from javadoc.destinationDir } artifacts { archives sourcesJar archives javadocJar } Pianini (UniBo) Continuous integration and delivery May 12, 2017 109 / 117
  121. 121. Continuous integration Deployment OSSRH account 1 Create and publish your own GPG key This is personal, and must not be shared Good setup guide available at: http://central.sonatype.org/pages/working-with-pgp-signatures.html 2 Create a new account on Sonatype’s Jira 3 Create a new ticket, requesting a new group or to be added to an existing one 4 In about two working days, you will get access to the repository, and your (signed) artifacts will be considered valid Pianini (UniBo) Continuous integration and delivery May 12, 2017 110 / 117
  122. 122. Continuous integration Deployment Automatically sign the artifacts This is the trickies part. Signing requires your private key to be exported to the build instance If the Travis build instance can download it, everybody can It must be encrypted, and the instance must know how to decrypt it But nobody else may be able to In short: use the instance public key to encrypt it (a secure build it’s actually more complicated than just this) Once the system is configured with the private GPG key, Gradle can be told to automatically sign and upload artifacts to Maven Central Pianini (UniBo) Continuous integration and delivery May 12, 2017 111 / 117
  123. 123. Continuous integration Deployment Summary of the whole process prom ote clone notify release w eb hook review deploy fork pull request pushpull ... cron trigger Pianini (UniBo) Continuous integration and delivery May 12, 2017 112 / 117
  124. 124. Bonus track Picking version numbers Without compliance to some sort of formal specification, version numbers are essentially useless for dependency management. By giving a name and clear definition to the above ideas, it becomes easy to communicate your intentions to the users of your software. — Semantic Versioning 2.0.0 (http://semver.org/) Semantic versioning Formally described, with RFC-style wording Three levels plus optional: MAJOR.MINOR.PATCH[-OTHER] MAJOR — Any incompatible API change Major 0 is for initial development: anything may change at any time. MINOR — Add functionality in a backwards-compatible manner PATCH — Backwards-compatible bug fixes OTHER — Optionally decorates the version with additional information. Pianini (UniBo) Continuous integration and delivery May 12, 2017 113 / 117
  125. 125. References References I Travis CI. The build environment. https://docs.travis-ci.com/user/ci-environment/. Accessed: 2017-05-08. Travis CI. Building a java project. https://docs.travis-ci.com/user/languages/java/. Accessed: 2017-05-08. Travis CI. Deployment. https://docs.travis-ci.com/user/deployment. Accessed: 2017-05-08. Darko Fabijan. Why we need continuous integration. https://semaphoreci.com/community/tutorials/continuous-integration. Accessed: 2017-05-03. Pianini (UniBo) Continuous integration and delivery May 12, 2017 114 / 117
  126. 126. References References II Martin Fowler. Continuous integration. https://www.martinfowler.com/articles/continuousIntegration.html. Accessed: 2017-05-02. Randall Munroe. xkcd: Compiling. https://xkcd.com/303/. Accessed: 2017-05-03. Terence Parr. The Definitive ANTLR 4 Reference. Pragmatic Bookshelf, 2nd edition, 2013. Danilo Pianini. Alchemist travis configuration. https://github.com/AlchemistSimulator/Alchemist/blob/ 6cec5427836b505486e9e44c247b2ac15821f8f2/.travis.yml. Accessed: 2017-05-08. Pianini (UniBo) Continuous integration and delivery May 12, 2017 115 / 117
  127. 127. References References III Danilo Pianini. Protelis travis configuration. https://github.com/Protelis/Protelis/blob/ e6ac8910a9107e1c9aab647290919ec4af759656/.travis.yml. Accessed: 2017-05-08. The CodingHorror team. New programming jargon. https://blog.codinghorror.com/new-programming-jargon/. Accessed: 2017-05-02. The IMVU team. Imvu: 3d avatar free chat, make new friends, dress up, shop. https://www.imvu.com/. Accessed: 2017-05-02. The LogBack team. Logback project. https://logback.qos.ch/. Accessed: 2017-05-05. Pianini (UniBo) Continuous integration and delivery May 12, 2017 116 / 117
  128. 128. References References IV The tuProlog team. How not to maintain your dependencies. https://bitbucket.org/danysk/exploded-repository-example. Accessed: 2017-05-05. Pianini (UniBo) Continuous integration and delivery May 12, 2017 117 / 117

×