Netflix Nebula - Gradle Summit 2014


Published on

Netflix has open sourced many of our Gradle plugins under the name Nebula. These plugins are there to lend our expertise and experience to building responsible projects, internally and externally. This talk will cover some of the ones we've published, why we want to share these with the community, how we tested and published them, and most importantly how you can contribute back to them.

Nebula started off as a set of strong opinions to make Gradle simple to use for our developers. But we quickly learned that we could use the same assumptions on our open source projects and on other Gradle plugins to make them easy to build, test and deploy. By standardizing plugin development, we've lowered the barrier to generating them, allowing us to keep our build modular and composable.

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I work on the Engineering Tools team at Netflix, where we build tools to support the developers. Today I’m here to talk about Netflix Build Language, or as we like to call it, nebula.
  • And I’m here to talk about how we try to exist in the open source space with plugins.
  • We had to transition out of a Ant/Ivy. Primarily a JVM shop, there are a few players in this space for build. A lot of conventions were in place and changing them to fit Maven’s model wasn’t an option.
  • Look after the users, and where we can add value.
  • Just couldn’t get the experience as smooth as we wanted it in our enterprise environment. Still haven’t.
    Patched Version of Gradle - Patch to cover a bug in Ivy. Patch to expose status/statusScheme.
    Custom Distribution - So we can embed some init.d scripts, e.g. add our repository servers and add our plugin’s jar.
    Custom Wrapper - To force some variables getting set, like memory defaults or GRADLE_USER_HOME. Plans on customizing to support for re-downloading of static URL
  • End user sees apply plugin nebula. Nebula extension block to hold other extensions via @Delegate annotation.
  • NetflixOSS pre-dated our internal work, but it was going down the Maven route. And we had a goal that we wanted to move to Gradle. We also had concerns about how to integration POMs and our internal Ivy files.
    That was done in a way to make the Gradle work really obvious, i.e. minimal work hidden in plugins, no custom DSL. It’s hideous at this point and based on Gradle 1.6. We had to rush it, to pre-empt projects going out with Maven. github/cloudbees/bintray project plugin forthcoming.
  • NetflixOSS fell way behind our internal development. Majority of our work was not Netflix specific, but turned out to just be what a responsible project needs.
    Also had growing pains with our plugin, so we wanted to decouple them.
    Wanted to make new plugins easy, with CI and SCM. We’re build people, right?
  • Found that many plugins we wanted to use also suffered from basic release engineering practices, which is clearly ironic given the space we’re in.
    Group plugin with SCM, CI and Deployment. Chosen of familiarity, I wish I had looked at more options. Common patterns to give the apply plugin nebula like experience, enabled via nebula-plugin-plugin. Lot of hard coding.
  • Exists as a dedicated Organization, with project names following a pattern.
  • We learned from the @NetflixOSS work that GitHub doesn’t maintain itself.
  • We have to eliminate the manual work. And Jenkins is a great place to do that, except that setting up Jenkins is manual work. I don’t want to fight Jenkins, so we have a script to manage this.
  • The nebula prefix for highly opinionated plugins. While the gradle prefix is for plugins that have general applicability. Many of them are used in the plugin-plugin since they’re so helpful.
  • Not really a plugin, but integrated into the Gradle ecosystem. Meant to be used in tests.
    * ProjectSpec has its limitation, since you can only really apply plugins. Though you can run a hidden evaluate(). Really fast and recommended when ever possible
    PluginProjectSpec is a small addition that just applies your plugin by name, tries it twice and tries it in multi-module project.
    IntegrationSpec run a real Gradle build.
  • All the nebula-plugins use these Test classes, so there’s plenty of examples.
  • Helper methods to create a project, to run a project, and to evaluate the effects of a execution. Will get it’s own directory to run in.
    GradleLauncher is necessary for some in-memory checks or debugging.
  • Not really a plugin, has tasks and Helper method. Meant for runtime.
    AlternativeArchiveTask is to provide an implementation of AbstractArchiveTask, which is not also a CopyTask, since those are greaten special.
  • beforeEvaluate is actually before after evaluates
    getTempDir gives a build directory for a task
    addDefaultGroup lets you set a group, optionally. Hard otherwise because project.getGroup() will provide the parent project’s name as the group.
  • Finally a real plugin. Though docs are the worst of the bunch, this is the most important. Actually a bunch of plugins. All meant to get a publication looking just right, down to the signing.
    Artifact plugins attempt to make the jars (which we’d all expect).
    Publishing plugins attempt to make resulting Ivy/POM file cleaner, primarily by using resolved variables and including excludes.
    nebula-sign looks for magical properties and conditionally signs, unlike other approached.
    To support these, we needed to use Publish 2.0, which isn’t fully baked. We were unable to use as is, so we made our CustomComponent. Allows any plugins to contribute artifacts, with more control of the resulting dependencies and confs (more important in Ivy).
  • Two plugins are available for ivy vs maven, nebula-ivy-publishing and nebula-maven-publishing.
  • ‘info-java' version of java being used
    ‘info-ci' tells about CI system and current build
    ‘info-scm' derives info about current SCM
    ‘info-jar' injects into the Jar’s MANIFEST.MF
    ‘info-props' creates file with values
    ‘info-jar-props' puts property file in jar for later retrieval
    All go through a broker that you can listen to.
  • contacts-manifest puts people into the JAR manifest
    contacts-pom puts people into developers section of POM
    Example of how we believe some plugins can talk to each other without a DSL in the way.
  • Then we can send emails on release by the notify role.
  • We believe in a reproducible build for every developer all the time. Dynamic versions make that hard. We can’t imagine manually editing versions. We’ve all had the experience of a new dependency get published and everyone is broken. We also have cases where we want to tweak a specific module, for a patch, which can be done via the command line. Inspired by systems like
    Let some automated job update your dependencies when they’ve proven valid.
  • Properly sets up classpath.
  • Does the things we’d expect in a responsible project. Even if they don’t like it.
    E.g. doesn’t fail on Javadoc error, maintains status even though the java plugin wants to walk all over it.
  • Adds OJO and Bintray, with publishing. Plugin Portal additions.
    Needs to be extracted out for different orgs, different package names. Very meta, since it applies itself.
  • Most popular plugin, even though we haven’t advertised them at all. Takes the CopySpec idea and applies it to redline and jdeb.
  • buildRpm and buildDeb are implicitly created, but can still be configured if needed.
  • Proposed.
  • Anytime we find a problem internally, we make a plugin to test the problem and fix it. Roll it out and get happy users. Many times, we’re wrapping another plugin and configuring them with our defaults.
  • A few manual steps to finalize release.
  • Lots of withType or .all {} calls
    Ability to create tasks in reaction
    Also try to abstract out logic out of Task, so that it can be called sequentially.
  • Sometimes we even got a third level
    Configure tasks
  • Reacting to the user before the task graph, forces afterEvaluate’s
    NamedDomainObjectSet configuration comes after events, so no ability to re-act, except by name
    @Outputs have to be Files, not in-memory String. Evaluation of File names for outputs can be tricky.
    Can’t debug tests through Tooling API
  • Netflix Nebula - Gradle Summit 2014

    1. 1. @quidryan@quidryan Netflix Build Language Justin Ryan <>
    2. 2. @quidryan@quidryan Justin Ryan <>
    3. 3. Why Netflix Uses Gradle • Better Dependency Management story • Flexible lifecycle • Groovy
    4. 4. How Netflix Uses Gradle • JVM Languages • Resolution • Code Quality • Publishing • Deployment Orchestration
    5. 5. How Netflix Sets up Gradle • Patched Gradle • Custom Distribution • Custom Wrapper
    6. 6. apply plugin: ‘nebula’! apply plugin: ‘java’! ! nebula {! readyForJava7 = true! }! ! dependencies {! compile ‘netflix:platform:latest.release’! } build.gradle
    7. 7. @NetflixOSS
    8. 8. Unite two builds • Model a responsible project • Componentize via plugins
    9. 9. nebula-plugins • Infrastructure • Use Github • Use CloudBees • Use Bintray • Mailing List • nebula-plugin-plugin
    10. 10. Github Repositories nebula-* or gradle-*-plugin
    11. 11. Ensuring Github • • Ensure repository has • Description • Web Hooks • Ensure “contrib" team has all repositories in it • Ensure a contrib team exists for every repository
    12. 12. Continuous Integration • Job DSL to create jobs • Per branch • Snapshot job • Release job • Lock job
    13. 13. CloudBees jobs Release, snapshot, and pull request per branch
    14. 14. Job DSL
    15. 15. Bintray Packages
    16. 16. Ensuring Bintray • • Ensure every repository has a package • Ensure package has the description as Github • Ensure license is set to Apache 2.0 • Ensure labels are “gradle" and “nebula”
    17. 17. nebula-plugin-plugin
    18. 18. nebula-test • ProjectSpec • PluginProjectSpec • IntegrationSpec • Thanks to Marcin and Luke • Runs with Tooling API or GradleLauncher
    19. 19. class PluginExampleSpec extends PluginProjectSpec {! @Override! String getPluginName() { return 'plugin-example' }! ! def ‘run task’() {! when:! project.plugins.apply(PluginExample)! ! then:! def t = project.tasks.get(‘example’)! ! when:!! ! then:! new File(projectDir, ‘build/example.txt’).exists()! }! } PluginProjectSpec
    20. 20. ! def 'setup and run build'() {! buildFile << '''! apply plugin: 'java'! '''.stripIndent()! ! when:! writeHelloWorld('nebula.hello')! ! then:! fileExists('src/main/java/nebula/hello/')! ! when:! def result = runTasksSuccessfully(‘build’, ‘-v’)! ! then:! fileExists(‘build/classes/main/nebula/hello/HelloWorld.class')! result.wasExecuted(':compileTestJava')! def output = result.standardOutput! output.contains('Skipping task ’:compileTestJava'')! } IntegrationSpec
    21. 21. nebula-core • Collection of tasks • Download • Untar • Unzip • AlternativeArchiveTask • CopySpecHelper • GradleHelper
    22. 22. class GradleHelper {! ! def beforeEvaluate(Closure beforeEvaluateClosure)! ! def getTempDir(String taskBaseName)! ! def addDefaultGroup(String defaultGroup)! }! ! class CopySpecHelper {! ! def visitCopySpec(CopySpecInternal copySpec, Closure closure)! ! def findCopySpec(CopySpecInternal delegateCopySpec, closure)! }! ! class ClassHelper {! ! String findSpecificationVersion(Class clazz)! ! Manifest findManifest(Class clazz)! ! def findManifestValue(Class clazz, String key, defaultValue)! } nebula-core helpers
    23. 23. nebula-publishing-plugin • Artifact plugins • nebula-javadoc-jar • nebula-source-jar • nebula-test-jar • Publishing plugins • resolved-ivy • resolved-maven • nebula-sign
    24. 24. apply plugin: ‘nebula-maven-publishing‘! apply plugin: ‘nebula-source-jar'! apply plugin: ‘nebula-javadoc-jar'! apply plugin: ‘nebula-test-jar'! apply plugin: ‘nebula-sign'! apply plugin: 'java' nebula-publishing.gradle my-plugin-1.12.0-javadoc.jar! my-plugin-1.12.0-javadoc.jar.md5! my-plugin-1.12.0-javadoc.jar.sha1! my-plugin-1.12.0-javadoc.jar.asc! my-plugin-1.12.0-sources.jar! my-plugin-1.12.0-sources.jar.md5! my-plugin-1.12.0-sources.jar.sha1! my-plugin-1.12.0-sources.jar.asc! my-plugin-1.12.0-tests.jar! my-plugin-1.12.0-tests.jar.md5! my-plugin-1.12.0-tests.jar.sha1! my-plugin-1.12.0-tests.jar.asc! my-plugin-1.12.0.jar! my-plugin-1.12.0.jar.md5! my-plugin-1.12.0.jar.sha1! my-plugin-1.12.0.jar.asc! my-plugin-1.12.0.pom! my-plugin-1.12.0.pom.md5! my-plugin-1.12.0.pom.sha1! my-plugin-1.12.0.pom.asc
    25. 25. gradle-info-plugin • Collects meta data • ‘info-java' • ‘info-ci' • 'info-scm' • Reports in key/value pairs • ‘info-jar' • ‘info-props'
    26. 26. buildscript {! ! repositories { jcenter() }! ! dependencies { ! ! ! classpath ''! ! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit:'! ! }! }! apply plugin: 'info' info.gradle
    27. 27. Manifest-Version=1.0!;1.12.1-SNAPSHOT! Implementation-Version=1.12.1-SNAPSHOT! Built-Status=integration! Built-By=jryan! Build-Date=2014-06-10_13:30:44! Gradle-Version=1.12-20140608201532+0000! Module-Source=!! Change=976292c! Build-Host=localhost! Build-Job=LOCAL! Build-Number=LOCAL! Build-Id=LOCAL! Created-By=1.7.0_45-b18 (Oracle Corporation)! Build-Java-Version=1.7.0_45!!! X-Compile-Target-JDK=1.7! X-Compile-Source-JDK=1.7 info.gradle output
    28. 28. gradle-contacts-plugin • Express people involved in the project • Make people and roles available to other plugins • contacts-manifest • contacts-pom
    29. 29. apply plugin: 'contacts'! contacts '', ‘'! ! contacts {! ''! '' {! roles 'notify', 'owner'! }! '' {! role 'techwriter'! }! ''! role 'notify'! }! }! contacts.gradle
    30. 30. gradle-scm-plugin • Attempt to provide SCM abstraction for other plugins • E.g. gradle-dependency-lock-plugin and gradle-info- plugin
    31. 31. gradle-dependency-lock-plugin • Developer declare their ideal situation • Save resolved version • If tests pass, commit to SCM
    32. 32. {! "com.github.townsfolk:gradle-release": { ! ! ! "locked": "1.2", "requested": "1.2" },! "com.jfrog.bintray.gradle:gradle-bintray-plugin": { ! ! ! "locked": "0.3", "requested": "0.3" },! "": { ! ! ! "locked": "1.12.0", "requested": "1.12.+" },! "": { ! ! ! "locked": "1.12.0", "requested": "1.12.+" },! "org.codehaus.groovy.modules.http-builder:http-builder": { ! ! ! "locked": "0.7.1", "requested": “latest.release" },! "org.jfrog.buildinfo:build-info-extractor-gradle": { ! ! ! "locked": "2.2.4", "requested": "2.2.+" }! } apply plugin: ‘gradle-dependency-lock'! lock.gradle ./gradlew generateLock
    33. 33. nebula-integtest-plugin • Sets up integTest source set • Adds integTestCompile and integTestRuntime configurations • Creates integrationTest task
    34. 34. nebula-project-plugin • Pull together other plugins • Responsible projects
    35. 35. nebula-plugin-plugin • Used by plugins • Strong opinions on how to publish • Force nebula-project- plugin on projects
    36. 36. gradle-ospackage-plugin • Merging ubuntu-packager-plugin and gradle-rpm- plugin • Uses CopySpec definition • Via just Java, generates RPMs and DEBs
    37. 37. ospackage {! ! os = LINUX! ! into '/opt/foo'! ! from ('dist') {! ! ! user 'builds'! ! ! exclude '**/*.md'! ! }! ! postInstall file('scripts/')! }! ! buildRpm {! ! requires('bar', '2.2', GREATER | EQUAL)! ! from (‘build/’)! ! link(‘/etc/init.d/foo’, '/opt/foo/bin/foo.sysv',)! }! ! buildDeb {! ! link('/etc/init/foo', '/opt/foo/bin/foo.upstart')! } ospackage.gradle
    38. 38. gradle-override-plugin • Take command line arguments • Intelligently apply in afterEvaluate • E.g. -Nfindbugs.enabled=false
    39. 39. Internal Plugins • netflix-repos • nebula-ospackage • nebula-grails • nebula-findbugs, etc • ivyimport • nebula-fixexcludes • nebula-intellij
    40. 40. buildscript {! ! repositories { jcenter() }! ! dependencies { ! ! ! classpath ‘'! ! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit:'! ! }! }! ! description ‘Example Plugin'! apply plugin: ‘nebula-plugin'! ! contacts {! ‘’ {! moniker 'Justin Ryan'! github 'quidryan'! }! } plugin.gradle
    41. 41. curl -s | bash! gvm install lazybones! cd ~/Projects/github/nebula-plugins! lazybones create nebula-plugin <name-of-project>! cd <name-of-project>! git init! git remote add origin<name-of- project>.git! ./gradlew clean build! git add -A! git add -f gradle/wrapper/gradle-wrapper.jar! git commit -m "Initial template”! git push --set-upstream origin master Making a plugin
    42. 42. Getting it out the door • Let “ensure” run • Run <name-of-project>-release • Link package to jcenter • Link package to gradle-plugins
    43. 43. Unopinionated Plugins
    44. 44. Opinionated Plugins
    45. 45. Gotchas • NamedDomainObjectSet • Debugging Tooling • File as @Output • afterEvaluate
    46. 46. Outstanding • CloudBees permissions • Bot to create repositories
    47. 47. Participating • Use individual plugins • Get on nebula-plugins Google Group • Move your plugin to nebula-plugins • Start a new plugin in nebula-plugins
    48. 48. @quidryan@quidryan We’re hiring Justin Ryan <> HOUSE of GRADLE