Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Netflix Nebula - Gradle Summit 2014

6,392 views

Published on

Netflix has open sourced many of our Gradle plugins under the name Nebula. These plugins are there to lend our expertise and experience to building responsible projects, internally and externally. This talk will cover some of the ones we've published, why we want to share these with the community, how we tested and published them, and most importantly how you can contribute back to them.

Nebula started off as a set of strong opinions to make Gradle simple to use for our developers. But we quickly learned that we could use the same assumptions on our open source projects and on other Gradle plugins to make them easy to build, test and deploy. By standardizing plugin development, we've lowered the barrier to generating them, allowing us to keep our build modular and composable.

Netflix Nebula - Gradle Summit 2014

  1. 1. @quidryan@quidryan http://www.slideshare.net/quidryan Netflix Build Language Justin Ryan <jryan@netflix.com>
  2. 2. @quidryan@quidryan http://www.slideshare.net/quidryan Justin Ryan <jryan@netflix.com>
  3. 3. Why Netflix Uses Gradle • Better Dependency Management story • Flexible lifecycle • Groovy
  4. 4. How Netflix Uses Gradle • JVM Languages • Resolution • Code Quality • Publishing • Deployment Orchestration
  5. 5. How Netflix Sets up Gradle • Patched Gradle • Custom Distribution • Custom Wrapper
  6. 6. apply plugin: ‘nebula’! apply plugin: ‘java’! ! nebula {! readyForJava7 = true! }! ! dependencies {! compile ‘netflix:platform:latest.release’! } build.gradle
  7. 7. @NetflixOSS github.com/Netflix netflix.github.io
  8. 8. Unite two builds • Model a responsible project • Componentize via plugins
  9. 9. nebula-plugins • Infrastructure • Use Github • Use CloudBees • Use Bintray • Mailing List • nebula-plugin-plugin
  10. 10. Github Repositories nebula-* or gradle-*-plugin
  11. 11. Ensuring Github • https://github.com/nebula-plugins/ensure • Ensure repository has • Description • Web Hooks • Ensure “contrib" team has all repositories in it • Ensure a contrib team exists for every repository
  12. 12. Continuous Integration • Job DSL to create jobs • Per branch • Snapshot job • Release job • Lock job
  13. 13. CloudBees jobs Release, snapshot, and pull request per branch
  14. 14. Job DSL
  15. 15. Bintray Packages
  16. 16. Ensuring Bintray • https://github.com/nebula-plugins/ensure • Ensure every repository has a package • Ensure package has the description as Github • Ensure license is set to Apache 2.0 • Ensure labels are “gradle" and “nebula”
  17. 17. nebula-plugin-plugin
  18. 18. nebula-test • ProjectSpec • PluginProjectSpec • IntegrationSpec • Thanks to Marcin and Luke • Runs with Tooling API or GradleLauncher
  19. 19. class PluginExampleSpec extends PluginProjectSpec {! @Override! String getPluginName() { return 'plugin-example' }! ! def ‘run task’() {! when:! project.plugins.apply(PluginExample)! ! then:! def t = project.tasks.get(‘example’)! ! when:! t.run()! ! then:! new File(projectDir, ‘build/example.txt’).exists()! }! } PluginProjectSpec
  20. 20. ! def 'setup and run build'() {! buildFile << '''! apply plugin: 'java'! '''.stripIndent()! ! when:! writeHelloWorld('nebula.hello')! ! then:! fileExists('src/main/java/nebula/hello/HelloWorld.java')! ! when:! def result = runTasksSuccessfully(‘build’, ‘-v’)! ! then:! fileExists(‘build/classes/main/nebula/hello/HelloWorld.class')! result.wasExecuted(':compileTestJava')! def output = result.standardOutput! output.contains('Skipping task ’:compileTestJava'')! } IntegrationSpec
  21. 21. nebula-core • Collection of tasks • Download • Untar • Unzip • AlternativeArchiveTask • CopySpecHelper • GradleHelper
  22. 22. class GradleHelper {! ! def beforeEvaluate(Closure beforeEvaluateClosure)! ! def getTempDir(String taskBaseName)! ! def addDefaultGroup(String defaultGroup)! }! ! class CopySpecHelper {! ! def visitCopySpec(CopySpecInternal copySpec, Closure closure)! ! def findCopySpec(CopySpecInternal delegateCopySpec, closure)! }! ! class ClassHelper {! ! String findSpecificationVersion(Class clazz)! ! Manifest findManifest(Class clazz)! ! def findManifestValue(Class clazz, String key, defaultValue)! } nebula-core helpers
  23. 23. nebula-publishing-plugin • Artifact plugins • nebula-javadoc-jar • nebula-source-jar • nebula-test-jar • Publishing plugins • resolved-ivy • resolved-maven • nebula-sign
  24. 24. apply plugin: ‘nebula-maven-publishing‘! apply plugin: ‘nebula-source-jar'! apply plugin: ‘nebula-javadoc-jar'! apply plugin: ‘nebula-test-jar'! apply plugin: ‘nebula-sign'! apply plugin: 'java' nebula-publishing.gradle my-plugin-1.12.0-javadoc.jar! my-plugin-1.12.0-javadoc.jar.md5! my-plugin-1.12.0-javadoc.jar.sha1! my-plugin-1.12.0-javadoc.jar.asc! my-plugin-1.12.0-sources.jar! my-plugin-1.12.0-sources.jar.md5! my-plugin-1.12.0-sources.jar.sha1! my-plugin-1.12.0-sources.jar.asc! my-plugin-1.12.0-tests.jar! my-plugin-1.12.0-tests.jar.md5! my-plugin-1.12.0-tests.jar.sha1! my-plugin-1.12.0-tests.jar.asc! my-plugin-1.12.0.jar! my-plugin-1.12.0.jar.md5! my-plugin-1.12.0.jar.sha1! my-plugin-1.12.0.jar.asc! my-plugin-1.12.0.pom! my-plugin-1.12.0.pom.md5! my-plugin-1.12.0.pom.sha1! my-plugin-1.12.0.pom.asc
  25. 25. gradle-info-plugin • Collects meta data • ‘info-java' • ‘info-ci' • 'info-scm' • Reports in key/value pairs • ‘info-jar' • ‘info-props'
  26. 26. buildscript {! ! repositories { jcenter() }! ! dependencies { ! ! ! classpath 'com.netflix.nebula:gradle-info-plugin:1.12.+'! ! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit: 3.2.0.201312181205-r'! ! }! }! apply plugin: 'info' info.gradle
  27. 27. Manifest-Version=1.0! Implementation-Title=com.netflix.nebula#my-plugin;1.12.1-SNAPSHOT! Implementation-Version=1.12.1-SNAPSHOT! Built-Status=integration! Built-By=jryan! Build-Date=2014-06-10_13:30:44! Gradle-Version=1.12-20140608201532+0000! Module-Source=! Module-Origin=git@github.com:nebula-plugins/my-plugin.git! Change=976292c! Build-Host=localhost! Build-Job=LOCAL! Build-Number=LOCAL! Build-Id=LOCAL! Created-By=1.7.0_45-b18 (Oracle Corporation)! Build-Java-Version=1.7.0_45! Module-Owner=justin@halfempty.org! Module-Email=justin@halfempty.org! X-Compile-Target-JDK=1.7! X-Compile-Source-JDK=1.7 info.gradle output
  28. 28. gradle-contacts-plugin • Express people involved in the project • Make people and roles available to other plugins • contacts-manifest • contacts-pom
  29. 29. apply plugin: 'contacts'! contacts 'minnie@disney.com', ‘mickey@disney.com'! ! contacts {! 'club@disney.com'! 'bobby@company.com' {! roles 'notify', 'owner'! }! 'billy@company.com' {! role 'techwriter'! }! 'downstream@netflix.com'! role 'notify'! }! }! contacts.gradle
  30. 30. gradle-scm-plugin • Attempt to provide SCM abstraction for other plugins • E.g. gradle-dependency-lock-plugin and gradle-info- plugin
  31. 31. gradle-dependency-lock-plugin • Developer declare their ideal situation • Save resolved version • If tests pass, commit to SCM
  32. 32. {! "com.github.townsfolk:gradle-release": { ! ! ! "locked": "1.2", "requested": "1.2" },! "com.jfrog.bintray.gradle:gradle-bintray-plugin": { ! ! ! "locked": "0.3", "requested": "0.3" },! "com.netflix.nebula:nebula-project-plugin": { ! ! ! "locked": "1.12.0", "requested": "1.12.+" },! "com.netflix.nebula:nebula-test": { ! ! ! "locked": "1.12.0", "requested": "1.12.+" },! "org.codehaus.groovy.modules.http-builder:http-builder": { ! ! ! "locked": "0.7.1", "requested": “latest.release" },! "org.jfrog.buildinfo:build-info-extractor-gradle": { ! ! ! "locked": "2.2.4", "requested": "2.2.+" }! } apply plugin: ‘gradle-dependency-lock'! lock.gradle ./gradlew generateLock
  33. 33. nebula-integtest-plugin • Sets up integTest source set • Adds integTestCompile and integTestRuntime configurations • Creates integrationTest task
  34. 34. nebula-project-plugin • Pull together other plugins • Responsible projects
  35. 35. nebula-plugin-plugin • Used by plugins • Strong opinions on how to publish • Force nebula-project- plugin on projects
  36. 36. gradle-ospackage-plugin • Merging ubuntu-packager-plugin and gradle-rpm- plugin • Uses CopySpec definition • Via just Java, generates RPMs and DEBs
  37. 37. ospackage {! ! os = LINUX! ! into '/opt/foo'! ! from ('dist') {! ! ! user 'builds'! ! ! exclude '**/*.md'! ! }! ! postInstall file('scripts/postInstall.sh')! }! ! buildRpm {! ! requires('bar', '2.2', GREATER | EQUAL)! ! from (‘build/metadata.properties’)! ! link(‘/etc/init.d/foo’, '/opt/foo/bin/foo.sysv',)! }! ! buildDeb {! ! link('/etc/init/foo', '/opt/foo/bin/foo.upstart')! } ospackage.gradle
  38. 38. gradle-override-plugin • Take command line arguments • Intelligently apply in afterEvaluate • E.g. -Nfindbugs.enabled=false
  39. 39. Internal Plugins • netflix-repos • nebula-ospackage • nebula-grails • nebula-findbugs, etc • ivyimport • nebula-fixexcludes • nebula-intellij
  40. 40. buildscript {! ! repositories { jcenter() }! ! dependencies { ! ! ! classpath ‘com.netflix.nebula:nebula-plugin-plugin:1.12.+'! ! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit: 3.2.0.201312181205-r'! ! }! }! ! description ‘Example Plugin'! apply plugin: ‘nebula-plugin'! ! contacts {! ‘jryan@netflix.com’ {! moniker 'Justin Ryan'! github 'quidryan'! }! } plugin.gradle
  41. 41. curl -s get.gvmtool.net | bash! gvm install lazybones! cd ~/Projects/github/nebula-plugins! lazybones create nebula-plugin <name-of-project>! cd <name-of-project>! git init! git remote add origin git@github.com:nebula-plugins/<name-of- project>.git! ./gradlew clean build! git add -A! git add -f gradle/wrapper/gradle-wrapper.jar! git commit -m "Initial template”! git push --set-upstream origin master Making a plugin https://github.com/nebula-plugins/nebula- plugins.github.io/wiki/New-Plugins
  42. 42. Getting it out the door • Let “ensure” run • Run <name-of-project>-release • Link package to jcenter • Link package to gradle-plugins
  43. 43. Unopinionated Plugins
  44. 44. Opinionated Plugins
  45. 45. Gotchas • NamedDomainObjectSet • Debugging Tooling • File as @Output • afterEvaluate
  46. 46. Outstanding • CloudBees permissions • Bot to create repositories
  47. 47. Participating • Use individual plugins • Get on nebula-plugins Google Group • Move your plugin to nebula-plugins • Start a new plugin in nebula-plugins
  48. 48. @quidryan@quidryan http://www.slideshare.net/quidryan We’re hiring Justin Ryan <jryan@netflix.com> HOUSE of GRADLE

×