Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 48

Leveraging Gradle @ Netflix (Madrid GUG Feb 2, 2021)

1

Share

Download to read offline

Gradle is an open-source build automation tool focused on flexibility, build reproducibility and performance. Over the years, this tool has evolved and introduced new concepts and features around dependency management, publication and other aspects on build and release of artifacts for the Java platform.

Keeping up to date with all these features across several projects can be challenging. How do you make sure that all your projects can be upgraded to the latest version of Gradle? What if you have thousands of projects and hundreds of engineers? How can you abstract common tasks for them and make sure that new releases work as expected?

At Netflix, we built Nebula, a collection of Gradle plugins that helps engineers remove boilerplate in Gradle build files, and makes building software the Netflix way easy. This reduces the cognitive load on developers, allowing them to focus on writing code.

In this talk, I’ll share with you our philosophy on how to build JVM artifacts and the pieces that help us boost the productivity of engineers at Netflix. I’ll talk about:
- What is Nebula
- What are the common problems we face and try to solve
- How we distribute it to every JVM engineer
- How we ensure that Nebula/Gradle changes do not break builds so we can ship new features with confidence at Netflix.

---

About Roberto: Roberto Perez Alcolea is a Senior Software Engineer at Netflix. He is a member of the Java Platform team providing the core language and framework components that enable the Java community at Netflix. He's an active maintainer of Netflix Nebula Plugins (https://nebula-plugins.github.io/) and passionate about Gradle. Prior to that, he spent several years building high performant APIs with Ratpack and web applications using Grails.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Leveraging Gradle @ Netflix (Madrid GUG Feb 2, 2021)

  1. 1. Leveragin g Gradl e @ Netflix February 2, 2021
  2. 2. Who am I? Leveraging Gradle @ Net fl ix Roberto Pérez Alcole a Senior Software Enginee r Java Platform @ Netfli x rperezalcolea@netflix.com @rpalcolea
  3. 3. JVM Buil d @ Netflix Leveraging Gradl e @ Net fl ix
  4. 4. •Many repositories (~3.2k ) •Binary integration (JARs ) •Microservices with fat client s •Hundreds of engineer s •Ivy + Ant in the past. Now we use Gradle ! •Java, Groovy, Scala, Kotlin, Clojure Leveraging Gradle @ Net fl ix
  5. 5. Builds per wee k Leveraging Gradl e @ Net fl ix
  6. 6. Artifact Publishin g Leveraging Gradl e @ Net fl ix We publish ~9k artifacts per day !
  7. 7. Consumers rely on latest releas e Leveraging Gradl e @ Net fl ix
  8. 8. JVM Buil d Problems & Challenge s @ Netflix Leveraging Gradl e @ Net fl ix
  9. 9. Broken Build s •Add/upgrade dependencie s •Unaligned dependencie s •Dependencies with bugs or security vulnerabilitie s •Gradle plugin is not compatibl e Leveraging Gradl e @ Net fl ix
  10. 10. Slow Build s •Dependency resolutio n •Plugins and custom tasks impacting con fi guration tim e •Using excludes on dependencies declaration s •Lack of parallelization Leveraging Gradl e @ Net fl ix
  11. 11. Metadat a •Bad Ivy or Maven metadata can lead to a poisoned dependency graph . •Dynamic dependencie s •Bad scope s •Dependency declaration for shaded librarie s Leveraging Gradl e @ Net fl ix
  12. 12. Reproducible Build s •Same binary dependencie s •File orde r •Avoid preserving timestamp s Leveraging Gradl e @ Net fl ix
  13. 13. Packagin g •Make sure that a java application can be deployed with easiness in our infrastructure •VMs on EC 2 •Docker containers in Titu s Leveraging Gradl e @ Net fl ix
  14. 14. Plugin Ecosyste m •Third party plugin compatibility with new versions of Gradl e •Strong opinions from external and internal plugins (forcing dependencies ) •Upgrade well known plugins for existing build s Leveraging Gradl e @ Net fl ix
  15. 15. Fast evolution of Gradl e •Deprecated APIs and the need to migrate folks to use the new one s •It isn’t only about the DSL but also for plugin authors at Net fl i x •Introduce new features and keep up with Gradle release s Leveraging Gradl e @ Net fl ix
  16. 16. Leveraging Gradl e @ Net fl ix Publishers lack visibility • Who will I break ? • Who is consuming my library ? • Who is using this API that I want to change?
  17. 17. What ca n We do? Leveraging Gradl e @ Net fl ix
  18. 18. Nebula t o The rescue Leveraging Gradl e @ Net fl ix
  19. 19. Leveraging Gradl e @ Net fl ix What is Nebula ? •Gradle plugins and internal custom distribution built to eliminate boilerplate build logic and provide sane convention s •Dependency managemen t •Publishin g •OS Packagin g •Lintin g •Dependency lockin g •And many others !
  20. 20. This plugin provides general purpose rule types on top of Gradle resolution strategies and module metadata, allowing rules to be published, versioned, shared between projects, and optionally dependency locked. •Duplicate classes caused by changes to group or artifact ids, without renaming package s •Duplicate classes caused by bundle dependencies, which do not con fl ict resolve against the 'regular' dependencies for that library •Lack of version alignment between libraries, where version alignment is needed for compatibility •Ensuring a minimum version of a library Resolution Rules Plugin Leveraging Gradl e @ Net fl ix
  21. 21. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/resolution-rules-plugin-example https://github.com/nebula-plugins/gradle-resolution-rules-plugin https://github.com/nebula-plugins/gradle-resolution-rules Resolution Rules Plugin
  22. 22. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/release-example https://github.com/nebula-plugins/nebula-publishing-plugin Publishing Plugin
  23. 23. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/lint-example https://github.com/nebula-plugins/gradle-lint-plugin Lint Plugin
  24. 24. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/ospackaging-example https://github.com/nebula-plugins/gradle-ospackage-plugin OSPackage Plugin
  25. 25. Leveraging Gradl e @ Net fl ix
  26. 26. How do we distribute Nebula? Leveraging Gradl e @ Net fl ix
  27. 27. Gradle custo m Distributions Leveraging Gradle @ Net fl ix • Apply internal and OSS plugin s • Custom init script s • Configure repositorie s • Configure status schemes (snapshot, candidate, release ) • Enable metrics collectio n • Enable/disable feature s • Default gradle.properties
  28. 28. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/custom-distribution-example Custom Gradle distribution
  29. 29. How do we validate new versions of Nebula? Leveraging Gradl e @ Net fl ix
  30. 30. •Leverage Niagara to clone every Nebula based project at Gradle on its own container and build it using the new version of Nebula . •Gradle nightly release s •Gradle candidate releases •Leverage Niagara to clone our OSS plugins and test them against nightly releases of Gradle. Acceptance Testin g Leveraging Gradl e @ Net fl ix
  31. 31. Leveraging Gradl e @ Net fl ix Nebula acceptance tests
  32. 32. How do we upgrade projects? Leveraging Gradl e @ Net fl ix
  33. 33. •Automatic Pull requests : •Dependency lock s •Nebula wrappe r •Lint rule s Managed Sourc e Evolving Continuous Integration at Netflix (Elise McCallum) | 2019 Leveraging Gradl e @ Net fl ix
  34. 34. Faster builds Leveraging Gradl e @ Net fl ix
  35. 35. •Build Cach e •Shared Read-only Dependency Cach e •Build Scan s •Gradle Enterpris e •Parallel execution Gradle Feature s Leveraging Gradl e @ Net fl ix
  36. 36. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/shared-dependency-cache- example Shared Dependency Cache
  37. 37. Leveraging Gradl e @ Net fl ix Gradle Enterprise (values)
  38. 38. Leveraging Gradl e @ Net fl ix Gradle Enterprise (Tags)
  39. 39. Did someone said visibility? Leveraging Gradl e @ Net fl ix
  40. 40. •Collection of services and UIs that enable artifact observability and the ability to effect change in the Net fl ix ecosystem . •High level goal is to decrease time and effort needed to propagate change through the Net fl ix ecosyste m •Useful for Library deprecation and Security Vulnerabilitie s Astri d Leveraging Gradl e @ Net fl ix
  41. 41. Astrid - Artifact Managemen t Leveraging Gradl e @ Net fl ix
  42. 42. Astrid - Artifact Managemen t Leveraging Gradl e @ Net fl ix
  43. 43. Astrid - Campaign s Leveraging Gradl e @ Net fl ix
  44. 44. What’s next for us? Leveraging Gradl e @ Net fl ix
  45. 45. •Distributed Testin g •Java Toolchain s •Gradle Metadata Adoptio n •Con fi guration Cache Gradle Features in Our Horizo n Leveraging Gradl e @ Net fl ix
  46. 46. Thank
 You. Q&A rperezalcolea@netflix.com @rpalcolea https://github.com/rpalcolea

×