Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

{py}gradle

842 views

Published on

{py}gradle poster at PyCon 2016.

Published in: Technology
  • Be the first to comment

{py}gradle

  1. 1. Over the past 1.5 years LinkedIn has been using {py}gradle on over 800 different projects successfully. Given the success of these projects we are going to offer the plugin to the open source community. We plan to open source {py}gradle in it’s current state, after LinkedIn-isms are removed with a 0 major version. When it’s open sourced, we will also present the plan forward for 1.0 and beyond. OPEN SOURCE BUILDING A POLYGLOT SERVICE? A Polyglot Service has a backend API server written in Python that uses gunicorn, Flask, and other internal and external Python dependencies, it has a frontend server written in Ember.js that uses NPM under the hood, it uses Python clients generated from Rest.li interface definition language (IDL) using a Java tool, and must emit the proper metadata that our deployment system uses to configure, canary, and deploy our service. How can we build a Polyglot Service? generateJava generatePython buildJava buildEmber buildRestli buildPython build uploadMetadata publish uploadArtifacts packDeployable runPytest pipInstallDeps makeDists createVenv Gradle is an open source modular build system. Gradle started its life building JVM-based code, and gained additional traction when they moved to Android and C/C++ over the past few years. Gradle gives developers a rich API to implement plugins: the primitive units that compose a Gradle build system. A plugin developer is able to implement a DSL to model any process or language that they wish, ultimately enabling us to build polyglot build systems. A plugin can be reused, mixed and matched with other plugins, extended, configured, and more. Gradle’s plugin architecture lets us build better build systems. ABOUT GRADLE BUILD.GRADLE apply plugin: 'python' apply plugin: 'python-api-server' def insertFromFile = { filePath -> // Read a Python requirements.txt file... } dependencies { // Installation requirements from file installRequires insertFromFile('requirements.txt') // Installation requirements installRequires 'pypi:Flask:0.0.0' installRequires 'linkedin:utils:0.0.0' // Test requirements testsRequire 'pypi:mock:0.0.0' // Setup requirements setupRequires 'pypi:setuptools-git:0.0.0' // Front end static resources requirements resource project(path: ':ember-frontend', configuration: 'static') } - Dependency management for FREE! - No need to emit metadata. The build system does that for me. - Different languages work together. Some cases where Gradle works better for version conflict resolution than the traditional Python build workflow. - Python picks up a new version on each rebuild. Gradle can reproduce the builds. - Python gets version conflict for major version updates. Gradle picks up the highest. - Python gets version conflict when one branch of the graph blacklists version as bad, but another branch does not. Gradle reliably blacklists the bad version. In complex dependency systems, we may want to do what LinkedIn calls “semantic version” testing. Upon a change to D, we want to test its changes against C, F, and G. If the application does not pin all its dependencies, it can get into cascading version conflicts during dependency updates (see below). In this example we can see that C, F, and G all depend on D at dif- ferent major versions. What should we do? Dependency management happens on graphs, where nodes represent packages and edges represent ver- sions or ranges of versions on other nodes. We care about dependency management because it gives us the ability to reproduce builds and safely evolve software as it changes time over time. Dependency management is challenging to get right, and solutions exist on a spectrum. On one side, we can practice version pinning in which a dependency pins all direct and transitive dependencies. This approach achieves reproducible builds, but makes safely evolving difficult. On the other side, we can employ floating versions and only specify direct dependencies. This approach achieves fluidity in evolv- ing software but makes reproducible builds impossible and doesn’t make evolution necessarily safe. DEPENDENCY MANAGEMENT A B E D C G F A B E D C G F A B E D C G F ABOUT SEMANTIC VERSIONING [ V 1 2 5Major Minor Patch [ [ A {py}gradle project uses setuptools, vir- tualenv, and pip to maintain a development and build environment that makes use of ev- erything that a Python developer is already fa- miliar with: setup.py, setup.cfg, require- ments.txt, tox.ini, and more. A {py}gradle project looks and feels like a Python project and produces backwards-compatible artifacts that work with non-{py}gradle-based Python projects. HOW TO USE $ tree ├── build.gradle ├── gradlew ├── backend-api │ ├── build.gradle │ ├── setup.cfg │ ├── setup.py │ ├── src │ │ └── backend │ │ └── __init__.py │ ├── requirements.txt │ └── test │ └── test_backend.py ├── <subproject> │ └── ... └── settings.gradle $ ./gradlew build {py}gradle uses the power of Ivy for dependency management. Then, it goes out of your way and lets you work as a pure Python developer. After the first build, the standard virtualenv develope- ment workflow can be used. BUILD SYSTEM COMPARISON ABSTRACT As our Python development team grew over the years and the number of packages and interdepen- dencies grew from tens to hundreds to thousands, we found that we were struggling to maintain our code base using the standard tooling that was available to us: setuptools, virtualenv, and pip. At the same time, our Java development team converged on a new build system – Gradle – that al- lowed them to write extendable build logic. We found that Gradle wasn’t just for Java, and we wrote a powerful and idiomatic Python build system that reused much of the build logic as our Java peers, without changing the way our Python developers work. {py}gradle Stephen Holsapple (sholsapp@linkedin.com, , sholsapp) Zvezdan Petković (zpetkovic@linkedin.com, zvezdan) Ethan Hall (ethall@linkedin.com, ethankhall) Loren Carvalho (lcarvalho@linkedin.com, sixninetynine) Gradle Maven Ant Setuptools Buildout Pants CMake Powerful Dependency Resolution API Decoupled Dependency Metadata Ivy-based Metadata Pluggable Scriptable Human-readable Build File Natively Polyglot Generic Artifact Hosting Support (*) to a certain extent, or not easily * * * Gradle allows us to have multiple projects, each in a separate directory, for each language used. Each project uses the gradle plugin(s) specific to that language. The artifacts produced in a build of each project can be used in another project. The ordering of build tasks gives us the full control of the overall service build. For advanced use, one can exclude tasks from running, or use multiple languages in the same project.

×