Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python Pants Build System for Large Codebases


Published on

This talk is geared towards Infrastructure and system engineers who are interested in learning about structuring a large monorepo codebase, consisting of multiple micro services that share many dependencies. This talk will introduce Pants as a build system for such large monolithic codebase and how it ties with today’s container ecosystem principles.

Published in: Engineering
  • I’ve personally never heard of companies who can produce a paper for you until word got around among my college groupmates. My professor asked me to write a research paper based on a field I have no idea about. My research skills are also very poor. So, I thought I’d give it a try. I chose a writer who matched my writing style and fulfilled every requirement I proposed. I turned my paper in and I actually got a good grade. I highly recommend ⇒ ⇐
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Python Pants Build System for Large Codebases

  1. 1. Build System for Large Codebases Angad Singh PyCon SG 2016 @angadsg
  2. 2. NUS Computer Engineering (2009-13) SRE at Twitter (2013-14) DevOps at Viki (2014-..) Contact: Twitter: @angadsg About me
  3. 3. Agenda Code Organization Pain Points Pants as a build tool Python PEX format Code Examples
  4. 4. Service A Service B Service C Project Repo A Project Repo B Project Repo C Code Organization Shared code Does not scale well for a large number of microservices Complex method of sharing libraries (publishing artifacts, versioning hell)
  5. 5. Code Organization Libraries repository Service A Service B Service C Libraries as Code Units Single Lint, Build, Test and Release process Easy to coordinate changes across modules Easier to setup development environment Tests run across modules are run together Promote the idea of writing shareable code Monorepo - A repository with a defined structure for organizing reusable components of code
  6. 6. Pain Points Virtualenv to manage dependencies for python projects is painful. Need something simpler. Need easier code sharing amongst projects. Fixing a bug in a function should not require changing versions of other downstream projects. Need standardization in testing and building process
  7. 7. Pants Build system for managing targets sharing a single repository Dependencies are managed in BUILD files that live alongside the code. History - Used to be a python wrapper around Ant build tool which generated build.xml files and handed the build files to ant. (Python + Ant = Pants) Later, rewritten to be an independent build tool with main support for JVM languages and Python.
  8. 8. Pants Define source tree - src/<lang> e.g. src/python/ BUILD files define targets at each leaf node in the source tree. DSL, which invokes python constructors in the background Targets can be either a binary (e.g PEX for python, JAR for Java) or a library which can be referenced by other targets.
  9. 9. PEX PEX files - Python Executables, similar in idea to a virtual environment. Generate Immutable artifacts, that will run on any server Run targets locally, without maintaining complex virtual environments Easier debugging through standardized versioning of 3rdparty dependencies
  10. 10. BUILD file python_binary(name=’cli’, dependencies = [ “src/python/3rdparty/python:requests”, “:shared_lib” ] source=’’ ) python_library(name=’shared_lib’, dependencies = [ “src/python/3rdparty/python:fabric” ] source=’’ )
  11. 11. Examples A simple python flask application Twitter Commons
  12. 12. Thank you