SlideShare a Scribd company logo
Build System for Large
Codebases
Angad Singh
PyCon SG 2016
@angadsg
NUS Computer Engineering (2009-13)
SRE at Twitter (2013-14)
DevOps at Viki (2014-..)
Contact: t.co/as
Twitter: @angadsg
About me
Agenda
Code Organization
Pain Points
Pants as a build tool
Python PEX format
Code Examples
Service A Service B Service C
Project
Repo A
Project
Repo B
Project
Repo C
Code Organization
Shared code
Does not scale well for a large number of
microservices
Complex method of sharing libraries (publishing
artifacts, versioning hell)
Code Organization
Libraries repository
Service A
Service B
Service C
Libraries as Code Units
Single Lint, Build, Test and Release process
Easy to coordinate changes across modules
Easier to setup development environment
Tests run across modules are run together
Promote the idea of writing shareable code
Monorepo
- A repository with a defined structure for organizing reusable components of
code
Pain Points
Virtualenv to manage dependencies for python projects is painful. Need
something simpler.
Need easier code sharing amongst projects. Fixing a bug in a function should not
require changing versions of other downstream projects.
Need standardization in testing and building process
Pants
Build system for managing targets sharing
a single repository
Dependencies are managed in BUILD files
that live alongside the code.
History - Used to be a python wrapper
around Ant build tool which generated
build.xml files and handed the build files
to ant. (Python + Ant = Pants)
Later, rewritten to be an independent build
tool with main support for JVM languages
and Python.
Pants
Define source tree - src/<lang> e.g. src/python/
BUILD files define targets at each leaf node in the source tree.
DSL, which invokes python constructors in the background
Targets can be either a binary (e.g PEX for python, JAR for Java) or a library
which can be referenced by other targets.
PEX
PEX files - Python Executables, similar in idea to a virtual environment.
Generate Immutable artifacts, that will run on any server
Run targets locally, without maintaining complex virtual environments
Easier debugging through standardized versioning of 3rdparty dependencies
BUILD file
python_binary(name=’cli’,
dependencies = [
“src/python/3rdparty/python:requests”,
“:shared_lib”
]
source=’cli.py’
)
python_library(name=’shared_lib’,
dependencies = [
“src/python/3rdparty/python:fabric”
]
source=’lib.py’
)
Examples
A simple python flask application
https://github.com/angad/pants-flask
Twitter Commons
https://github.com/twitter/commons
Thank you

More Related Content

What's hot

Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
AnirudhaGaikwad4
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
ManishJha237
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
Agung Wahyudi
 
Fluent-bit
Fluent-bitFluent-bit
Fluent-bit
eventdotsjp
 
Introduction to python programming, Why Python?, Applications of Python
Introduction to python programming, Why Python?, Applications of PythonIntroduction to python programming, Why Python?, Applications of Python
Introduction to python programming, Why Python?, Applications of Python
Pro Guide
 
Collaborative Real-Time Editing: Shane Carr
Collaborative Real-Time Editing: Shane CarrCollaborative Real-Time Editing: Shane Carr
Collaborative Real-Time Editing: Shane Carr
Redis Labs
 
PHP Internals
PHP InternalsPHP Internals
PHP Internals
Nico Loubser
 
Introduction To Python
Introduction To PythonIntroduction To Python
Introduction To Python
Vanessa Rene
 
A Dive Into ELF Binaries
A Dive Into ELF BinariesA Dive Into ELF Binaries
A Dive Into ELF Binaries
Bhashit Pandya
 
Null
NullNull
Cmpe202 01 Research
Cmpe202 01 ResearchCmpe202 01 Research
Cmpe202 01 Research
vladimirkorshak
 
Crosslingual search-engine
Crosslingual search-engineCrosslingual search-engine
Crosslingual search-engine
Carlos Badenes-Olmedo
 
Experience protocol buffer on android
Experience protocol buffer on androidExperience protocol buffer on android
Experience protocol buffer on android
Richard Chang
 
Getting Started with Python
Getting Started with PythonGetting Started with Python
Getting Started with Python
Sankhya_Analytics
 
Introduction to python for Beginners
Introduction to python for Beginners Introduction to python for Beginners
Introduction to python for Beginners
Sujith Kumar
 
OPEN SOURCE SOFTWARE
OPEN SOURCE SOFTWAREOPEN SOURCE SOFTWARE
OPEN SOURCE SOFTWARE
ritajindal2
 
[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów
[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów
[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów
Future Processing
 
Solr fusion lt elag2014
Solr fusion lt elag2014Solr fusion lt elag2014
Solr fusion lt elag2014
Leander Seige
 
Cvpr2010 open source vision software, intro and training part vi robot operat...
Cvpr2010 open source vision software, intro and training part vi robot operat...Cvpr2010 open source vision software, intro and training part vi robot operat...
Cvpr2010 open source vision software, intro and training part vi robot operat...
zukun
 
Domain specific languages in eclipse with Xtext (Zeus, UGent)
Domain specific languages in eclipse with Xtext (Zeus, UGent)Domain specific languages in eclipse with Xtext (Zeus, UGent)
Domain specific languages in eclipse with Xtext (Zeus, UGent)
Sigasi
 

What's hot (20)

Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
 
Fluent-bit
Fluent-bitFluent-bit
Fluent-bit
 
Introduction to python programming, Why Python?, Applications of Python
Introduction to python programming, Why Python?, Applications of PythonIntroduction to python programming, Why Python?, Applications of Python
Introduction to python programming, Why Python?, Applications of Python
 
Collaborative Real-Time Editing: Shane Carr
Collaborative Real-Time Editing: Shane CarrCollaborative Real-Time Editing: Shane Carr
Collaborative Real-Time Editing: Shane Carr
 
PHP Internals
PHP InternalsPHP Internals
PHP Internals
 
Introduction To Python
Introduction To PythonIntroduction To Python
Introduction To Python
 
A Dive Into ELF Binaries
A Dive Into ELF BinariesA Dive Into ELF Binaries
A Dive Into ELF Binaries
 
Null
NullNull
Null
 
Cmpe202 01 Research
Cmpe202 01 ResearchCmpe202 01 Research
Cmpe202 01 Research
 
Crosslingual search-engine
Crosslingual search-engineCrosslingual search-engine
Crosslingual search-engine
 
Experience protocol buffer on android
Experience protocol buffer on androidExperience protocol buffer on android
Experience protocol buffer on android
 
Getting Started with Python
Getting Started with PythonGetting Started with Python
Getting Started with Python
 
Introduction to python for Beginners
Introduction to python for Beginners Introduction to python for Beginners
Introduction to python for Beginners
 
OPEN SOURCE SOFTWARE
OPEN SOURCE SOFTWAREOPEN SOURCE SOFTWARE
OPEN SOURCE SOFTWARE
 
[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów
[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów
[FDD 2016] Marek Śledziński - Microsoft Windows a sprawa kontenerów
 
Solr fusion lt elag2014
Solr fusion lt elag2014Solr fusion lt elag2014
Solr fusion lt elag2014
 
Cvpr2010 open source vision software, intro and training part vi robot operat...
Cvpr2010 open source vision software, intro and training part vi robot operat...Cvpr2010 open source vision software, intro and training part vi robot operat...
Cvpr2010 open source vision software, intro and training part vi robot operat...
 
Domain specific languages in eclipse with Xtext (Zeus, UGent)
Domain specific languages in eclipse with Xtext (Zeus, UGent)Domain specific languages in eclipse with Xtext (Zeus, UGent)
Domain specific languages in eclipse with Xtext (Zeus, UGent)
 

Viewers also liked

Business selectors
Business selectorsBusiness selectors
Business selectors
benwaine
 
Apostila De Dispositivos EléTricos
Apostila De Dispositivos EléTricosApostila De Dispositivos EléTricos
Apostila De Dispositivos EléTricos
elkbcion
 
Linux Malware Analysis
Linux Malware Analysis	Linux Malware Analysis
Linux Malware Analysis
Cysinfo Cyber Security Community
 
Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)
Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)
Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)
Gaurav Srivastav
 
Bridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and KafkaBridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and Kafka
Pengfei (Jason) Li
 
Platform - Technical architecture
Platform - Technical architecturePlatform - Technical architecture
Platform - Technical architecture
David Rundle
 
CloudStack EU user group - Trillian
CloudStack EU user group - TrillianCloudStack EU user group - Trillian
CloudStack EU user group - Trillian
ShapeBlue
 
Java standards in WCM
Java standards in WCMJava standards in WCM
Java standards in WCM
Paolo Mottadelli
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
Brian Brazil
 
ITV& Bashton
ITV& Bashton ITV& Bashton
ITV& Bashton
Amazon Web Services
 
Reactive Cloud Security | AWS Public Sector Summit 2016
Reactive Cloud Security | AWS Public Sector Summit 2016Reactive Cloud Security | AWS Public Sector Summit 2016
Reactive Cloud Security | AWS Public Sector Summit 2016
Amazon Web Services
 
Apache Ambari: Managing Hadoop and YARN
Apache Ambari: Managing Hadoop and YARNApache Ambari: Managing Hadoop and YARN
Apache Ambari: Managing Hadoop and YARN
Hortonworks
 
Automated Infrastructure Security: Monitoring using FOSS
Automated Infrastructure Security: Monitoring using FOSSAutomated Infrastructure Security: Monitoring using FOSS
Automated Infrastructure Security: Monitoring using FOSS
Sonatype
 
Choosing the right data storage in the Cloud.
Choosing the right data storage in the Cloud. Choosing the right data storage in the Cloud.
Choosing the right data storage in the Cloud.
Amazon Web Services
 
Roxar Multiphase Meter
Roxar Multiphase MeterRoxar Multiphase Meter
Roxar Multiphase Meter
ali_elkaseh
 
NSM (Network Security Monitoring) - Tecland Chapeco
NSM (Network Security Monitoring) - Tecland ChapecoNSM (Network Security Monitoring) - Tecland Chapeco
NSM (Network Security Monitoring) - Tecland Chapeco
Rodrigo Montoro
 
Security For Humans
Security For HumansSecurity For Humans
Security For Humans
conjur_inc
 
Sunbrella Ottomans by Outdoor Elegance
Sunbrella Ottomans by Outdoor EleganceSunbrella Ottomans by Outdoor Elegance
Sunbrella Ottomans by Outdoor Elegance
OutdoorEleganceAus
 
Aws + Puppet = Dynamic Scale
Aws + Puppet = Dynamic ScaleAws + Puppet = Dynamic Scale
Aws + Puppet = Dynamic Scale
Puppet
 
Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...
Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...
Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...
Precisely
 

Viewers also liked (20)

Business selectors
Business selectorsBusiness selectors
Business selectors
 
Apostila De Dispositivos EléTricos
Apostila De Dispositivos EléTricosApostila De Dispositivos EléTricos
Apostila De Dispositivos EléTricos
 
Linux Malware Analysis
Linux Malware Analysis	Linux Malware Analysis
Linux Malware Analysis
 
Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)
Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)
Gaurav dev ops (AWS, Linux, Automation-ansible, jenkins:CI and CD:Ansible)
 
Bridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and KafkaBridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and Kafka
 
Platform - Technical architecture
Platform - Technical architecturePlatform - Technical architecture
Platform - Technical architecture
 
CloudStack EU user group - Trillian
CloudStack EU user group - TrillianCloudStack EU user group - Trillian
CloudStack EU user group - Trillian
 
Java standards in WCM
Java standards in WCMJava standards in WCM
Java standards in WCM
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
ITV& Bashton
ITV& Bashton ITV& Bashton
ITV& Bashton
 
Reactive Cloud Security | AWS Public Sector Summit 2016
Reactive Cloud Security | AWS Public Sector Summit 2016Reactive Cloud Security | AWS Public Sector Summit 2016
Reactive Cloud Security | AWS Public Sector Summit 2016
 
Apache Ambari: Managing Hadoop and YARN
Apache Ambari: Managing Hadoop and YARNApache Ambari: Managing Hadoop and YARN
Apache Ambari: Managing Hadoop and YARN
 
Automated Infrastructure Security: Monitoring using FOSS
Automated Infrastructure Security: Monitoring using FOSSAutomated Infrastructure Security: Monitoring using FOSS
Automated Infrastructure Security: Monitoring using FOSS
 
Choosing the right data storage in the Cloud.
Choosing the right data storage in the Cloud. Choosing the right data storage in the Cloud.
Choosing the right data storage in the Cloud.
 
Roxar Multiphase Meter
Roxar Multiphase MeterRoxar Multiphase Meter
Roxar Multiphase Meter
 
NSM (Network Security Monitoring) - Tecland Chapeco
NSM (Network Security Monitoring) - Tecland ChapecoNSM (Network Security Monitoring) - Tecland Chapeco
NSM (Network Security Monitoring) - Tecland Chapeco
 
Security For Humans
Security For HumansSecurity For Humans
Security For Humans
 
Sunbrella Ottomans by Outdoor Elegance
Sunbrella Ottomans by Outdoor EleganceSunbrella Ottomans by Outdoor Elegance
Sunbrella Ottomans by Outdoor Elegance
 
Aws + Puppet = Dynamic Scale
Aws + Puppet = Dynamic ScaleAws + Puppet = Dynamic Scale
Aws + Puppet = Dynamic Scale
 
Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...
Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...
Machine Learning & IT Service Intelligence for the Enterprise: The Future is ...
 

Similar to Python Pants Build System for Large Codebases

Visual studio
Visual studioVisual studio
Visual studio
anupathak17jul
 
.NET 4 Demystified - Sandeep Joshi
.NET 4 Demystified - Sandeep Joshi.NET 4 Demystified - Sandeep Joshi
.NET 4 Demystified - Sandeep Joshi
Spiffy
 
CHX PYTHON INTRO
CHX PYTHON INTROCHX PYTHON INTRO
CHX PYTHON INTRO
Kai Liu
 
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
Maarten Balliauw
 
Socket programming-in-python
Socket programming-in-pythonSocket programming-in-python
Socket programming-in-python
Yuvaraja Ravi
 
Introduction to .NET with C# @ university of wayamba
Introduction to .NET with C# @ university of wayambaIntroduction to .NET with C# @ university of wayamba
Introduction to .NET with C# @ university of wayamba
Prageeth Sandakalum
 
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...
Maarten Balliauw
 
Net framework
Net frameworkNet framework
Net framework
jhsri
 
Anaconda Python KNIME & Orange Installation
Anaconda Python KNIME & Orange InstallationAnaconda Python KNIME & Orange Installation
Anaconda Python KNIME & Orange Installation
Girinath Pillai
 
Introduction to Python Basics Programming
Introduction to Python Basics ProgrammingIntroduction to Python Basics Programming
Introduction to Python Basics Programming
Raveendra R
 
Python_final_print_vison_academy_9822506209.pdf
Python_final_print_vison_academy_9822506209.pdfPython_final_print_vison_academy_9822506209.pdf
Python_final_print_vison_academy_9822506209.pdf
VisionAcademyProfSac
 
Localization (l10n) - The Process
Localization (l10n) - The ProcessLocalization (l10n) - The Process
Localization (l10n) - The Process
Sundeep Anand
 
Socket Programming In Python
Socket Programming In PythonSocket Programming In Python
Socket Programming In Python
didip
 
Researh toolbox - Data analysis with python
Researh toolbox  - Data analysis with pythonResearh toolbox  - Data analysis with python
Researh toolbox - Data analysis with python
Umair ul Hassan
 
Researh toolbox-data-analysis-with-python
Researh toolbox-data-analysis-with-pythonResearh toolbox-data-analysis-with-python
Researh toolbox-data-analysis-with-python
Waternomics
 
Dot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part iDot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part i
Rakesh Joshi
 
Dot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part iDot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part i
Rakesh Joshi
 
2018 20 best id es for python programming
2018 20 best id es for python programming2018 20 best id es for python programming
2018 20 best id es for python programming
SyedBrothersRealEsta
 
20 best ide's for python programming in 2018
 20 best ide's for python programming in 2018 20 best ide's for python programming in 2018
20 best ide's for python programming in 2018
Qamar Abbas
 
Pottnet Meetup Essen - ASP.Net Core
Pottnet Meetup Essen - ASP.Net CorePottnet Meetup Essen - ASP.Net Core
Pottnet Meetup Essen - ASP.Net Core
Malte Lantin
 

Similar to Python Pants Build System for Large Codebases (20)

Visual studio
Visual studioVisual studio
Visual studio
 
.NET 4 Demystified - Sandeep Joshi
.NET 4 Demystified - Sandeep Joshi.NET 4 Demystified - Sandeep Joshi
.NET 4 Demystified - Sandeep Joshi
 
CHX PYTHON INTRO
CHX PYTHON INTROCHX PYTHON INTRO
CHX PYTHON INTRO
 
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...
 
Socket programming-in-python
Socket programming-in-pythonSocket programming-in-python
Socket programming-in-python
 
Introduction to .NET with C# @ university of wayamba
Introduction to .NET with C# @ university of wayambaIntroduction to .NET with C# @ university of wayamba
Introduction to .NET with C# @ university of wayamba
 
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...
 
Net framework
Net frameworkNet framework
Net framework
 
Anaconda Python KNIME & Orange Installation
Anaconda Python KNIME & Orange InstallationAnaconda Python KNIME & Orange Installation
Anaconda Python KNIME & Orange Installation
 
Introduction to Python Basics Programming
Introduction to Python Basics ProgrammingIntroduction to Python Basics Programming
Introduction to Python Basics Programming
 
Python_final_print_vison_academy_9822506209.pdf
Python_final_print_vison_academy_9822506209.pdfPython_final_print_vison_academy_9822506209.pdf
Python_final_print_vison_academy_9822506209.pdf
 
Localization (l10n) - The Process
Localization (l10n) - The ProcessLocalization (l10n) - The Process
Localization (l10n) - The Process
 
Socket Programming In Python
Socket Programming In PythonSocket Programming In Python
Socket Programming In Python
 
Researh toolbox - Data analysis with python
Researh toolbox  - Data analysis with pythonResearh toolbox  - Data analysis with python
Researh toolbox - Data analysis with python
 
Researh toolbox-data-analysis-with-python
Researh toolbox-data-analysis-with-pythonResearh toolbox-data-analysis-with-python
Researh toolbox-data-analysis-with-python
 
Dot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part iDot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part i
 
Dot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part iDot net-interview-questions-and-answers part i
Dot net-interview-questions-and-answers part i
 
2018 20 best id es for python programming
2018 20 best id es for python programming2018 20 best id es for python programming
2018 20 best id es for python programming
 
20 best ide's for python programming in 2018
 20 best ide's for python programming in 2018 20 best ide's for python programming in 2018
20 best ide's for python programming in 2018
 
Pottnet Meetup Essen - ASP.Net Core
Pottnet Meetup Essen - ASP.Net CorePottnet Meetup Essen - ASP.Net Core
Pottnet Meetup Essen - ASP.Net Core
 

Recently uploaded

Textile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdfTextile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdf
NazakatAliKhoso2
 
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdfBPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
MIGUELANGEL966976
 
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...Electric vehicle and photovoltaic advanced roles in enhancing the financial p...
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...
IJECEIAES
 
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
IEEE Aerospace and Electronic Systems Society as a Graduate Student MemberIEEE Aerospace and Electronic Systems Society as a Graduate Student Member
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
VICTOR MAESTRE RAMIREZ
 
Computational Engineering IITH Presentation
Computational Engineering IITH PresentationComputational Engineering IITH Presentation
Computational Engineering IITH Presentation
co23btech11018
 
NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT
NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENTNATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT
NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT
Addu25809
 
International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...
gerogepatton
 
Understanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine LearningUnderstanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine Learning
SUTEJAS
 
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
shadow0702a
 
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
171ticu
 
spirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptxspirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptx
Madan Karki
 
一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理
一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理
一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理
ecqow
 
132/33KV substation case study Presentation
132/33KV substation case study Presentation132/33KV substation case study Presentation
132/33KV substation case study Presentation
kandramariana6
 
Generative AI leverages algorithms to create various forms of content
Generative AI leverages algorithms to create various forms of contentGenerative AI leverages algorithms to create various forms of content
Generative AI leverages algorithms to create various forms of content
Hitesh Mohapatra
 
Engine Lubrication performance System.pdf
Engine Lubrication performance System.pdfEngine Lubrication performance System.pdf
Engine Lubrication performance System.pdf
mamamaam477
 
Manufacturing Process of molasses based distillery ppt.pptx
Manufacturing Process of molasses based distillery ppt.pptxManufacturing Process of molasses based distillery ppt.pptx
Manufacturing Process of molasses based distillery ppt.pptx
Madan Karki
 
gray level transformation unit 3(image processing))
gray level transformation unit 3(image processing))gray level transformation unit 3(image processing))
gray level transformation unit 3(image processing))
shivani5543
 
LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant
LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by AnantLLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant
LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant
Anant Corporation
 
Material for memory and display system h
Material for memory and display system hMaterial for memory and display system h
Material for memory and display system h
gowrishankartb2005
 
Advanced control scheme of doubly fed induction generator for wind turbine us...
Advanced control scheme of doubly fed induction generator for wind turbine us...Advanced control scheme of doubly fed induction generator for wind turbine us...
Advanced control scheme of doubly fed induction generator for wind turbine us...
IJECEIAES
 

Recently uploaded (20)

Textile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdfTextile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdf
 
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdfBPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
 
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...Electric vehicle and photovoltaic advanced roles in enhancing the financial p...
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...
 
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
IEEE Aerospace and Electronic Systems Society as a Graduate Student MemberIEEE Aerospace and Electronic Systems Society as a Graduate Student Member
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
 
Computational Engineering IITH Presentation
Computational Engineering IITH PresentationComputational Engineering IITH Presentation
Computational Engineering IITH Presentation
 
NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT
NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENTNATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT
NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT
 
International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...
 
Understanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine LearningUnderstanding Inductive Bias in Machine Learning
Understanding Inductive Bias in Machine Learning
 
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
 
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
 
spirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptxspirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptx
 
一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理
一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理
一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理
 
132/33KV substation case study Presentation
132/33KV substation case study Presentation132/33KV substation case study Presentation
132/33KV substation case study Presentation
 
Generative AI leverages algorithms to create various forms of content
Generative AI leverages algorithms to create various forms of contentGenerative AI leverages algorithms to create various forms of content
Generative AI leverages algorithms to create various forms of content
 
Engine Lubrication performance System.pdf
Engine Lubrication performance System.pdfEngine Lubrication performance System.pdf
Engine Lubrication performance System.pdf
 
Manufacturing Process of molasses based distillery ppt.pptx
Manufacturing Process of molasses based distillery ppt.pptxManufacturing Process of molasses based distillery ppt.pptx
Manufacturing Process of molasses based distillery ppt.pptx
 
gray level transformation unit 3(image processing))
gray level transformation unit 3(image processing))gray level transformation unit 3(image processing))
gray level transformation unit 3(image processing))
 
LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant
LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by AnantLLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant
LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant
 
Material for memory and display system h
Material for memory and display system hMaterial for memory and display system h
Material for memory and display system h
 
Advanced control scheme of doubly fed induction generator for wind turbine us...
Advanced control scheme of doubly fed induction generator for wind turbine us...Advanced control scheme of doubly fed induction generator for wind turbine us...
Advanced control scheme of doubly fed induction generator for wind turbine us...
 

Python Pants Build System for Large Codebases

  • 1. Build System for Large Codebases Angad Singh PyCon SG 2016 @angadsg
  • 2. NUS Computer Engineering (2009-13) SRE at Twitter (2013-14) DevOps at Viki (2014-..) Contact: t.co/as Twitter: @angadsg About me
  • 3. Agenda Code Organization Pain Points Pants as a build tool Python PEX format Code Examples
  • 4. Service A Service B Service C Project Repo A Project Repo B Project Repo C Code Organization Shared code Does not scale well for a large number of microservices Complex method of sharing libraries (publishing artifacts, versioning hell)
  • 5. Code Organization Libraries repository Service A Service B Service C Libraries as Code Units Single Lint, Build, Test and Release process Easy to coordinate changes across modules Easier to setup development environment Tests run across modules are run together Promote the idea of writing shareable code Monorepo - A repository with a defined structure for organizing reusable components of code
  • 6. Pain Points Virtualenv to manage dependencies for python projects is painful. Need something simpler. Need easier code sharing amongst projects. Fixing a bug in a function should not require changing versions of other downstream projects. Need standardization in testing and building process
  • 7. Pants Build system for managing targets sharing a single repository Dependencies are managed in BUILD files that live alongside the code. History - Used to be a python wrapper around Ant build tool which generated build.xml files and handed the build files to ant. (Python + Ant = Pants) Later, rewritten to be an independent build tool with main support for JVM languages and Python.
  • 8. Pants Define source tree - src/<lang> e.g. src/python/ BUILD files define targets at each leaf node in the source tree. DSL, which invokes python constructors in the background Targets can be either a binary (e.g PEX for python, JAR for Java) or a library which can be referenced by other targets.
  • 9. PEX PEX files - Python Executables, similar in idea to a virtual environment. Generate Immutable artifacts, that will run on any server Run targets locally, without maintaining complex virtual environments Easier debugging through standardized versioning of 3rdparty dependencies
  • 10. BUILD file python_binary(name=’cli’, dependencies = [ “src/python/3rdparty/python:requests”, “:shared_lib” ] source=’cli.py’ ) python_library(name=’shared_lib’, dependencies = [ “src/python/3rdparty/python:fabric” ] source=’lib.py’ )
  • 11. Examples A simple python flask application https://github.com/angad/pants-flask Twitter Commons https://github.com/twitter/commons

Editor's Notes

  1. I am Angad and today I am going to tell you about build systems for large code bases.
  2. First, about me - I graduated from NUS after attending many tutorials/lectures in this very room. I joined twitter after I graduated, to work as a Site Reliability Engineer. I came back to Singapore in 2014 to work with Viki as an infrastructure and devops lead.
  3. I am going to talk about Code Organization for large codebases first and then some of the Pain points that need to be solved for improved developer productivity. We will go over the Pants as a build tool and how it generates easily portable PEX files. We will then go over some examples of pants usage. I might be talking about Pants here as I am most familiar with it and it is written in python - but a build system for large codebases is a common concept amongst large companies and there are other open source tools as well such as Buck by facebook, Bazel by google and i urge you to explore them as well. In the end, using a build tool is far better than not using one.
  4. Lets start with Code organization. When you are a small team or a startup, a very basic and intuitive way to organize code is to have project repositories for each of your microservices. And microservices is a great idea if your team is growing and you need separation of responsibilities. The popularization of microservices has led to a growth of some bad practices as well. Microservices does not mean Microrepositories. Splitting a monolith service into a microservice is the trend of the day and it will be very tempting to start by splitting the source into multiple micro repositories. With github offering unlimited free private repositories, this sounds like a logical and tempting solution. Now this is good because all your developers can create as many repositories as they want and start writing a lot of code for your company. But this does not scale well for a large company. Success of a large team is heavily dependent on building on top of knowledge and code of other people. When there is a lot of shared code, Juggling a multimodule project over multiple repositories can be quite painful. You end up using some form of artifact sharing and end up in a versioning hell where all services are using a different version of the shared library. And if you want to fix some bug in a shared library, you have to update the version in all downstream repositories. That is a productivity nightmare.
  5. So instead of thinking of units of code as projects, think of libraries as the units of code. Now each of the services can just be composed of these library units. We can have single lint, build, test and release process. We can easily enforce a style guide as all the code is in one place. Much easier to setup a development environment. This promotes the idea of writing code that is shareable and reusable from the beginning. This is quite a popular concept amongst large companies. This started with Google and then Facebook, Twitter and a bunch of companies have Monorepo. Which is a large repository with all the code, with a defined structure for organizing reusable components.
  6. So we now know that there is some benefit of thinking of code as libraries which can be reused. In summary, we need a system which can solve the following pain points. First, we need an easier way of sharing code. Next we need standardization in testing and building process. Next, I have spent a fare share of time managing virtual environments for python projects and that is quite painful if you have multiple python projects. We need something more automated.
  7. So here comes Pants. Pants is a build tool with support for multiple languages, writen in Python. It was developed at Twitter and Foursquare to manage multiple build targets in a single repository. Dependencies for pants are managed in BUILD files that live alongside the code. You might be wondering why it is called Pants and why is it such a weird name for a build tool. Pants started as a build tool helper for Ant, a Java build tool. It used to be a simple tool that used to spew out hundreds of build.xml files and then invoke ant to work on them. Hence the name Pants from Python + Ant. But then it was rewritten as an independent build tool with main support for JVM based languages and Python.
  8. Lets go over some basic concepts of Pants. You start by defining a source tree which is organized by language. For every leaf node in this source tree, you place BUILD files which define the target. BUILD files are similar to writing python but is a pants specific DSL. It essentially invokes python constructors in pants. Targets can either be a library or a binary. Binaries can be put in a docker container and run on any server and libraries can be referenced by any other targets.
  9. One important aspect of pants is its ability to generate PEX files. PEX files are special python executables. You can think of them as a statically compiled golang binary or a fat Java JAR file. They are similar in idea to what a virtual environment is but its essentially all packed in together and made executable. Its a zip file with a python directive and a special __main__.py that allows you to interact with PEX runtime. This follows the same philosophy that we see today in docker containers where you develop immutable containers and create or destroy them as needed. Similarly, PEX files are immutable artifacts. You package your application to be able to run on any server that can run python. You can also run all targets locally without having to maintain complex virtual environments. This also helps in debugging as your projects can use standard versions of 3rdparty dependencies.
  10. Lets go over an example of a BUILD file. As mentioned, this is Pants DSL but it is essentially a function call. python_library will create a library named :shared_lib that can be used by other targets. You define dependencies in a simple array and your source file. Now we want to use this library to create a CLI binary. We specify the dependencies for the CLI and include the shared library as one of the dependency.
  11. This is a very simple introduction to Pants. And there are some open source repositories using pants. Lets go over a simple project that I created for this talk.