Download as PDF, PPTX
![Extract Transform Load
• Not new. Popular concept in the 1970s [1] [2]
• Everywhere. Commerce, websites, marketing, finance, …
• Update secondary data-stores from a master data-store.
• Insert initial data somewhere
• Bi-directional API calls
• Every day, compute time reports, edit invoices if threshold then send e-mail.
• …
[1] https://en.wikipedia.org/wiki/Extract,_transform,_load
[2] https://www.sas.com/en_us/insights/data-management/what-is-etl.html](https://image.slidesharecdn.com/pyparis-bonoboetl-170612154441/85/Simple-ETL-in-python-3-5-with-Bonobo-PyParis-2017-8-320.jpg)
![Classes!
from bonobo.config import Configurable, Option, Service
class QueryDatabase(Configurable):
# Simple string option
table_name = Option(str, default='customers')
# External dependency
database = Service('database.default')
def call(self, database, **row):
customer = database.query(self.table_name, customer_id=row['clientId'])
return {
**row,
'is_customer': bool(customer),
}](https://image.slidesharecdn.com/pyparis-bonoboetl-170612154441/85/Simple-ETL-in-python-3-5-with-Bonobo-PyParis-2017-54-320.jpg)
![Docker Extension
$ pip install bonobo[docker]
$ bonobo runc myjob.py
PREVIEW](https://image.slidesharecdn.com/pyparis-bonoboetl-170612154441/85/Simple-ETL-in-python-3-5-with-Bonobo-PyParis-2017-71-320.jpg)
Uploaded byRomain Dorgueil
Simple ETL in python 3.5+ with Bonobo - PyParis 2017
The document discusses Bonobo, a simple ETL framework for Python 3.5 and above, designed for data integration using code as configuration. It emphasizes Bonobo's ability to create, manage, and execute ETL jobs easily, incorporating libraries like joblib and pandas, while noting its distinct features compared to existing tools. The text also outlines the history, current status, and future plans for the Bonobo framework, including open-source goals and enhancements for developer experience.
More Related Content
Similar to Simple ETL in python 3.5+ with Bonobo - PyParis 2017
Simple ETL in python 3.5+ with Bonobo - PyParis 2017
- 1. bonobo Simple ETL inPython 3.5+
- 2. Romain Dorgueil @rdorgueil CTO/Hacker inResidence Technical Co-founder (Solo) Founder Eng. Manager Developer L’Atelier BNP Paribas WeAreTheShops RDC Dist. Agency Sensio/SensioLabs AffiliationWizard Felt too young in a Linux Cauldron Dismantler of Atari computers Basic literacy using a Minitel Guitars & accordions Off by one baby Inception
- 3.
- 4. STARTUP ACCELERATION PROGRAMS NOHYPE, JUST BUSINESS launchpad.atelier.net
- 5. bonobo Simple ETL inPython 3.5+
- 6. • History &Market • Extract, transform, load • Basics – Bonobo One-O-One • Concepts & Candies • The Future – State & Plans
- 7. Once upon atime…
- 8. Extract Transform Load •Not new. Popular concept in the 1970s [1] [2] • Everywhere. Commerce, websites, marketing, finance, … • Update secondary data-stores from a master data-store. • Insert initial data somewhere • Bi-directional API calls • Every day, compute time reports, edit invoices if threshold then send e-mail. • … [1] https://en.wikipedia.org/wiki/Extract,_transform,_load [2] https://www.sas.com/en_us/insights/data-management/what-is-etl.html
- 9. Small Automation Tools •Mostly aimed at simple recurring tasks. • Cloud / SaaS only.
- 10. Data Integration Tools •Pentaho Data Integration (IDE/Java) • Talend Open Studio (IDE/Java) • CloverETL (IDE/Java)
- 11.
- 14. Data Integration Tools •Java + IDE based, for most of them • Data transformations are blocks • IO flow managed by connections • Execution
- 15. In the Pythonworld … • Bubbles (https://github.com/stiivi/bubbles) • PETL (https://github.com/alimanfoo/petl) • and now… Bonobo (https://www.bonobo-project.org/) You can also use amazing libraries including Joblib, Dask, Pandas, Toolz, but ETL is not their main focus.
- 16. Big Data Tools •Can do anything. And probably more. Fast. • Either needs an infrastructure, or cloud based.
- 17.
- 18. Partner 1 DataIntegration
- 19. WE GOT DEALS!!!
- 20. Partner 1 Partner2 Partner 3 Partner 4 Partner 5 Partner 6 Partner 7 Partner 8 Partner 9 …
- 21. Tiny bug there… Canyou fix it ?
- 23. I want… • Adata integration / ETL tool using code as configuration. • Preferably Python code. • Something that can be tested (I mean, by a machine). • Something that can use inheritance. • Fast install on laptop, thought to run on servers too.
- 24.
- 25. It is … •A framework to write ETL jobs in Python 3 (3.5+) • Using the same concepts as the old ETLs • But for coders! • You can use OOP!
- 26. It is NOT… • Pandas / R Dataframes • Dask • Luigi / Airflow • Hadoop / Big Data • A monkey
- 27. Bonobo - One- O - One
- 28. Let’s create aproject
- 29. Let’s create aproject ~ $ pip install bonobo cookiecutter
- 30. Let’s create aproject ~ $ pip install bonobo cookiecutter ~ $ bonobo init pyparis
- 31. Let’s create aproject ~ $ pip install bonobo cookiecutter ~ $ bonobo init pyparis ~ $ cd pyparis
- 32. Let’s create aproject ~ $ pip install bonobo cookiecutter ~ $ bonobo init pyparis ~ $ cd pyparis ~/pyparis $ ls -lsah total 32 0 drwxr-xr-x 7 rd staff 238B 30 mai 18:02 . 0 drwxr-xr-x 4 rd staff 136B 30 mai 18:03 .. 0 -rw-r--r-- 1 rd staff 0B 30 mai 18:02 .env 8 -rw-r--r-- 1 rd staff 6B 30 mai 18:02 .gitignore 8 -rw-r--r-- 1 rd staff 333B 30 mai 18:02 main.py 8 -rw-r--r-- 1 rd staff 140B 30 mai 18:02 _services.py 8 -rw-r--r-- 1 rd staff 7B 30 mai 18:02 requirements.txt
- 33. It works! $ bonoborun . 1 3 5 ... 37 39 41 - range in=1 out=42 - Filter in=42 out=21 - print in=21
- 34.
- 35.
- 36. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i
- 37. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n
- 38. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n)
- 39. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, )
- 40. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph)
- 41. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42)
- 42. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42) bonobo.Filter(lambda n: n % 2)
- 43. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42) bonobo.Filter(lambda n: n % 2) print
- 44. Let’s rewrite main.py importbonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42) bonobo.Filter(lambda n: n % 2) print graph = bonobo.Graph( range(42), bonobo.Filter(filter=lambda x: x % 2), print, ) if __name__ == '__main__': bonobo.run(graph)
- 45.
- 46.
- 47. bonobo.run(graph) or in ashell… $ bonobo run main.py
- 48.
- 49.
- 50. Transformations ? a.k.a nodes inthe graph
- 51. Functions! def get_more_infos(api, **row): more= api.query(row.get('id')) return { **row, **(more or {}), }
- 52. Generators! def join_orders(order_api, **row): fororder in order_api.get(row.get('customer_id')): yield { **row, **order, }
- 53. Iterators! extract = ( 'foo', 'bar', 'baz', ) extract= range(0, 1001, 7)
- 54. Classes! from bonobo.config importConfigurable, Option, Service class QueryDatabase(Configurable): # Simple string option table_name = Option(str, default='customers') # External dependency database = Service('database.default') def call(self, database, **row): customer = database.query(self.table_name, customer_id=row['clientId']) return { **row, 'is_customer': bool(customer), }
- 55.
- 56. get_services() COINBASE_API_KEY, COINBASE_API_SECRET =environ.get('...'), environ.get('...') KRAKEN_API_KEY, KRAKEN_API_SECRET = environ.get('...'), environ.get('...') POLONIEX_API_KEY, POLONIEX_API_SECRET = environ.get('...'), environ.get('...') def get_services(): return { 'fs': bonobo.open_fs( path.join(path.dirname(__file__), './data') ), 'coinbase.client': coinbase.wallet.client.Client( COINBASE_API_KEY, COINBASE_API_SECRET ), 'kraken.client': krakenex.API( KRAKEN_API_KEY, KRAKEN_API_SECRET ), 'poloniex.client': poloniex.Poloniex( apikey=POLONIEX_API_KEY, secret=POLONIEX_API_SECRET ), 'google.client': get_google_api_client(), }
- 57. run() will handleinjection import bonobo from bonobo.commands.run import get_default_services graph = bonobo.Graph() if __name__ == '__main__': bonobo.run( graph, services=get_default_services(__file__) )
- 58. … class QueryDatabase(Configurable): # Externaldependency database = Service('database.default') def call(self, database, **row): return { … }
- 59.
- 60. Files bonobo.FileReader( path: str, *, eol: str, encoding:str, ioformat, mode: str ) bonobo.FileWriter( path: str, *, eol: str, encoding: str, ioformat, mode: str )
- 61. CSV bonobo.CsvReader( path: str, *, eol: str, encoding:str, ioformat, mode: str, delimiter: str, quotechar: str, headers: tuple, skip: int ) bonobo.CsvWriter( path: str, *, eol: str, encoding: str, ioformat, mode: str, delimiter: str, quotechar: str, headers: tuple )
- 62. JSON bonobo.JsonReader( path: str, *, eol: str, encoding:str, ioformat, mode: str ) bonobo.JsonWriter( path: str, *, eol: str, encoding: str, ioformat, mode: str )
- 63. Pickle bonobo.PickleReader( path: str, *, eol: str, encoding:str, ioformat, item_names: tuple, mode: str ) bonobo.PickleWriter( path: str, *, eol: str, encoding: str, ioformat, item_names: tuple, mode: str )
- 64.
- 65.
- 66. Filters @bonobo.Filter def NoneShallPass(username): return username== 'romain' bonobo.graph( # ... NoneShallPass(), # ... ) bonobo.Filter(lambda username: username == 'romain')
- 67.
- 68.
- 69.
- 70.
- 71. Docker Extension $ pipinstall bonobo[docker] $ bonobo runc myjob.py PREVIEW
- 72.
- 73. RIP rdc.etl • 2013-2015,python 2.7. • Brings most of the execution algorithms. • Different context. • Learn from mistakes.
- 74. Bonobo is young •First commit : December 2016 • 21 releases, ~400 commits, 4 contributors • Current stable 0.4.1, released yesterday
- 75. __future__ • Stay 100%Open-Source • Light & Focused : Graphs and Executions • The rest goes to Extensions.
- 76. Pre 1.0 Roadmap •Towards stable API • More tests, more documentation • Developer experience first Target: 1.0 before 2018
- 77. Post 1.0 Roadmap •Scheduling • Monitoring • More execution strategies • Optimizations
- 78. Hacker News iscrazy
- 79. Data Processing forHumans
- 80.
- 81.
- 82.