PyData Paris 2015 - Track 4.2 Jean Maynier
•
0 likes•1,258 views
Jean Maynier (Kpler): “Python, SQLalchemy and Scrapy for real-time data processing at Kpler” Bio: I lead the engineering and operational activities of Kpler (formerly eCO2market). Founded in Paris in 2009, Kpler is an intelligence company providing transparency solutions in energy markets. We develop proprietary technologies that systematically aggregate data from hundreds of sources, ranging from logistics and commercial, to governmental and shipping databases. By connecting the dots across fragmented information landscapes, we are able to deliver our clients with unique real-time market coverage.
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
More from Pôle Systematic Paris-Region
More from Pôle Systematic Paris-Region (20)
Recently uploaded
Recently uploaded (20)
PyData Paris 2015 - Track 4.2 Jean Maynier
- 1. Python, SQLAlchemy and Scrapy at Kpler Jean Maynier CTO
- 2. Business : commodities markets Domain: trading / maritime transport Customers: physical traders / analysts Needs: follow cargoes flows and prices impact
- 3. Commercial Database “Meta” AIS (10 networks) Ports Authorities Gas Inventories Methodology :4 data layers Thefirst “Cargo-Tracking” system, bridging thegapbetween LNGShipping andTrading.
- 4. Why python ? Simple, fast to prototype Data libraries (sqlAlchemy, Pandas, Scikit-learn) Data oriented community Scrapy: best scraping framework Step 1
- 5. Scrapy Simplify crawling Reusable extractors (xpath, css selectors) PaaS: Scrapinghub Blacklist: Tor, crawlera for IP rotation Step 2
- 6. Process raw data Clean and normalize data Positions Port calls Contracts Prices Python PubSub system Persist data : sqlAlchemy & Postgresql Step 3
- 7. Enrich data Done Vessel dock/undock events Predict next destination Find buyer/seller Basic routing Todo / WIP Estimate price Routing with weather Portfolio optimisation Fleet optimisation Step 4
- 8. Architecture Websites sources API sources Scraping Python/scrapy Cache Elasticseach Data ETL + models python Web REST API scala client mobile client Excel
- 9. Metrics trends Present DBs < 100Gb 50 sources 500 vessels Positions every 3min Future 1-10 Tb 100 sources 10k to 100k vessels Position every 30s Performance problem !
- 10. Batch to data streaming Parallelization Granularity : item vs source Handle failure, no data loss, monitoring Akka streaming, Spark streaming, Celery, Storm Step 5
- 11. Storm at Kpler: POC Websites sources API sources Scraping Python/scrapy ETL + models Position Dock/undock Custom prices Match prices Next destination Trades Cargo volumes
- 12. We are recruiting, join us ! jobs@kpler.com