Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Future of pandas
Similar to Future of pandas (20)
Recently uploaded
Recently uploaded (20)
Future of pandas
- 1. Future of Pandas Jeff Reback PyData NYC November 2017
- 2. • The information presented here is offered for informational purposes only and should not be used for any other purpose (including, without limitation, the making of investment decisions). Examples provided herein are for illustrative purposes only and are not necessarily based on actual data. Nothing herein constitutes: an offer to sell or the solicitation of any offer to buy any security or other interest; tax advice; or investment advice. This presentation shall remain the property of Two Sigma Investments, LP (“Two Sigma”) and Two Sigma reserves the right to require the return of this presentation at any time. • Some of the images, logos or other material used herein may be protected by copyright and/or trademark. If so, such copyrights and/or trademarks are most likely owned by the entity that created the material and are used purely for identification and comment as fair use under international copyright and/or trademark laws. Use of such image, copyright or trademark does not imply any association with such organization (or endorsement of such organization) by Two Sigma, nor vice versa. • Copyright © 2017 TWO SIGMA INVESTMENTS, LP. All rights reserved IMPORTANT LEGAL INFORMATION
- 3. @jreback ● Former quant ● Senior Engineer at Two Sigma, working on holistic approaches to modeling ● Core committer to pandas for last 5 years ● Managed pandas since 2013
- 4. Kudos!
- 6. Overview ● State of the Pandas ○ The Good ○ The Bad ○ The Ugly
- 7. Overview ● State of the Pandas ○ The Good ○ The Bad ○ The Ugly ● The Present
- 8. Overview ● State of the Pandas ○ The Good ○ The Bad ○ The Ugly ● The Present ● The Future
- 9. The Good
- 13. pandas’s role in the Python Data Ecosystem pandas Numerical Computing IO / Data Access Data Visualization Statistics + Machine Learning Libraries Users
- 14. The Bad
- 15. http://wesmckinney.com/blog/apache-arrow-pandas-internals/ @wesm "10 Things I Hate About pandas"
- 16. DataTypes - what are we Missing?
- 17. DataTypes - Missing values can cause dtype changes
- 18. ● Complex groupby operations awkward and slow ● Copy Semantics API
- 19. API - Opaque UDFs
- 20. API - Opaque UDFs
- 21. API - Opaque UDFs
- 22. API - Aggregation Syntax
- 23. API - Copy Semantics
- 24. ● In-memory format that is custom ● Eager evaluation model, no query planning ● "Slow", limited multicore algorithms for large datasets Performance
- 25. Data Tooling Spectrum Small Data “Medium” Data “Big” Data < 5GB 5-100GB > 100 GB pandas starts to fail as an effective tool somewhere around the 10 GB mark
- 26. Block Storage
- 27. Block Storage Float 1.0 2.0 1.0 2.0 1.0 2.0 1.0 2.0 Int 1 2 3 4 Index RangeIndex (0, 4, 1) Columns Index [‘A’, ‘B’, ‘C’] Block Manager AxesBlocks
- 28. The Ugly
- 29. Big Data Unfriendly Each system has its own internal memory format 70-80% computation wasted on serialization and deserialization Similar functionality implemented in multiple projects
- 30. The Present
- 31. CategoricalDtype efficient memory & first class Categoricals efficient IO out-of-core and multi-core In current pandas
- 32. The Future
- 33. pandas2 architecture pandas2 Arrow-optimized data connectors Arrow in-memory format Parallel Dataflow Execution Engine Apache Arrow Python user API, User-defined functions Logical Data Frame Expression Graphs Ibis DataFrame semantics & compatibility
- 41. Ibis in a nutshell Ibis Python code Compiler Back End compiled code Abstract Syntax Tree
- 42. pandas2 architecture pandas2 Arrow-optimized data connectors Arrow in-memory format Parallel Dataflow Execution Engine Apache Arrow Python user API, User-defined functions Logical Data Frame Expression Graphs Ibis DataFrame semantics & compatibility
- 43. Apache Arrow project The Arrow supports zero-copy reads and is optimized for data locality. Fast Arrow acts as a new high-performance interface between various systems. Flexible Apache Arrow is backed by many key open source projects. Standard
- 44. Big Data friendly All systems utilize the same memory format No overhead for cross-system communication Projects can share functionality (eg, Parquet-to-Arrow reader)
- 45. DataFrame
- 46. Computation ● Kernel functions: atomic units of computation ● Operator nodes: input/output types, operator parallelism properties
- 47. Physical Operator Graphs Log Add b a (a + b).log().sum() Sum
- 48. Parallel Evaluation of Operator Graphs a b tmp tmp 2 out Add SumLog
- 49. pandas2 architecture pandas2 Arrow-optimized data connectors Arrow in-memory format Parallel Dataflow Execution Engine Apache Arrow Python user API, User-defined functions Logical Data Frame Expression Graphs Ibis DataFrame semantics & compatibility
- 51. Status & Links pandas2 https://github.com/pandas-dev/pandas2 Ibis https://github.com/ibis-project/ibis 0.12.0 released in October. Arrow https://github.com/apache/arrow 0.7.1 released in October.
- 52. What can the community do? http://pandas.pydata.org/
Editor's Notes
- mention when *I* started working on pandas
- mention when *I* started working on pandas
- pandas is the go-to pre & post-processor. The UX and some of the UI.
- Warty missing data support Limited, non-extensible type metadata Weak support for categorical data
- Issues: output shape of the result its not obvious how to predict performance
- output shape, 2) performance, maybe show unwrapping groupy.. note in ibis that we automatically do this. not obvious how to predict performance.
- output shape, 2) performance, maybe show unwrapping groupy.. note in ibis that we automatically do this. not obvious how to predict performance.
- no type checking on args not syntactful can’t easily rename
- what is a view? leaky implementation. copy-on-write to the rescue.
- you need 2x - 5x of the data size to be comfortable. other talks: Li Jin & streamz
- why this storage methods: input was traditionally 2-d numpy arrays, zero-copy
- predicting performance is hard when you add/remove blocks we are storing in fortran ordering, meaning columnar formats.
- We have implemented a number of short-term fixes. These won’t solve some of the major issues, but can be helpful in removing rough edges and incrementally improving performance.
- We want to de-couple our *intent* from the *execution* pandas2 drop-in replacements. 95% compat (missing value support changes, deferred execution, copy-on-write, Table API).
- talk about SQLAlchemy here.
- mention mypy here
- described our intent, dtypes; deferred execution, this allow query planning
- can chain predicates
- We want to de-couple our *intent* from the *execution* pandas2 drop-in replacements. 95% compat (missing value support changes, deferred execution, copy-on-write, Table API).
- this is what Arrow is today In the future, this will also include computation. but i think you want to lead w/ the full stack here i.e. operator call graph -> operators -> on high performance in memory format. (otherwise the following slides get confusing) WM: done, and added definition of pandas2 compute / simd / vectors gpu dataframe (goAI /
- top image - looks like a dataframe (but is actually a Table, e.g. no indices)
- distinguish betwen logical & physical plans, use (a+b).log().sum() Inspired by prior art in peer ecosystems Analytic SQL databases Modern deep learning frameworks Kernel Function -> UFuncs
- We want to de-couple our *intent* from the *execution* pandas2 drop-in replacements. 95% compat (missing value support changes, deferred execution, copy-on-write, Table API).
- Mention contributors? e.g. Phil c Ibis BigQuery, File backends (CSV, HDF5, parquet) - Done Spark backend - Wanted Arrow Compute kernels Hearsurgeon / Mechanic joke
- Ibis BigQuery, File backends (CSV, HDF5, parquet) - Done Spark backend - Wanted Arrow Compute kernels