A look inside pandas design and development

A look inside pandas design and development

• 1.
  A look insidepandas design and development Wes McKinney Lambda Foundry, Inc. @wesmckinn NYC Python Meetup, 1/10/2012 1
• 2.
  a.k.a. “Pragmatic Python for high performance data analysis” 2
• 3.
  a.k.a. “Rise ofthe pandas” 3
• 4.
  Me 4
• 5.
  More like... SPEED!!! 5
• 6.
  Or maybe... (j/k) 6
• 7.
  Me • Mathematician atheart • 3 years in the quant finance industry • Last 2: statistics + freelance + open source • My new company: Lambda Foundry • Building analytics and tools for finance and other domains 7
• 8.
  Me • Blog: http://blog.wesmckinney.com •GitHub: http://github.com/wesm • Twitter: @wesmckinn • Working on “Python for Data Analysis” for O’Reilly Media • Giving PyCon tutorial on pandas (!) 8
• 9.
  pandas? • http://pandas.sf.net • Swiss-armyknife of (in-memory) data manipulation in Python • Like R’s data.frame on steroids • Excellent performance • Easy-to-use, highly consistent API • A foundation for data analysis in Python 9
• 10.
  pandas • In heavyproduction use in the financial industry • Generally much better performance than other open source alternatives (e.g. R) • Hope: basis for the “next generation” data analytical environment in Python 10
• 11.
  Simplifying data wrangling •Data munging / preparation / cleaning / integration is slow, error prone, and time consuming • Everyone already <3’s Python for data wrangling: pandas takes it to the next level 11
• 12.
  Explosive pandas growth •Last 6 months: 240 files changed 49428 insertions(+), 15358 deletions(-) Cython-generated C removed 12
• 13.
  Rigorous unit testing •Need to be able to trust your $1e3/e6/e9s to pandas • > 98% line coverage as measured by coverage.py • v0.3.0 (2/19/2011): 533 test functions • v0.7.0 (1/09/2012): 1272 test functions 13
• 14.
  Some development asides •I get a lot of questions about my dev env • Emacs + IPython FTW • Indispensible development tools • pdb (and IPython-enhanced pdb) • pylint / pyflakes (integrated with Emacs) • nose • coverage.py • grin, for searching code. >> ack/grep IMHO 14
• 15.
  IPython • Matthew Goodman:“If you are not using this tool, you are doing it wrong!” • Tab completion, introspection, interactive debugger, command history • Designed to enhance your productivity in every way. I can’t live without it • IPython HTML notebook is a game changer 15
• 16.
  Profiling and optimization •%time, %timeit in IPython • %prun, to profile a statement with cProfile • %run -p to profile whole programs • line_profiler module, for line-by-line timing • Optimization: find right algorithm first. Cython-ize the bottlenecks (if need be) 16
• 17.
  Other things thatmatter • Follow PEP8 religiously • Naming conventions, other code style • 80 character per line hard limit • Test more than you think you need to, aim for 100% line coverage • Avoid long functions (> 50 lines), refactor aggressively 17
• 18.
  I’m serious about function length http://gist.github.com/1580880 18
• 19.
  Don’t make amess Uncle Bob YouTube: “What killed Smalltalk could kill s/Ruby/Python, too” 19
• 20.
  Other stuff • Goodkeyboard 20
• 21.
  Other stuff • Bigmonitors 21
• 22.
  Other stuff • Ergonomicchair (good hacking posture) 22
• 23.
  pandas DataFrame • Jack-of-trades tabular data structure In [10]: tips[:10] Out[10]: total_bill tip sex smoker day time size 1 16.99 1.01 Female No Sun Dinner 2 2 10.34 1.66 Male No Sun Dinner 3 3 21.01 3.50 Male No Sun Dinner 3 4 23.68 3.31 Male No Sun Dinner 2 5 24.59 3.61 Female No Sun Dinner 4 6 25.29 4.71 Male No Sun Dinner 4 7 8.770 2.00 Male No Sun Dinner 2 8 26.88 3.12 Male No Sun Dinner 4 9 15.04 1.96 Male No Sun Dinner 2 10 14.78 3.23 Male No Sun Dinner 2 23
• 24.
  DataFrame • Heterogeneous columns •Data alignment and axis indexing • No-copy data selection (!) • Agile reshaping • Fast joining, merging, concatenation 24
• 25.
  DataFrame • Axis indexingenable rich data alignment, joins / merges, reshaping, selection, etc. day Fri Sat Sun Thur sex smoker Female No 3.125 2.725 3.329 2.460 Yes 2.683 2.869 3.500 2.990 Male No 2.500 3.257 3.115 2.942 Yes 2.741 2.879 3.521 3.058 25
• 26.
  Let’s have alittle fun To the IPython Notebook, Batman http://ashleyw.co.uk/project/food-nutrient-database 26
• 27.
  Axis indexing, thespecial pandas-flavored sauce • Enables “alignment-free” programming • Prevents major source of data munging frustration and errors • Fast (O(1) or O(log n)) selecting data • Powerful way of describing reshape / join / merge / pivot-table operations 27
• 28.
  Data alignment, joinops • The brains live in the axis index • Indexes know how to do set logic • Join/align ops: produce “indexers” • Mapping between source/output • Indexer passed to fast “take” function 28
• 29.
  Index join example left right joined lidx ridx a -1 0 d a b 1 1 b JOIN b c 2 2 c c d 0 -1 e e 3 -1 left_values.take(lidx, axis) reindexed data 29
• 30.
  Implementing index joins •Completely irregular case: use hash tables • Monotonic / increasing values • Faster specialized left/right/inner/outer join routines, especially for native types (int32/64, datetime64) • Lookup hash table is persisted inside the Index object! 30
• 31.
  Um, hash table? left joined indexer { } a -1 d 0 b 1 b 1 map c 2 c 2 d 0 e 3 e 3 31
• 32.
  Hash tables • Formthe core of many critical pandas algorithms • unique (for set intersection / union) • “factor”ize • groupby • join / merge / align 32
• 33.
  GroupBy, a brief algorithmic exploration • Simple problem: compute group sums for a vector given group identifications labels values b -1 unique group b 3 labels sums a 2 a 2 a 3 b 4 b 2 a -4 a 1 33
• 34.
  GroupBy: Algo #1 unique_labels= np.unique(labels) results = np.empty(len(unique_labels)) for i, label in enumerate(unique_labels): results[i] = values[labels == label].sum() For all these examples, assume N data points and K unique groups 34
• 35.
  GroupBy: Algo #1,don’t do this unique_labels = np.unique(labels) results = np.empty(len(unique_labels)) for i, label in enumerate(unique_labels): results[i] = values[labels == label].sum() Some obvious problems • O(N * K) comparisons. Slow for large K • K passes through values • numpy.unique is pretty slow (more on this later) 35
• 36.
  GroupBy: Algo #2 Makethis dict in O(N) (pseudocode) g_inds = {label : [i where labels[i] == label]} Now for i, label in enumerate(unique_labels): indices = g_inds[label] label_values = values.take(indices) result[i] = label_values.sum() Pros: one pass through values. ~O(N) for N >> K Cons: g_inds can be built in O(N), but too many list/dict API calls, even using Cython 36
• 37.
  GroupBy: Algo #3,much faster • “Factorize” labels • Produce vectorto the unique observedK-1 corresponding of integers from 0, ..., values (use a hash table) result = np.zeros(k) for i, j in enumerate(factorized_labels): result[j] += values[i] Pros: avoid expensive dict-of-lists creation. Avoid numpy.unique and have option to not to sort the unique labels, skipping O(K lg K) work 37
• 38.
  Speed comparisons • Testcase: 100,000 data points, 5,000 groups • Algo 3, don’t sort groups: 5.46 ms • Algo 3, sort groups: 10.6 ms • Algo 2: 155 ms (14.6x slower) • Algo 1: 10.49 seconds (990x slower) • Algos 2/3 implemented in Cython 38
• 39.
  GroupBy • Situation issignificantly more complicated in the multi-key case. • More on this later 39
• 40.
  Algo 3, profiled In[32]: %prun for _ in xrange(100) algo3_nosort() cumtime filename:lineno(function) 0.592 <string>:1(<module>) 0.584 groupby_ex.py:37(algo3_nosort) 0.535 {method 'factorize' of DictFactorizer' objects} 0.047 {pandas._tseries.group_add} 0.002 numeric.py:65(zeros_like) 0.001 {method 'fill' of 'numpy.ndarray' objects} 0.000 {numpy.core.multiarray.empty_like} 0.000 {numpy.core.multiarray.empty} Curious 40
• 41.
  Slaves to algorithms •Turns out that numpy.unique works by sorting, not a hash table. Thus O(N log N) versus O(N) • Takes > 70% of the runtime of Algo #2 • Factorize is the new bottleneck, possible to go faster?! 41
• 42.
  Unique-ing faster Basic algorithmusing a dict, do this in Cython table = {} uniques = [] for value in values: if value not in table: table[value] = None # dummy uniques.append(value) if sort: uniques.sort() Performance may depend on the number of unique groups (due to dict resizing) 42
• 43.
  Unique-ing faster No Sort:at best ~70x faster, worst 6.5x faster Sort: at best ~70x faster, worst 1.7x faster 43
• 44.
  Remember 44
• 45.
  Can we gofaster? • Python dictimplementations one of the best hash table is renowned as anywhere • But: • No abilityresizings arbitrary to preallocate, subject to • We don’t care about reference counting, throw away table once done • Hm, what to do, what to do? 45
• 46.
  Enter klib • http://github.com/attractivechaos/klib •Small, portable C data structures and algorithms • khash: fast, memory-efficient hash table • Hack a Cython interface (pxd file) and we’re in business 46
• 47.
  khash Cython interface cdefextern from "khash.h": ctypedef struct kh_pymap_t: khint_t n_buckets, size, n_occupied, upper_bound uint32_t *flags PyObject **keys Py_ssize_t *vals inline kh_pymap_t* kh_init_pymap() inline void kh_destroy_pymap(kh_pymap_t*) inline khint_t kh_get_pymap(kh_pymap_t*, PyObject*) inline khint_t kh_put_pymap(kh_pymap_t*, PyObject*, int*) inline void kh_clear_pymap(kh_pymap_t*) inline void kh_resize_pymap(kh_pymap_t*, khint_t) inline void kh_del_pymap(kh_pymap_t*, khint_t) bint kh_exist_pymap(kh_pymap_t*, khiter_t) 47
• 48.
  PyDict vs. khashunique Conclusions: dict resizing makes a big impact 48
• 49.
  Use strcmp inC 49
• 50.
  Gloves come off with int64 PyObject* boxing / PyRichCompare obvious culprit 50
• 51.
  Some NumPy-fu • Thinkabout the sorted factorize algorithm • Want to compute sorted unique labels • Also compute integer ids relative to the unique values, without making 2 passes through a hash table! sorter = uniques.argsort() reverse_indexer = np.empty(len(sorter)) reverse_indexer.put(sorter, np.arange(len(sorter))) labels = reverse_indexer.take(labels) 51
• 52.
  Aside, for theR community • R’s factor function is suboptimal • Makes two hash table passes • unique uniquify and sort • match ids relative to unique labels • This is highly fixable • R’s integer unique is about 40% slower than my khash_int64 unique 52
• 53.
  Multi-key GroupBy • Significantlymore complicated because the number of possible key combinations may be very large • Example, group by two sets of labels • 1000 unique values in each • “Key space”: 1,000,000, even though observed key pairs may be small 53
• 54.
  Multi-key GroupBy Simplified Algorithm id1, count1 = factorize(label1) id2, count2 = factorize(label2) group_id = id1 * count2 + id2 nobs = count1 * count2 if nobs > LARGE_NUMBER: group_id, nobs = factorize(group_id) result = group_add(data, group_id, nobs) 54
• 55.
  Multi-GroupBy • Pathological, butrealistic example • 50,000 values, 1e4 unique keys x 2, key space 1e8 • Compress key space: 9.2 ms • Don’t compress: 1.2s (!) • I actually discovered this problem while writing this talk (!!) 55
• 56.
  Speaking of performance •Testing the correctness of code is easy: write unit tests • How to systematically test performance? • Need to catch performance regressions • Being mildly performance obsessed, I got very tired of playing performance whack-a- mole with pandas 56
• 57.
  vbench project • http://github.com/wesm/vbench •Run benchmarks for each version of your codebase • vbench checks out each revision of your codebase, builds it, and runs all the benchmarks you define • Results stored in a SQLite database • Only works with git right now 57
• 58.
  vbench join_dataframe_index_single_key_bigger = Benchmark("df.join(df_key2, on='key2')", setup, name='join_dataframe_index_single_key_bigger') 58
• 59.
  vbench stmt3 = "df.groupby(['key1','key2']).sum()" groupby_multi_cython = Benchmark(stmt3, setup, name="groupby_multi_cython", start_date=datetime(2011, 7, 1)) 59
• 60.
  Fast database joins •Problem: SQL-compatible left, right, inner, outer joins • Row duplication • Join on index and / or join on columns • Sorting vs. not sorting • Algorithmically closely related to groupby etc. 60
• 61.
  Row duplication left right outer join key lvalue key rvalue key lvalue rvalue foo 1 foo 5 foo 1 5 foo 2 foo 6 foo 1 6 bar 3 bar 7 foo 2 5 baz 4 qux 8 foo 2 6 bar 3 7 baz 4 NA qux NA 8 61
• 62.
  Join indexers left right outer join key lvalue key rvalue key lidx ridx foo 1 foo 5 foo 0 0 foo 2 foo 6 foo 0 1 bar 3 bar 7 foo 1 0 baz 4 qux 8 foo 1 1 bar 2 2 baz 3 -1 qux -1 3 62
• 63.
  Join indexers left right outer join key lvalue key rvalue key lidx ridx foo 1 foo 5 foo 0 0 foo 2 foo 6 foo 0 1 bar 3 bar 7 foo 1 0 baz 4 qux 8 foo 1 1 bar 2 2 baz 3 -1 Problem: factorized keys qux -1 3 need to be sorted! 63
• 64.
  An algorithmic observation •If N values are known to be from the range 0 through K - 1, can be sorted in O(N) • Variant of counting sort • For our purposes, only compute the sorting indexer (argsort) 64
• 65.
  Winning join algorithm sort keys don’t sort keys Factorize keys columns O(K log K) or O(N) Compute / compress group indexes O(N) (refactorize) "Sort" by group indexes O(N) (counting sort) Compute left / right join indexers for join method O(N_output) Remap indexers relative to original row ordering O(N_output) O(N_output) (this step is actually Move data efficiently into output DataFrame fairly nontrivial) 65
• 66.
  “You’re like CLR,I’m like CLRS” - “Kill Dash Nine”, by Monzy 66
• 67.
  Join test case •Left:pairs rows, 2 key columns, 8k unique key 80k • Right: 8k rows, 2 key columns, 8k unique key pairs • 6k matching key pairs between the tables, many-to-one join • One column of numerical values in each 67
• 68.
  Join test case •Many-to-many case: stack right DataFrame on top of itself to yield 16k rows, 2 rows for each key pair • Aside: sorting the pesky O(K log K)), not the runtime (that unique keys dominates included in these benchmarks 68
• 69.
  Quick, algebra! Many-to-one Many-to-many • Left join: 80k rows • Left join: 140k rows • Right join: 62k rows • Right join: 124k rows • Inner join: 60k rows • Inner join: 120k rows • Outer join: 82k rows • Outer join: 144k rows 69
• 70.
  Results vs. someR packages * relative timings 70
• 71.
  Results vs SQLite3 Absolute timings * outer is LEFT OUTER in SQLite3 Note: In SQLite3 doing something like 71
• 72.
  DataFrame sort bycolumns • Applied same ideas / tools to “sort by multiple columns op” yesterday 72
• 73.
  The bottom line •Just a flavor: pretty much all of pandas has seen the same level of design effort and performance scrutiny • Make sure whoever implemented your data structures and algorithms care about performance. A lot. • Python has amazingly powerful and productive tools for implementation work 73
• 74.
  Thanks! • Follow meon Twitter: @wesmckinn • Blog: http://blog.wesmckinney.com • Exciting Python things ahead in 2012 74