The needs and requirements for Data Transformation technologies be it Big Data, Machine Learning, Deep Learning or Simple Search and Reporting is still maturing due to the fundamental focus loss on Data Structural Patterns that can enable it. This presentation is oriented towards it.

1. Pandas - Data
Transformational Data
Structure Patterns and
Challenges
Rajesh Manickadas
OrangeScape
07 Sept 2018

2. Objective
The Objective of this Presentation is to elaborate on Numpy/Pandas and more
in the following light
● Differentiate python data structures and numpy/pandas
● What is Data Transformational Design Patterns ?
● Numpy / Pandas Data Structures and Usage
● Contemplate on Such Patterns for Future
PROGRAM = DATA STRUCTURES + ALGORITHMS

3. Python Data Structures - Primer
A Refresher to Python Data Structures
Tuples
Immutable
Containers
Lists
Mutable
Containers
Dict
Key Indexed
Containers

4. Python Data Structures - Functional Optimization Patterns
The Prime Objective is to optimize the data structures for functional programming
Scalars are Python Objects designed
with functional optimization
patterns.
>>> a = 45
>>> b = 45
>>> id(a)
16790784
>>> id(b)
16790784
A
B
45
16790784
List and Lists and List of Lists and
List of List of Lists….Arrays ?
Good for Functional Work and Not Designed for Large Data
Processing
Examples: Transpose, Slicing, Pivoting, Vectorization

5. Data Transformational Design Pattern Needs
● Data is Memory. Large Data is Huge Memory. Memory is Expensive !
● Data in Real Time changes all the time. It's not a csv :). - Speed !
○ Data Warehouses Vs Databases Vs Pandas
● We try to move from the Functional arena to a Data arena - Data
Structures are to be designed for Data Processing Algorithms
○ Data Needs are Dimensions, Measurable, Searchable, Visualize, Views etc.
● The World of Big Table, Bigquery, Hadoop et al is mixed up with Offline
Data, Slow Processing (Design Needs), Append only, Queryless
● It's just not scientific. Its Business !!
○ Realtime Vs Offline/Batch
○ Reporting and Intelligence Vs Analysis and Research
○ Simple Lookups are going to be tricky in future
Exodus from Functional Program Optimization to Data Transformation Optimization

6. ndarray
NumPy Data Structures - ndarray - Starting of the Data
Transformational Patterns - “Forget the logic, focus on the data needs”
Ndarrays - Data Transformation Objectives
Meta
Data
Data
Buffer
Metadata Objectives
Flexibility - Ability to twist the data in a performant and pythonic way ex. Transpose, Shape
Reuse - Reuse of the Data Buffer ex. Views
Abstraction - Vectorization,Broadcasting
Data Buffer - Speed and Memory Optimized
A Chunk of Memory starting at a particular location
Moving the pointer ex. Strides - Row Major Order/Column Major Order

7. NumPy Data Structures - ndarrayNdarrays - Data Transformation Optimizations
PyArrayObect
typedef struct PyArrayObject {
PyObject_HEAD
char *data;
int nd;
npy_intp *dimensions;
npy_intp *strides;
PyObject *base;
PyArray_Descr *descr;
int flags;
PyObject *weakreflist;
} PyArrayObject;
>>> import numpy as np
>>> matx = np.arange(15)
>>> id(matx)
139892166884368
>>> mat3x5 = matx.reshape(3,5)
>>> id(mat3x5)
139892020117712
>>> matx
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14])
>>> mat3x5
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
>>> matx[4] = 100
>>> matx
array([ 0, 1, 2, 3, 100, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14])
>>> mat3x5
array([[ 0, 1, 2, 3, 100],
[ 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14]])
>>> _
Dim:2
strides:(40,8)
shape:(3,5)
reshape
Ndarray 1:
matx
Ndarray 2:
mat3x5
Dim:1
strides:(8,)
shape:(15,)
DATA

8. NumPy Data Structures - More Concepts
The More you know, The More you apply the Data Transformational Patterns for Optimizations (to
reduce memory footprints, improve execution speed etc). It is a Swiss Army Knife
Broadcasting
N-D Iterators
Indexing
Scalar Types
Routines
Shapes and
Views

9. Pandas - Where Python Meets the Tables(Databases)
For what people see is what they manipulate
Series
(1n)
DataFrame
(2n)
Panels
(3n)
Tables
DataFrame
Data
Indexing
Set Algebra
Immutable
Ordered Set
Hash/Dict
Joins
Unions
Filters
Intersections

10. Pandas Data Structures- Differentiating from Databases/SQL
Pandas take on Data
● Select and Filters - Shaping and Slicing
● Joins - Joins, Merge, Concat
● Aggregation and Operations - Vectorization, Broadcasting
● Advanced/Dynamic Aggregation
○ Dimensions and Measures Patterns
■ Pivots
● Close Collaboration between the data structure and algorithms
○ Statistical Functions
○ Scientific Functions
○ Machine Learning etc.
○ SciPy

11. Block Manager - There enters the manager !!!The Manager Data Transformational Pattern, If you want to call it “Under the hood” or “Internals” I am fine with it

12. Pandas - Indexing a DataFrame
Indexing Organization
Year Total Gas Liquid Solid
1997 250255 12561 66649 159191
1998 255310 12990 71750 158106
1999 271548 11549 77852 169087
2000 281389 11974 82834 172812
...
Label Index
DateTime
Index
Data
Array, ordered,
immutable,
hashtable,int64
Array, ordered,
immutable,
hashtable,
timestamp
Ndarray
data
dype
Index (axis)
columns

13. Pandas - Time Series - C02 Emissions in India (1858- 2014)Time Series Example
>>> import numpy as np
>>> import pandas as pd
>>> import matplotlib.pyplot as plt
>>> dateparse = lambda dates:
pd.datetime.strptime(dates, '%Y')
>>> co2emission =
pd.read_table('inco2.csv',delimiter=',',header='infer',
parse_dates=True,
index_col='Year',date_parser=dateparse)
>>> co2emission.plot()
<matplotlib.axes.AxesSubplot object at
0x7fd79d20bcd0>
>>> plt.show()
>>> co2solidemission = co2emission['Solid']
>>> co2solidemission.plot()
<matplotlib.axes.AxesSubplot object at
0x7fd79be3bf50>
>>> plt.show()
>>> co2solidemission.mean()
50129.979310344825

14. Data Transformational Patterns - Where Pandas Fits
Courtesy:Dremio

15. Challenges
“Nowadays, my rule of thumb for pandas
is that you should have 5 to 10 times as
much RAM as the size of your dataset. So
if you have a 10 GB dataset, you should really
have about 64, preferably 128 GB of RAM if you
want to avoid memory management problems.”
- Wes McKinney
BDFL, Pandas
“10 Things I hate about Pandas”
1. Internals too far from "the metal"
2. No support for memory-mapped datasets
3. Poor performance in database and file ingest /
export
4. Warty missing data support
5. Lack of transparency into memory use, RAM
management
6. Weak support for categorical data
7. Complex groupby operations awkward and slow
8. Appending data to a DataFrame tedious and
very costly
9. Limited, non-extensible type metadata
10. Eager evaluation model, no query planning
11. "Slow", limited multicore algorithms for large
datasets
- Wes McKinney
BDFL, Pandas

16. Contemplate on Design Patterns for Realtime Analytics and Big Data
● In-Memory Sessions
● Distributed processing
● Realtime Data Collaboration, Unified Datastores
○ Portable Data Frames - Apache Sparrow
● Strings - Data Management
● Performance - Numba, PyPy
● Snapshots and Visualizations
What can future be...

17. Thank You
Q & A