In this deck from the 2019 Stanford HPC Conference, Srinivasan Parthasarathy from Ohio State University presents: Scalable Machine Learning: The Role of Stratified Data Sharding. "With the increasing popularity of structured data stores, social networks and Web 2.0 and 3.0 applications, complex data formats, such as trees and graphs, are becoming ubiquitous. Managing and learning from such large and complex data stores, on modern computational eco-systems, to realize actionable information efficiently, is daunting. In this talk I will begin with discussing some of these challenges. Subsequently I will discuss a critical element at the heart of this challenge relates to the sharding, placement, storage and access of such tera- and peta- scale data. In this work we develop a novel distributed framework to ease the burden on the programmer and propose an agile and intelligent placement service layer as a flexible yet unified means to address this challenge. Central to our framework is the notion of stratification which seeks to initially group structurally (or semantically) similar entities into strata. Subsequently strata are partitioned within this eco-system according to the needs of the application to maximize locality, balance load, minimize data skew or even take into account energy consumption. Results on several real-world applications validate the efficacy and efficiency of our approach. (Notes: Joint work with Y. Wang (Airbnb) and A. Chakrabarti (MSR))." Srinivasan Parthasarathy, Professor of Computer Science & Engineering, The Ohio State University Srinivasan Parthasarathy is a Professor of Computer Science and Engineering and the director of the data mining research laboratory at Ohio State. His research interests span databases, data mining and high performance computing. He is among a handful of researchers nationwide to have won both the Department of Energy and National Science Foundation Career awards. He and his students have won multiple best paper awards or "best of" nominations from leading forums in the field including: SIAM Data Mining, ACM SIGKDD, VLDB, ISMB, WWW, ICDM, and ACM Bioinformatics. He chairs the SIAM data mining conference steering committee and serves on the action board of ACM TKDD and ACM DMKD --leading journals in the field. Since 2012 he also helped lead the creation of OSU's first-of-a-kind nationwide (USA) undergraduate major in data analytics and serves as one of its founding directors. Watch the video: https://youtu.be/hOJI8e0p-UI Learn more: http://web.cse.ohio-state.edu/~parthasarathy.2/ and http://hpcadvisorycouncil.com/events/2019/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. Scalable Learning & Analytics:
Role of Stratified Data Sharding
Srinivasan Parthasarathy
Data Mining Research Lab
Ohio State University

2. The Data Deluge: Data Data Everywhere
22
180 zettabytes will be created in 2025 [IDC report]

3. 600$
to buy a disk drive that can store all of the world’s
music
3
[McKinsey Global Institute Special Report, June ’11]
Data Storage is Cheap

4. Data does not exist in isolation.
4

5. Data almost always exists in
connection with other data – integral
part of the value proposition.
5
“There’s gold in them there mountains of data”
- Gill Press, Forbes Contributor

6. 6
Social networks Protein Interactions Internet
VLSI networks Scientific Simulations
Neighborhood
graphs

7. 7
Big Data Challenge: All this data is only useful if we can extract interesting and actionable
information from large complex data stores efficiently
Projected to be a $200B industry in 2020. [IDC report]

8. MapReduce, MPI, Spark, etc.
Distributed Data Processing is Central
to Addressing the Big Data Challenge
8
Source: blog.mayflower.de

9. Key Challenge: Data Placement (Sharding)
• Locality of reference
– Placing related items in proximity improves efficiency
• Mitigating Impact of Data Skew
– Critical for big data workloads!
• Interactive Response Times
– Operate on a sample with statistical guarantees
• Heterogeneity and Energy Aware
– Heterogeneous compute and storage resources are ubiquitous

10. Stratified Sampling in a Slide
• Roots in Stratified
Sampling (Cochran’48)
• Group related data into
“homogeneous strata”
• Sample each strata
• Proportional
Allocation (shown)
• Optimal Allocation

11. But, here we want to partition/shard
• For Locality
• Elements within a strata are placed together
• For Mitigating Skew
• Each partition is a proportionally
allocated stratified sample
• For Interactivity
• Optimally allocate one partition
• Proportionally allocate the rest
• Accounting for Energy/Heterogeneity
• More on this later -- time permitting
Apps have varying requirements: ONE SIZE DOES NOT FIT ALL!

12. Our Vision: Stratified Data Placement
Key Value Stores
e.g. Memcached
Redis
Key Value Stores
e.g. Memcached
Redis
MPI & Partitioned
Global Addresses
Space Systems
(PGAS)
e.g. Global Arrays
MPI & Partitioned
Global Addresses
Space Systems
(PGAS)
e.g. Global Arrays
HADOOP/SHARK/
Azure
(HDFS/RDD/Blob)
HADOOP/SHARK/
Azure
(HDFS/RDD/Blob)
STRATIFIED DATA SHARDING & PLACEMENTSTRATIFIED DATA SHARDING & PLACEMENT

13. UEFA Desiderata (Performance & Productivity)
• Usable
• Easy to use for domain scientist or developer
• Efficient and Effective
• In terms of end-to-end performance
• Flexible
• To application needs (Locality vs Skew Mitigation)
• Adaptive
• To changes in data, system, or interaction

14. Key Challenge: Creating Strata
(of Complex Data)
• What about Clustering?
• Non-trivial for data with complex structure
• Potentially expensive
• Variable sized entities
• 4-step approach [ICDE’13; ICDE’15]
1. Pivotization
2. Sketching
3. Stratification
4. Partition according to application needs

15. Step 1: Pivotization
Problem: Need to simplify complex
representation.
Key Idea: Think Globally Act
Locally
– Construct set or vector of local
features that collectively
captures global properties
Solution: Specific to Data & Domain
TEXT  SHINGLES (BRODER’98]Spatial/Image Data Kernels/KLSH
(Leslie’03, Kulis’10, Chakrabarti’16]
Graph Pivots (e.g. Adjacency lists [Buehrer’15],
Wedge Decompositions [Seshadri’11], Graphlets [Pruzlj’09])
PIVOTTRANSFORMATIONS
A
B C
LE
A
B C
LE F
.
.
.
.
Δ
1
2Δ
5
DATA (Δ)
A
B C
A
F C
A
E C
A
F L
B
E F
A
E L
A
B L
A
B C
A
E C
A
E L
A
B L
.
.
.
.
(PS-1)
(PS-25)
PIVOT SETS (PS)
Trees (XML, Linguistic data) [Tatikonda’10]

16. Step 2. Sketching
• Problem: Pivot sets may be
variable length, similarity
computation is expensive:
O(n^2)
• Key Idea: Use Sketching
• Solution: Locality Sensitive
Hashing [Broder’98, Indyk’99, Charikar’01]
– Resulting representation is fixed-
length (k)
– Tradeoff: Representation Fidelity
vs. Sketch size
– Can handle kernel functions
[Kulis’09] and statistical priors
[Satuluri’12, Chakrabarti’15, ‘16]
A
B C
A
F C
A
E C
A
F L
B
E F
A
E L
A
B L
A
B C
A
E C
A
E L
A
B L
.
.
.
.
(PS-1)
(PS-25)
PIVOT SETS (PS)
MINWISEHASHINGonPIVOTSETS
{1050, 2020,
3130,1800}
(SK-1)
{1050, 2020,
7225, 2020}
(SK-25)
.
.
.
.
.
.
SKETCHES(SK)

17. 20
Key Fact
For two sets A, B, and a min-hash function mhi():
Unbiased estimator for Sim using k hashes:

18. Step 3: Stratification
Problem: Group related entities
into strata
Key Idea: Inspired by W.
Cochran’s work on stratified
sampling [1940s]
Solutions:
•Sort pivot sets directly (skip sketch
step) – Pivot Sort
•Directly use output of LSH/Minwise
Hash – SketchSort
•Cluster sketches with fast variant of k-
modes – SketchCluster
SKETCHSORTorSKETCHCLUSTER
S-1
:
:
S-4
( 1, SK-1)Δ
( 5, SK-5)Δ
( 12,SK-12)Δ
( 25,SK-25)Δ
:
:
:
S-5
:
:
:
S-128
:
:
:
MINWISEHASHINGonPIVOTSETS
{1050, 2020,
3130,1800}
(SK-1)
{1050, 2020,
7225, 2020}
(SK-25)
.
.
.
.
.
.
SKETCHES(SK)

19. Step 4: Sharding and Placement
• Problem: How to partition stratified data?
• Key Ideas: Guided by application hints and system state.
• Solutions:
1. Proportional Allocation: Split each stratum uniformly
proportionally across all partitions  mitigates skew
1. Optimal Allocation for first strata, proportional for
rest [C77]
1. All-in-One : Place each stratum in its entirety within a
partition
IMPORTANT NOTE: We use sketches to create strata – but
partitioning happens on original data.

20. SKETCHSORTORSKETCHCLUSTER
S-1
:
:
S-4
( 1, SK-1)Δ
( 5, SK-5)Δ
( 12,SK-12)Δ
( 25,SK-25)Δ
:
:
:
S-5
:
:
:
S-128
:
:
:
PARTITIONING&REPLICATION
P-1
:
P-2
S-4
S-7
S-8
S-12
:
S-128
P-3
:
:
:
P-8
S-3
S-4
S-9
S-12
: S-
127
PIVOTTRANSFORMATIONS
A
B C
LE
A
B C
LE F
.
.
.
.
1Δ
25Δ
DATA (Δ)
A
B C
A
F C
A
E C
A
F L
B
E F
A
E L
A
B L
A
B C
A
E C
A
E L
A
B L
.
.
.
.
(PS-1)
(PS-25)
PIVOT SETS (PS)
MINWISEHASHINGonPIVOTSETS
{1050, 2020,
3130,1800}
(SK-1)
{1050, 2020,
7225, 2020}
(SK-25)
.
.
.
.
.
.
SKETCHES(SK) Strata (S)

21. Empirical Evaluation
• We report wall clock times
• All times include cost of placement
• Evaluations on several key analytic tasks
• Top-K algorithms [Fagin], Outlier Detection [Ghoting’08, Otey’06],
Frequent Tree[Zaki’05, Tatikonda’09] and Graph Mining [Buehrer’06,
Yan’02, Nijlsson’04], XML Indexing [Tatikonda’07], Community
detection in Social/Biological data [Ucar’06, Satuluri’11], Web Graph
Compression [Chellapilla’08-09; Vigna’11, LZ’77], Itemset Mining
[Buehrer-Fuhry’15]
• All applications are run straight out of the box – the only thing
the user specifies relates to locality, skew, and interaction.

22. Frequent Tree Mining
[Tatikonda’09]
• Used Widely
– Transactions, graphs, trees
• Approach
1. Distribute Data
• Proportional
Allocation
1. Run Phase 1
2. Exchange Meta Data
3. Run Phase 2
4. Final Reduction
• Sharding mainly impacts steps
1-3. Steps 3 and 5 are
sequential.
Proposed approaches shows 100X gains

23. FTM Phase 1: Drilling Down
•
• Data Dependent Workload Skew is mitigated
• Payload-aware sharding helps!

24. WebGraph Compression [Vigna et
al 2011]
Critical application for search
companies
Key Requirement: Locality
Approach:
• Distribute data via placement
• Run compression algorithm
in parallel
• Parameters (similar to FTM)
– Use adjacency/triangle pivots
– Use All-in-one partitioning

25. Comparison
with Metis
• Metis
– Balanced partitions
– Optimizes KL criterion
– Static algorithm
• Our approaches
– Much Faster
– Qualitatively comparable
or better

26. Step 4: Energy- and Heterogeneity- Aware
Partitioning (ICPP’17)
• Modern Datacenters are
increasingly heterogeneous
– Computation
– Storage
– Green Energy Harvesting
• Sharding and placement while
accounting for heterogeneity
is challenging
– Pareto Optimal Model

27. Overview of Pareto Framework

28. 32
Evaluation – Pareto Frontiers
Swiss (Tree Mining) RCV: Text Mining UK: Web Analytics

29. Take Home Message
• In todays analytics world data has complex structure
• Stratitifed Data Placement has important role to play
– Over 2 orders of magnitude improvement over state-of-art for a multitude of
analytic tasks.
– Preliminary results on heterogeneous- energy-aware systems show
significant promise!
Key Value Stores
e.g. Memcached
Redis
Key Value Stores
e.g. Memcached
Redis
MPI & Partitioned
Global Addresses
Space Systems
(PGAS)
e.g. Global Arrays
MPI & Partitioned
Global Addresses
Space Systems
(PGAS)
e.g. Global Arrays
HADOOP/SHARK/
Azure
(HDFS/RDD/Blob)
HADOOP/SHARK/
Azure
(HDFS/RDD/Blob)
STRATIFIED DATA SHARDING &PLACEMENTSTRATIFIED DATA SHARDING &PLACEMENT

30. Thanks
• Organizers
• Audience
• Students and Collaborators
– Ye Wang (AirBnB), A. Chakrabarty (MSR),
– P. Sadayappan, C. Stewart (OSU)
• Funding agencies
– NSF, DOE, NIH, Google, IBM