Chicago Hadoop in Finance - Ted Dunning

1©MapR Technologies - Confidential
Scalability in Hadoop and
Similar Systems

Big is the next big thing
 Big data and Hadoop are exploding
 Companies are being funded
 Books are being written
 Applications sprouting up everywhere
2

Slow Motion Explosion
3

Hadoop Explosion
4

Why Now?
 But Moore’s law has applied for a long time
 Why is Hadoop exploding now?
 Why not 10 years ago?
 Why not 20?
58/13/2013

Size Matters, but …
 If it were just availability of data then existing big companies would
adopt big data technology first
6

Size Matters, but …
 If it were just availability of data then existing big companies would
adopt big data technology first
They didn’t
7

Or Maybe Cost
 If it were just a net positive value then finance companies should
adopt first because they have higher opportunity value / byte
8

Or Maybe Cost
 If it were just a net positive value then finance companies should
adopt first because they have higher opportunity value / byte
They didn’t
9

Backwards adoption
 Under almost any threshold argument startups would not adopt
big data technology first
10

Backwards adoption
 Under almost any threshold argument startups would not adopt
big data technology first
They did
11

Everywhere at Once?
 Something very strange is happening
– Big data is being applied at many different scales
– At many value scales
– By large companies and small
12

Everywhere at Once?
 Something very strange is happening
– Big data is being applied at many different scales
– At many value scales
– By large companies and small
Why?
13

More data is being produced more quickly
Data sizes are bigger than even a very large computer can hold
Cost to create and store continues to decrease
The Conventional Answer

Analytics Scaling Laws
 Analytics scaling is all about the 80-20 rule
– Big gains for little initial effort
– Rapidly diminishing returns
 The key to net value is how costs scale
– Old school – exponential scaling
– Big data – linear scaling, low constant
 Cost/performance has changed radically
– IF you can use many commodity boxes

We knew that
We should have
known that
We didn’t know that!
You’re kidding, people do that?

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value
Anybody with eyes
Intern with a spreadsheet
In-house analytics
Industry-wide data consortium
NSA, non-proliferation

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value
2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value
Net value optimum has a
sharp peak well before
maximum effort

But scaling laws are changing
both slope and shape

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value
More than just a little

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value
They are changing a LOT!

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value
Initially, linear cost scaling
actually makes things worse
A tipping point is reached and
things change radically …

Pre-requisites for Tipping
 To reach the tipping point,
 Algorithms must scale out horizontally
– On commodity hardware
– That can and will fail
 Data practice must change
– Denormalized is the new black
– Flexible data dictionaries are the rule
– Structured data becomes rare

Yeah… but wait

The Standard Sort of Model
 People talk about the law of large numbers as if it were …
 Well, as if it were a law
 It’s not …
 It is a context and assumption dependent theorem

What if …
 These assumptions are:
 Changes have a
– stationary,
– independent,
– finite variance distribution
 What happens if these assumptions are wrong?
 And which of them is really wrong?

For Example
Time
Stuﬀ

Time
Stuﬀ
End point
has nice
tractable
distribution

What if the Assumptions are Wrong?
 Take the finite variance as a simple example
 This leads to Levy stable distributions
 Like the Cauchy distribution

Is it Really Different?

Time
Stuﬀ

What About Real Life?

But is it Really Infinite Variance?
 Or are there other kinds of phenomena that show this?
 What about the independence assumption?
 What if the supposedly independent components of the system
communicate?
 Like we do. Everyday. All the time.

Why the Difference?
Law of large
numbers
Infinite
variance
Interacting
agents
Apologies and credit to
Simon DaDeo, SFI
The space of
all things that
change
The space of
interacting
things

What Happens with Interactions
 Social phenomena defeat the law of large numbers
 Distributions are well modeled by “rich get richer” processes
– Pittman-Yar process, Indian Buffet
 Limiting dstributions are heavy tailed, power law
 We see these distributions everywhere
– price of cotton in the 19th century
– word frequencies
– popularity of Github projects
– equity pricing and volumes
– sizes of cities
– popularity of web-sites

What are the
Implications?

2,0000 500 1000 1500
1
0
0.25
0.5
0.75
Scale
Value

In a Nutshell
 Scalability is much more important than we thought
 Mashups are more important than we thought
 Network effects are more important than we thought
 Exploration is more important than we thought
 Hadoop style linear scaling must be mixed with ad hoc analysis

Thank You

whoami?
 Ted Dunning
– @ted_dunning
– tdunning@maprtech.com (MapR distribution for Hadoop)
– tdunning@apache.com (Mahout, Hadoop, Lucene, Zookeeper, Drill)
– ted.dunning@gmail.com (me)
 More info:
http://www.mapr.com/company/events/hadoop-in-finance-2012

Chicago Hadoop in Finance - Ted Dunning

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (9)

Similar to Chicago Hadoop in Finance - Ted Dunning

Similar to Chicago Hadoop in Finance - Ted Dunning (20)

More from MapR Technologies

More from MapR Technologies (20)

Recently uploaded

Recently uploaded (20)

Chicago Hadoop in Finance - Ted Dunning

Editor's Notes