BigML's take on Big Data

BigML Inc, 2012 Geneva, October 12, 2012

Agenda

·•Short intro
·•The Big Data Revolution
·•What is BigML?
·•Behind the scenes
·•Coming down the pike
·•Hacking with the BigML API
BigML Inc, 2012 Geneva, October 12, 2012 2

Francisco J Martin
Background:
• 5-year degree in Computer Science, UPV
• Ph.D. in Artificial Intelligence, UPC
• Postdoc (Machine Learning), Oregon State University
• Founder and CEO at iSOCO
• Founder and CEO at Strands
• Co-authored 6 patents acquired by Apple Inc
• Directly raised $75+MM in venture capital and cashed
out additional $18+MM for early investors
• Directly sold and negotiated $30+MM in licenses

BigML:
• Co-founder and CEO
• Joined: January 2011
• Tasks: Product conceptualization, design, and architecture
• Develops: BigML middle-end and public API
• 1202 (19%) of commits to total BigML code base

Academia vs the Real-world

Neo, sooner or
later you're going
to realize, just as
I did, that there's
a difference
between knowing the
path, and walking
the path


Walking the data path
Large-scale
Machine Learning

Recommender
Systems

Everything
Machine Learning

Personalization
Music, video,
Multi-agent ﬁtness, ﬁnance
Learning Intrusion
Detection

E-commerce

ata
D
8-queen problem

1996 1999 2002 2004 2011 2012
Academia iSOCO Academia Strands Inc BigML Inc

BigML Status
·•Founded in Jan 2011
·•9 FTE, 1 PT
·•5 Ph.Ds
·•4 patent applications
·•Advisors and BA:
US Patent Application No. 61/555,615
For: VISUALIZATION AND INTERACTION WITH COMPACT REPRESENTATION OF DECISION TREES
Filed: November, 2011

For: METHODS FOR BUILDING AND USING DECISION TREES IN A DISTRIBUTED ENVIRONMENT

For: EVOLVING PARALLEL SYSTEM TO AUTOMATICALLY IMPROVE THE PERFORMANCE OF DISTRIBUTED SYSTEMS

For: SYSTEM AND METHODS TO EXCHANGE ACTIONABLE PREDICTIVE MODELS IN A VIRTUAL MARKETPLACE
Filed: October, 2012


From the trenches

Beneath Hill 60 BigML Team


Agenda

·•Short intro
·•What is BigML?

Big Data
What is Big Data? What is a Data Scientist?

How not to start with Big Data? What is Data-driven
Decision Making?


Trends

http://strata.oreilly.com/2011/08/building-data-startups.html


What’s Big Data?

Big Data means way
too many different
things to
many different people

“when the human cost of making the decision of throwing
something away became higher than the machine cost of
continuing to store it” George Dyson

What’s Big Data?
The 3 v’s The 3 I’s
Volume Immediate
(big, enormous, huge, vast, immense, very In the sense that you need to do something
large, etc) about it

Variety Intimidating
(heterogenous, diverse, complex, multiple What if you do not?
sources, sensors, etc)

Velocity Ill-defined
(speed, dynamic real-time, streamed, etc) What is it? Anyway?

BigML Inc, 2012
Data matters!!! Geneva, October 12, 2012 12

Machine Learning
Even if we, human
beings, are learning
machines, we are really
bad at processing small
amounts of data

Machines are good at
quickly processing huge
amounts of data.
Machine Learning can
make them learn from
data

It’s all about machine learning

Forget plastics.
It’s all about
machine learning
http://www.youtube.com/watch?v=PSxihhBzCjk

It's as if the machines have been in training all their lives to
adapt and make use of the Big Data now being thrown at them
- a combination of Moore's Law and the cloud mixed in with
Machine Learning finally makes it all possible. --- Jeff Bussgang


Learning from Data
Unknown Model
f : X -> Y
Example: ideal credit approval formula
f1 f2 fn label
x1
Training Examples
(x1, l1), (x2, l2), ..., (xN, lN)
xN
Example: historical records of credit customers

Models Final Model
M Learning g~f
Example: set of candidate Algorithm Example: learned credit
credit approval formulas approval formula

Based on Learning from Data by Y. Abu-Mostafa, M. Magdon-Ismail and H. Lin


What’s Big Machine Learning?

Volume Large-scale machine
What to do when data is too big to fit within the
system memory of a single computer? learning
Clean, refine, update, join, merge, aggregate,
Variety structure or deconstruct data until it matches
the required input format or (why not) just
generate/store data in the right format

Velocity Stream Algorithms


Machine Learning
...or you can deal with that!


Does More Data beat Better Algorithms?

More features
More examples

The Unreasonable Effectiveness of Data

More Data or Better Models.
Xavier Amatriain


What’s Big Data?

Global realization that
learning from data (i.e., Machine Learning)
can help us better analyze our past, understand our present,
and predict our future. --- Francisco J Martin

Data Past Present Future


Big Data

Decision Making?


Is Wikipedia right?

Really? Seriously?? Are you kidding me???

Data can’t be wrong?


McKinsey can’t be wrong
Critical Shortage Of “Data Scientist”
Talent Predicted By 2018


HBR can’t be wrong


Wikipedia is right!


If Data Scientists don’t exist
can they be created?


The first Data Scientist

Computer
Statistician
Scientist

Mathematician

Hans’ brain, the first Data Scientist

The magic formula

A data scientist is“part
analyst, part artist.”

Anjul Bhambhri,Vice President of Big Data
Products at IBM


Are Data Scientists super heroes?


The most powerful human super hero

http://photos.oregonlive.com/photo-essay/2012/06/ashton_eaton_sets_decathlon_wo.html

Are Data Scientists super heroes?

High school
Events Decathlon World Record World Record
World Record

100 m 10.21 10.08 9.58

Long Jump 8.23 m 8.16 m 8.95 m
Shot Put 14.20 m 20.65 m 23.12 m
High Jump 2.05 m 2.31 m 2.45 m
400 m 46.70 44.69 43.18

110 m hurdles 13.70 13.74 12.80

Discus throw 42.81 m 61.38 m 74.08 m
Pole Vault 5.30 m 5.56 m 6.14 m
Javelin Throw 58.87 m 73.74 m 98.48 m
1500m 4:14.48 3:38.26 3:26.00


The Wikipedia is always right!


BigML’s Data Science Team
UI
Design
Visualization

Oscar Rovira, MSc*

Infrastructure, Cloud-based
Bea Garcia, BSc
Product Design

Common Sense
Business and Justin Donaldson Ph.D.

Computing
Architecture, Francisco J Martin, PhD
Software Design,
Distributed Systems Jos Verwoerd, MSc

Poul Petersen, MSc

Large-scale and learning
algorithm implementation Jao, PhD

Charles Parker,
Machine Learning Research PhD Adam Ashenfelter, MSc

Tom Dietterich, PhD


Take Away

Oscar Rovira, MSc*

Bea Garcia, BSc

Justin Donaldson Ph.D.

Francisco J Martin, PhD

Jos Verwoerd, MSc

Poul Petersen, MSc

Jao, PhD

Charles Parker,
PhD Adam Ashenfelter, MSc

Tom Dietterich, PhD

So instead of trying to quickly create “mediocre data scientists”, Universities
should focus on creating excellent mathematicians, statisticians, computer
scientists, software architects, designers, etc who are fabulous team players


Big Data

Decision Making?


Iris Dataset

http://en.wikipedia.org/wiki/Iris_ﬂower_data_set


Digesting Big Data
Assimilation
(making insights actionable)
Almost no attention!!!
(reject bad data, wrong insights)

Absorption
Egestion

(deriving insights)

Digestion
(processing)
Too much attention!!!
Ingestion
(capturing and storing)


Big Data meets Hadoop
·•Hadoop has been excessively
promoted as the way to make
Big Data problems easy.

·•There are quite a few vendors
pushing different Hadoop flavors
to the market.

However, Hadoop is complex, slow, expensive
and batch

Big Data and Hadoop
Running Hadoop on a cluster - The New IT sport of 2012


Real-Time Hadoop?

Really? Seriously?? Are you kidding me???

Why not Hadoop?
Hadoop on a cluster is the right solution for jobs where the input data is multi-terabyte or larger

·•Evidence suggests that many MapReduce-like jobs process relatively small input data sets
(less than 14 GB)
·•Iterative-machine learning algorithms, do not map trivially to MapReduce.
·•Memory has reached a GB/$ ratio such that it is now technically and financially feasible to
have servers with 100s GB of DRAM
·•In terms of hardware and programmer time, this may be a better option for the majority of
data processing jobs.
Rowstron, A. et al, Nobody ever got ﬁred for using Hadoop
on a cluster, Microsoft Research, Cambridge, 2012

·•Hadoop is bad at iterative algorithms: High job startup costs and awkward to retain state
across iterations
·•High sensitivity to skew: iteration speed bounded by slowest task.
·•Potentially poor cluster utilization: must shuffle all data to a single reducer.
Large-Scale Machine Learning at Twitter, Jimmy Lin


Making Big Data Small
Hadoop Streaming Algorithms

·•Complex ·•Simple
·•Slow ·•Fast
·•Batch ·•Real-time
·•Expensive ·•Cheap
Noel Welsh, Strata conference, London, October 2012


Self-imposed Shackles
Once a baby elephant accepts the
limitation imposed on him it becomes
a permanent belief, or in his case, a
conditioned reaction. Now as the
elephant grows into adulthood, he
has the power to easily pull the
stake out of the ground, but his
conditioning has taught him that the
effort will not only be futile, it will be
painful as well.
http://www.selfgrowth.com/articles/Martinez1.html

Tackling Big Data with Hadoop on a cluster is like
self-imposing shackles on your own project


Starting with Big Data
•Buy a few machines and set up a cluster.
•Installing and running any flavor of Hadoop.
•Figure out how to implement complex map-reduce
algorithms to compute a few analytics.

•Start with a very small data sample.
•Use free or cloud-based tools to build a first predictive
model that you can understand.
•Check if the model gives you any practical insight.
•Use the model to generate predictions and see if it can
improve your performance.
•Check how more data can improve the model.
•Check if more sophisticated models can beat your model
•Iterate.
•Check if the volume, variety, and velocity of your data
require a behind-the-firewall/ cloud solution or a batch/stream
solution.

Big Data

How not to deal with Big Data? What is Data-driven
Decision Making?


Data-Driven Decisions
Automated, data-driven decisions will significantly
impact more industries than any other information
system since “computers” were people

http://www.nytimes.com/2011/04/24/business/24unboxed.html


The “HiPPO” (Highest Paid Person’s Opinion) is dead


Predictive Analytics

Descriptive Analytics Predictive Analytics
Traditional, backward-looking business Machine Learning
analytics


Predictive Model

“The goal of a predictive
model is not
to predict the future but
to help you make a better
decision in the present”

Taken from Paul Saffo, HBR


Data-Driven Decision Making

Analytics and
Predictive Analytics
combined with
Experience&Intuition

It’s time to switch the attention

Assimilation
(making insights actionable)
More attention!!!
(reject bad data, wrong insights)

Absorption More focus on the models and
Egestion

(deriving insights) how to operationalize them than
on the infrastructure to generate
them
Digestion
(processing)
less attention!!!
Ingestion
(capturing and storing)


Take aways
•Big Data is just data
•It’s all about machine learning

•Try to excel in one of the data science disciplines

•Don’t shackle yourself to the wrong platform
•Trying to predict the future can help you make the right
decision in the present

•Focus on evaluation and actionability of models and not
on how they are built

Agenda

·•Short intro
·•What is BigML?

BigML Goal

Highly Scalable, Cloud-based Machine Learning Service
Simple, Easy-to-Use and Seamless-to-
Integrate


BigML vs ML

You can deal ...or you can deal with that!
with this...

BigML 1-click model


BigML vs Big Data

You can deal ...or you can deal with that!
with this...

BigML 1-click model


How it Works


Machine Learning Made Easy

True


Simple is not easy

“Any fool can make something complicated. It
takes a genius to make it simple.”
― Woody Guthrie


Fully Web based


RESTful API


Agenda

·•Short intro
·•What is BigML? - Demo

Agenda

·•Short intro
·•What is BigML?

BigML’ Software Architecture
Front-end
[Neutronia]

[Medusa]

[CuriousYellow]

[Sky]

Middle-end
[Apian]

Backend
[Wintermute]

Infrastructure
[Sauron] Boto, Fabric

BigML’s AWS-based Architecture


Why Tree Models?

·•Highly scalable
·•Graphically representable and interactive
·•Easily understandable
·•Easily translatable into rules, PMML, and code.
·•Easily upgradable with ensembles: boosting, bagging, and
random forests, etc
·•Top performers! http://www.niculescu-mizil.org/papers/empirical.icml06.pdfempirical.icml06.pdf

BigML Histograms
BigML's trees and dataset summaries use histograms with the following traits:

Streaming Memory constrained Dynamic
Data is never kept in memory The less memory allocated, the The histogram bins adjust
but needs only one pass over lossier the compressed themselves as they observe the
the data to capture the distribution. data.
distribution.

Robust to ordered Merge friendly More...
data
So it works even if the data For parallelization and http://blog.bigml.com/
stream is non-stationary distribution. 2012/06/18/bigmls-fancy-
histograms/


BigML Streaming Trees
BigML's trees are:

CART Grown breadth first

Classification & Regression So partial trees are
Trees meaningful

Built Hoeffding-style Friendly for parallelization

So they consume streaming Can work over multiple
data and can split "early" cores or multiple computers


Growing a Streaming Tree

·•Each split breaks the data
into subsets.

·•The split should make the
subsets as distinct from one
another as possible.

·•Subsets are chosen to
maximize information gain
(classification) or minimize
squared error (regression).


Distributed Streaming Trees


Streaming Trees - Early Splits


Agenda

·•Short intro
·•What is BigML?

Automatic Evaluations


A marketplace for predictive models


Simple is not easy

“Any fool can make something complicated. It
takes a genius to make it simple.”
― Woody Guthrie


Machine Learning Made Easy

True


Agenda

·•Short intro
·•Demo

Back to the trenches

Gallipoli

Good Reading
Big Data Trends - David Feinleib
http://www.slideshare.net/bigdatalandscape/big-data-trends

Hey Graduates: Forget Plastics - It's All About Machine Learning. Jess Bussgang.
http://bostonvcblog.typepad.com/vc/2012/05/forget-plastics-its-all-about-machine-learning.html

More Data or Better Models. Xavier Amatriain
http://technocalifornia.blogspot.ch/2012/07/more-data-or-better-models.html

Making Big Data Small. Noel Welsh
http://strataconf.com/strataeu/public/schedule/detail/25984

Data Killed the HiPPO star. Jeff Jordan, Andreessen Horowitz
http://gigaom.com/2012/02/18/data-killed-the-hippo-star/

When There’s No Such Thing as Too Much Information. Steve Lohr
http://www.nytimes.com/2011/04/24/business/24unboxed.html

Nobody ever got ﬁred for using Hadoop on a cluster. Antony Rowstron, Dushyanth Narayanan, Austin Donnelly, Greg
O’Shea, Andrew Douglas
http://research.microsoft.com/pubs/163083/hotcbp12%20ﬁnal.pdf

Six Rules for Effective Forecasting. Paul Saffo
http://www.usc.edu/schools/annenberg/asc/projects/wkc/pdf/200912digitalleadership_saffo.pdf

Large-scale Machine Learning at Twitter. Jimmy Lin and Alek Kolcz
http://www.umiacs.umd.edu/~jimmylin/publications/Lin_Kolcz_SIGMOD2012.pdf


BigML's take on Big Data

Recommended

Recommended

More Related Content

What's hot

What's hot (16)

Similar to BigML's take on Big Data

Similar to BigML's take on Big Data (20)

Recently uploaded

Recently uploaded (20)

BigML's take on Big Data