Mastering the New Matrix - Big Data

Mastering
the New Matrix:
Big Data

Agenda
The Matrix
Data & Data Everywhere
Big Data & Four Vs
Seeing Big Data
Are you Neo?
2014 © BizKnowlogy - Philip Topham

The Matrix
NEO - Agent Smith

Thomas A. Anderson is a man living two lives. By day he is an
average computer programmer and by night a hacker known
as Neo. Neo has always questioned his reality, but the truth is
far beyond his imagination. Neo finds himself targeted by the
police when he is contacted by Morpheus, a legendary
computer hacker branded a terrorist by the government.
Morpheus awakens Neo to the real world, a ravaged
wasteland where most of humanity have been captured by a
race of machines that live off of the humans' body heat and
electrochemical energy and who imprison their minds within
an artificial reality known as the Matrix. As a rebel against the
machines, Neo must return to the Matrix and confront the
agents: super-powerful computer programs devoted to
snuffing out Neo and the entire human rebellion.

Agent Smith
Keeps us Trapped In Data

Trapped by Data
What is Data?

Taste
Data and the Human Senses
Touch Sight SoundSmell
How would you describe a Caribbean island?

• Number?
• Spreadsheet?
• Software?
• Smart phone?
What is Data?

An abstract
representation of
something…
Five. 5. 五 (go).
• Feet. Miles.
• Pennies. Dollars
• Atomic number1
• Speed. Lightyears.
• Miles Per Gallon.
• Acceleration.
Meters/Second2
Data is…..
1 Boron

Five. 5. 五 (go).
• Five Story Building
Data is…
An abstract representation of something…
made meaningful in context

Wind Speed
Data Goes Beyond the Five Senses
Temperature GPS Salinity
How would you describe a Caribbean island?

Microsensor Data
TasteTouch Sight SoundSmell
Touch screen Digitizer
Motion sensor Accelerometer
Ambient light sensor
Proximity sensor
Digital cameras
Gyroscope
Moisture Sensor
Cellular, WiFi, Bluetooth

Sensors, Machines, People
The Internet of Things
Data is Everywhere

Is Big Data
The Modern Matrix - Are we Trapped?

Humungous
Gargantuan
Immense
Enormous
Colossal
Tremendous
Monumental
Titanic
Elephantine
Brobdingnagian
Mammoth
Extensive
Sizable
Huge
Great
Vast
Voluminous
Spacious
Whopping
Astronomical
What is ???

Junk Drawer by Marc Miller
https://www.flickr.com/photos/markmarkmark/4551476025/
Velocity
Variety
Volume
Veracity
video
voice
text
sensors
many
truths
real-time
Four Vs
What is Big Data?

Recent History
➡ Hourly
Who attacked our website?
Realtime
➡ Realtime watching
Who is attacking our website now??
➡ Weekly / Monthly
What did you buy this week?
➡ Realtime spending alerts
Are you over spending now?
➡ Preferences
What web add do I show you?
➡ Realtime buying
Make instant suggestions
➡ Daily fraud alerts
Did any fraud happen today?
➡ Realtime fraud detection
Block fraudulent use now.
Velocity

Online in 60 Seconds is an infographic that was produced by qmee.com (2013)

From the dawn of time civilization to 2003, humankind
generated five exabytes of data. Now we product five
exabytes every two days…and the pace is accelerating
Eric Schmidt, Executive Director Google
Volume

What’s an Exabyte?
One non-stop DVD film, starting
50,000 years ago, when Homo
Sapiens, first arrived in North
America equals one Exabyte!
What’s an Exabyte?

Structured
➡ Highly organized data, saved in
repeatable ways that allows for easy
access, manageability, and use
between computer systems.
Unstructured
➡ Recorded data, saved without much
regard for interoperability across
computer systems (a specific program
must be used to use or search the
data)
http://www-i6.informatik.rwth-aachen.de/web/Research/speech_recog.html
Speech recognition waveformData entry form
Variety

Structured
➡ Databases
➡ Data Entry forms
➡ Sensor data (at a point in time)
➡ eg. GPS
Unstructured
➡ Freeform text, speech, movies
➡ Sensor data
➡ Chemical sensors
➡ Heat probes
➡ Wave detectors (sound, light, X-
rays, etc.)
➡ Motion (accelerometers,
gyroscopes)
➡ Magnetometers
➡ Pressure
Human data curation Little to no curation
Variety

Structured
➡ Known source
➡ Data curated (check for errors)
Unstructured
➡ Uncertain source
➡ Unclear trail of custody
➡ Occasionally purposeful untruths or
half-truths
➡ Hidden meaning (sarcasm)
➡ Unclear encoding or changing
encoding
Human data curation Little to no curation
Veracity

Learning to See Big Data

Standard Computing
The Old Way

Big Data
Does NOT FIT into
One computer
But…60 Trillion Web pages!!
http://www.google.com/insidesearch/howsearchworks/thestory/

Thousands
of Computers
Parallel Computing
How did Google Do it?

Divide and Conquer
Page Rank

Divide and Conquer
Count Words

Divide and Conquer
Count Words
frequency
Book 1:
“The quick brown fox…”
Book 2:
“Bill was as quick as a fox”

Divide and Conquer: Mapping Data
“The quick
brown
fox…”
“Bill was as
quick as a
fox”
Brown: 1
Fox: 1
Quick: 1
Bill: 1
Fox: 1
Quick: 1
Was: 1
Ignore little words

Divide and Conquer: Reducing Results
Brown: 1
Fox: 1
Quick: 1
Bill: 1
Fox: 1
Quick: 1
Was: 1
Bill: 1
Brown: 1
Fox: 2
Quick: 2
Was: 1
A-M
N-Z

If they Count Words why Page 17

Divide and Conquer
Add Location
Page 17Page 1

Divide and Conquer
Apply Information Theory
Dress
Carrot
Lemon
Cupcake +

Divide and Conquer
• Count words
• Word Importance
Title, Heading, Font Size
• Geolocation
• Related Importance

How did Amazon Do it?
Divide and Conquer
Count BehaviorsWesterns

Divide and Conquer
Count Behaviors
Did I browse (click)? buy?

National League Baseball

I’m not Amazon or Google?
Now What?

Are you Neo?
Will you look beyond the data?
Will you create a new story?

Big Data Thinking
➡ DATA - Unimportant alone
➡ CONTEXT - Creates meaning
➡ BIG DATA - Is overwhelming. It has no context
➡ BUSINESS - Our world, always has context
➡ DIVIDE and CONQUER - Split into small parts
Five. 5. 五 (go).
Five Feet
Five Gazillion ???
5% Increased Sales
1 + 2 + 3….

Big Data Skills
Base Skills:
Mathematics, Algorithms & Data structures
Filter&Mine
Acquire&Clean
Represent
Visualize
Domain Expertise
Data Skills
divide and conquer

Big Data Thinking
Patterns Emerge

Our Reflected Selves = Aggregated Data in Context
The Modern Matrix

ONENEO

Mastering the New Matrix - Big Data

Recommended

Recommended

More Related Content

Similar to Mastering the New Matrix - Big Data

Similar to Mastering the New Matrix - Big Data (20)

Recently uploaded

Recently uploaded (20)

Mastering the New Matrix - Big Data