More Related Content
Similar to Strata hadoop world 2015 context computing - jonas keynote - final (20)
Strata hadoop world 2015 context computing - jonas keynote - final
- 1. © 2015 IBM Corporation
Context Computing
Strata + Hadoop World 2015
Jeff Jonas, IBM Fellow
Chief Scientist, Context Computing
http://www.twitter.com/jeffjonas
www.jeffjonas.typepad.com
- 2. © 2015 IBM Corporation
Jeff Jonas
IBM Fellow
Chief Scientist, Context Computing
Founded Systems Research & Development (SRD) in 1985
Architected, designed, developed roughly 100 systems over
the last three decades
– Financial services
– Defense, intelligence
– Manufacturing
– Humanitarian efforts
Acquired by IBM in 2005
Currently focused on Context Computing, Sensemaking and
Privacy by Design
- 3. © 2015 IBM Corporation
No Context
Newsletter
Subscriber
- 4. © 2015 IBM Corporation
Context
“Better understanding something by taking into
account the things around it.”
- 5. © 2015 IBM Corporation
I ducked as the bat flew my way.
Another exciting baseball game.
- 6. © 2015 IBM Corporation
In Context
Social
Media
Influencer
Newsletter
Subscriber
Loyalty
Club Member
High Value
Customer
Job
Applicant Watch
Listed Party
- 7. © 2015 IBM Corporation
Context Accumulating
Context
Accumulation
Contextualized
Observations
Observation
(Any kind of data from
any kind of sensor)
- 8. © 2015 IBM Corporation
Context Informs Decisioning
Context
Accumulation
Contextualized
Observations
Observation
In Context
Decisioning
Act
Data Finds Data Relevance Finds You
Observation
(Any kind of data from
any kind of sensor)
- 9. © 2015 IBM Corporation
The Puzzle Metaphor
Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes, colors
What it represents is unknown – there is no picture on hand
Is it one puzzle, 15 puzzles, or 1,500 different puzzles?
Some pieces are duplicates, missing, incomplete or have errors
Some pieces may even be professionally fabricated lies
Until you take the pieces to the table, it is nearly impossible to assess the scene
- 10. © 2015 IBM Corporation
Puzzling Images: Courtesy Ravensburger © 2011
270 pieces
90%
200 pieces
66%
150 pieces
50%
6 pieces
2%
30 pieces
10%
(duplicates)
- 15. © 2015 IBM Corporation
Duplicates in Front Of Your Eyes
- 16. © 2015 IBM Corporation
First Duplicate Found Here
- 18. © 2015 IBM Corporation
Incremental Context – Incremental Discovery
6:40pm START
22min “Hey, this one is a duplicate!”
35min “I think some pieces are missing.”
37min “Looks like a bunch of hillbillies on a porch.”
44min “Hillbillies, playing guitars, sitting on a porch, near a barber sign
and a banjo!”
- 20. © 2015 IBM Corporation
Incremental Context – Incremental Discovery
47min “We should take the sky and grass off the table.”
2hr “Let’s switch sides, and see if we can make sense of this from
different perspectives.”
2hr10m “Wait, there are three … no, four puzzles.”
2hr18m “I think you threw in a few random pieces.”
- 22. © 2015 IBM Corporation
How Context Accumulates
With each new observation one asserts: 1) unrelated; 2) related; or 3) connected
Must favor the false negative
New observations sometimes reverse earlier assertions
Some observations produce novel discovery
The emerging picture helps focus collection interests
- 23. © 2015 IBM Corporation
Big Data [in context]. New Physics.
More data: better the predictions
– Lower false positives
– Lower false negatives
More data: bad data good
– Suddenly glad your data is not perfect
More data: less compute
- 24. © 2015 IBM Corporation
Big Data
Pile of ______ Information In Context
- 25. © 2015 IBM Corporation
One Essential Form of Context: “Entity Resolution”
Is it 5 people each with 1 account or is it 1 person with 5 accounts?
Is it 20 cases of SARS in 20 cities or one case reported 20 times?
If one cannot count, one cannot estimate vector or velocity (direction, speed).
Without vector and velocity prediction is nearly impossible.
- 26. © 2015 IBM Corporation
Who is Fang Wong?
Fang Wong
Top 100 Customer
F A Wong
Seattle, DOB: 6/12/82
Former Customer
@FangWong
2.5M Followers
FangWong@Email.com
Newsletter Subscriber
Fang Wong
FangWong@Email.com
Marketing Department’s
Prospect List
- 27. © 2015 IBM Corporation
Resolving the Fang Wong
Fang Wong
Top 100 Customer
F A Wong
Seattle, DOB: 6/12/82
Former Customer
@FangWong
2.5M Followers
FangWong@Email.com
Newsletter Subscriber
Fang Wong
FangWong@Email.com
Marketing Department’s
Prospect List
- 28. © 2015 IBM Corporation
Resolving the Fang Wong
Fang Wong
Top 100 Customer
2.5M Followers
Newsletter Subscriber
- 29. © 2015 IBM Corporation
Graphing the (resolved) Fang Wong
Bill Smith
Member of the Board
Employee
Customer
Customer
Fraudster
Fang Wong
Top 100 Customer
2.5M Followers
Newsletter Subscriber
- 30. © 2015 IBM Corporation
Contextualizing Sandy Maden
Bill Smith
Member of the Board
Sandy Maden
New Account
Employee
Lives With
Co-signer
Former
Customer
Customer Customer
Customer
Fraudster
Fang Wong
Top 100 Customer
2.5M Followers
Newsletter Subscriber
- 31. © 2015 IBM Corporation
“Entities”
Bill Smith
Member of the Board
Lives With
Co-signer
Sandy Maden
New AccountFormer
Customer
Employee
Customer Customer
Customer
Fraudster
Fang Wong
Top 100 Customer
2.5M Followers
Newsletter Subscriber
Company
Boat
Plane
Asteroid
Car
- 33. © 2015 IBM Corporation
Single Detection
Image courtesy of: Eva Lilly, Institute of Astronomy, University of Hawaii
- 34. © 2015 IBM Corporation
From Orphans to Orbits
Single
Detections
(trash)
Tracklette
Track
Orbit
Forecasting
Named entity: S100ZUtza
Single Detection
(orphan)
Anticipation
- 35. © 2015 IBM Corporation
http://www.space.com/7854-slam-asteroids-suspected-space-collision.html
- 36. © 2015 IBM Corporation
"We have directly
observed a collision
between asteroids for
the first time, instead of
having to infer that
they happened from
million-year-old
remains."
Colin Snodgrass
Planetary Scientist
Max Planck Institute for Solar System
Research
- 37. © 2015 IBM Corporation
Geospatial Context via “Space Time Boxes”
- 38. © 2015 IBM Corporation
Detecting Colocation
TIME
1 day
1 hour
Determine
encounter
distance
and time
Space Time Boxes
- 39. © 2015 IBM Corporation
Computing 600k Asteroid Interactions over 25 Years
4-5 orders of magnitude improvement
Initial Analysis
Adding 1 New Trajectory
Space-Time Box Method
2,880 CPU hours
15 CPU minutes
N-body Simulation Method
10,000,000 CPU hours
4,000 CPU hours
- 40. © 2015 IBM Corporation
Asteroid vs. Asteroid Encounters
Encounter Distance Asteroid 1 Size Asteroid 2 Size
May 1, 2032 299km 00A9170 2-4km 0008758 4-9km
Nov 24, 2016 449km 00P5634 1-2km 0055711 2-5km
Jan 11, 2018 449km K08E88J 530-1200m 00N0062 2-4km
- 41. © 2015 IBM Corporation
June 12th, 2015
Hi Jeff & the gang,
I have great news! On Tuesday I happened to observe a close encounter
you guys predicted - one 1 km and the other one 2 km in diameter!
To my knowledge this is the first case ever of direct observation of a close
encounter in the small main belt asteroids. The closest point of encounter
unfortunately happened during bright daylight in Hawaii, so I missed it …
Cheers!
Eva -
- 42. © 2015 IBM Corporation
Image courtesy of: Eva Lilly, Institute of Astronomy, University of Hawaii
- 44. © 2015 IBM Corporation
Action
Red Analytics
Green Analytics
Blue Analytics
Observation
Space
Old School: Isolated Analytics
- 45. © 2015 IBM Corporation
Observation
Space
ActionInformation
In Context
Next: General Purpose Context Computing
Data Finds Data Relevance Finds You
Context Computing
- 46. © 2015 IBM Corporation
Observation
Space
ActionInformation
In Context
Data Finds Data Relevance Finds You
Context Computing
Helping Focusing Human Attention
General Purpose
• Marketing
• Customer service
• Fraud detection
• Asteroid hunting
Simultaneously!
- 47. © 2015 IBM Corporation
Making Data Work: Recommendations
Widen the observation space
Accumulate context to improve understanding
Deliver significantly higher quality outcomes … everywhere
– Life sciences
– Financial services
– Public safety
Leverage Hadoop/Spark to accelerate innovation
- 48. © 2015 IBM Corporation
More
Blog: www.jeffjonas.typepad.com
Email: JeffJonas@us.ibm.com
Next: San Francisco, Nov 10-12, @Datapalooza
- 49. © 2015 IBM Corporation
Context Computing
Strata + Hadoop World 2015
Jeff Jonas, IBM Fellow
Chief Scientist, Context Computing
http://www.twitter.com/jeffjonas
www.jeffjonas.typepad.com