This document discusses options for data analytic technology platforms to address big data problems. It begins by distinguishing between problems that truly involve big data versus just large data problems. Examples of big data problems include recommendations, financial analysis, internet security monitoring, social media network analysis, genomics, and sensor data. The key characteristics of big data problems are that the data sets are too large to download, data is generated rapidly requiring near real-time analysis, and the problems involve diverse data types. The document then outlines the governing principle for choosing a platform as processing needing to be close to the data due to data size. Examples of platforms used for different applications are discussed to illustrate this principle. The decision making process for choosing a platform is described as

1. Data Analytic Technology Platforms
Options and Tradeoffs
J Singh
January 7, 2014

2. Do you have a “Big Data” problem?
• Or do you have a big “data problem”?
© DataThinks 2013-14
2
2

3. Some Big Data problems (1)
• Recommendations
© DataThinks 2013-14
3
3

4. Some Big Data problems (2)
• Financial Analysis
– Really Big Data if we want Real Time analysis
© DataThinks 2013-14
4
4

5. Some Big Data problems (3)
• Internet Infrastructure Security Monitoring
© DataThinks 2013-14
5
5

6. Other Big Data problems
• Network graph problems (Social Media data)
• Bioinformatics problems (Genomics data)
• Physics/engineering problems (Sensor data)
•…
• Key characteristics
1. Not much common between problems
2. Data too big to download or upload.
3. Data changes fast, requires near-real-time analysis.
© DataThinks 2013-14
6
6

7. Just Big “Data Problems” (JBDP)
• Most problems on Kaggle
• Popular data sets (e.g., Amazon, Kaggle, …, data sets)
– If it can be downloaded,
– If it doesn’t change very often, …
– It’s a JBDP
© DataThinks 2013-14
7
7

8. About us
• Technology and analytics service based on Big Data
problems, focused on small & medium companies
• Analytics products
– App Kinetics – Application analytics for servicing users
– Pop Kinetics – Population analytics for targeting prospects
© DataThinks 2013-14
8
8

9. Background for this talk
• Experience building the “Kinetics”
products
– Harvest the kinetic energy of
your data for the benefit of your
business 
• Prior work.
– Like-you: an application that
trolls through Facebook data to
find users who like the same
things you do
© DataThinks 2013-14
9
9

10. Governing Principle for Platform Choices
• Big Data is difficult to move
– If you can move it easily, how big can it really be?
• Processing needs to be brought closer to the data
– Moving the data to processing is a losing proposition.
• Connector solutions for a database won’t scale
© DataThinks 2013-14
10
10

11. Implications of the Governing Principle
• Architecture has to be optimized across the entire pipeline
– Lesson learned:
•
•
•
•
The architecture is a giant jig-saw puzzle
Best of breed solutions may not fit!
Importance of caching in the pipeline
Vendor lock-in may be inevitable
– Cost, Data Volume and Bandwidth are primary drivers
• Different stacks for different applications
– App Kinetics: MongoDB-based stack
– Pop Kinetics: S3, Elastic Map Reduce-based stack
– Similarities: Google App Engine, Google Map Reduce
© DataThinks 2013-14
11
11

12. Governing Principle in Action
Function
Data
Collection
App Kinetics
Pop Kinetics
Custom “probes”
Like-You
Facebook API
Facebook API
Data Storage MongoDB
Amazon S3
Google Datastore
Analysis
Mongo M/R (JS)
PyMongo
(Python)
Amazon EMR
(Hadoop, Python)
Google App
Engine M/R
(Python)
Visualization
HTML+D3 (JS)
Text
HTML+JS
Recommend
ations
Text
© DataThinks 2013-14
12
12

13. The decision-making process
• An iterative process (like solving a jig-saw puzzle)
– Not linear or formulaic
• What is the objective?
• About the data
– Discovery?
– Volume
• If there is a market?
• If the concept is
feasible?
–
–
–
–
• Rate of Growth
– Velocity
– Variety
Time to market?
Hitting a cost target?
A scalable solution?
Minimizing lock-in?
– Format
– Location, location, …
© DataThinks 2013-14
13

14. Thank you
• J Singh
– Principal, DataThinks
• j.singh@datathinks.org
– Adj. Prof, WPI
© DataThinks 2013-14
14
14