Do Big Data and NoSQL Fit Your Needs?

The VP R&D Open Seminar

Big Data Workshop
moshe.kaplan@brightaqua.com
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

Presentation Objectives

http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/


Why Do I Care?
From 0 to 100 (US mass adaptation)
Phone:
Radio:
TV:
Mobile:
Internet:
Facebook:

100 yrs
40 yrs
30 yrs
20 yrs
10 yrs
2 yrs


The Internet Industry


The Prime Suspect


Assumptions…


Where did it Fail?
Get an Answer, Fast and Cheap


Where did it Fail?
I Just Want “Class Persistency Storage”
and Changing Schema on Demand


Where did it Fail?
Be Always Available, Even w/ an Old Answer


Where did it Fail?
Get Me Fast and Good Enough Answer


Where did it Fail?
Data is Too Big, and Storage is $$$
But CPU and Network are Even More

http://www.powerbyte.com/Isilon.html


It is all great, but…
I Need to Meet Compliance

http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg


I Need a Vendor


I Need Reporting
http://www.novell.com/communities/node/5851/get-ready-sentinel-61


I Need Transactions

http://www.novell.com/communities/node/5851/get-ready-sentinel-61


We Need Training for the Data Analysts
db.article.aggregate(
{ $group : {
_id : "$author",
< GROUP BY author
docsPerAuthor : { $sum : 1 }, < SUM(1) = N
viewsPerAuthor : { $sum : "$pageViews" }
< SUM(pageViews)
}}
);


General Architecture

Client

Server

Database


Apps


CLIENT SIDE

It’s a World Made of Pixels



SERVER SIDE

General Strategies
Online
In Memory Databases and Q
Log files processing


In Memory Databases


700 Inserts/Sec

In Memory Engine
3000 Inserts/Sec
Amazon
AWS
Standard
Large
Instance

InnoDB Engine
700 Inserts/Sec



General Strategies

DATA SIDE

Strategy A - Sharding


Strategy B – MapReduce



Strategy C - NoSQL
insert
get
multiget

remove
truncate

<Key, Value>
http://wiki.apache.org/cassandra/API



MongoDB

DOCUMENT DATABASES

When Should I Choose NoSQL?

•
•
•

Eventually Consistent
Document Store
Key Value
http://guyharrison.squarespace.com/blog/tag/nosql


Same Terminology

Database
Table
Row

 Database
 Collection
 Document


A Blog Case Study in RDBMS

http://www.slideshare.net/nateabele/building-apps-with-mongodb


And as a SW Engineer would like it to be…

http://www.slideshare.net/nateabele/building-apps-with-mongodb


Classic RDBMS Replication


Auto Selection Using Quorum

Selection Methods:
• Low Priority
• Hidden
• (Weighted) Voting


MongoDB and Sharding

http://www.10gen.com/products/mongodb



Cassandra

EVENTUALLY CONSISTENT

Product Architecture
http://horicky.blogspot.co.il/2010/10/bigtable-model-with-cassandra-and-hbase.html


Key Concepts
Fast Answer
Not Always Right
Can Lose Data
Autosync

Bottom Line:






Use the memory
Multiple instances
Multiple instances
Client timestamp

Integrated Memcached + MySQL


Azure Table Storage: Key Concepts
Very Large Tables 
Partitioning
Get by Key

Portioning Key
Sort

Single Sort Key
Simple Rows

Basic Types
No Joins, No Grouping, No Multiple Sorting
Bottom Line:
Simple Very Large Tables  LDAP



Hadoop

MAP REDUCE

Count Pageviews by Date

Map The Challenge
(Count on every node)

Reduce The Answers
(Get a Single Answer)


Word Count
function map(String name, String document):
// name: document name
// document: document contents
for each word w in document:
emit (w, 1)
function reduce(String word, Iterator partialCounts):
// word: a word
// partialCounts: list of aggregated counts
sum = 0
for each pc in partialCounts:
sum += ParseInt(pc) emit (word, sum)


Hadoop Architecture


Hadoop as a Service

http://www.windowsazure.com/en-us/manage/services/hdinsight/get-started-hdinsight/


Excel Integration

http://www.windowsazure.com/en-us/manage/services/hdinsight/get-started-hdinsight/



COLUMN ORIENTED
DATABASES

Column Oriented Database Databases
+ INSERT
+ GROUP BY, SUM …
+ Compression
- Join
- DELETE, UPDATE


Cloud Services

http://www.theregister.co.uk/2012/11/28/amazon_aws_redshift_data_warehousing/


Google Big Query



FEEDBACK SYSTEMS

Customer Feedback (Kampyle)


Heatmaps (Clicktale)


User Interaction (Totango)



MongoDB

BUSINESS MONITORING

Funnel Monitoring

http://blog.clicktale.com/2011/01/18/new-clicktale-product-launches-for-2011/


Monitoring is not your CPU utilization



SHARDING IN DEPTH

Sharding Again


Vertical Sharding


Horizontal Sharding
Static Hashing
Complex growth
Simple

Mod 10 = 0
Mod 10 = 1
Mod 10 = 2
Mod 10 = 3
Mod 10 = 4
Mod 10 = 5
Mod 10 = 6
Mod 10 = 7
Mod 10 = 8
Mod 10 = 9


Horizontal Sharding
Key locations are defined in a directory
Simple growth
Directory is SPOF
The Directory Can
be Very Large


Horizontal Sharding
Static Hashing with Directory Mapping
Simple Growth
The Small Directory Can be Cached on
Each App Server

Mod 1000 = 4


Horizontal Sharding
Each key is signed by the DB#
on creation
Simple growth
The Key Store Can be Cached on
Each App Server


The Bottom Line: Grow ∞

Thank you!
and Keep Performing!

Moshe Kaplan


Do Big Data and NoSQL Fit Your Needs?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Do Big Data and NoSQL Fit Your Needs?

Similar to Do Big Data and NoSQL Fit Your Needs? (20)

More from Moshe Kaplan

More from Moshe Kaplan (16)

Recently uploaded

Recently uploaded (16)

Do Big Data and NoSQL Fit Your Needs?