• Save
Do Big Data and NoSQL Fit Your Needs?
Upcoming SlideShare
Loading in...5
×
 

Do Big Data and NoSQL Fit Your Needs?

on

  • 1,259 views

Many of us consider

Many of us consider

Statistics

Views

Total Views
1,259
Views on SlideShare
925
Embed Views
334

Actions

Likes
1
Downloads
0
Comments
0

26 Embeds 334

http://top-performance.blogspot.co.il 181
http://top-performance.blogspot.com 61
http://top-performance.blogspot.in 19
http://top-performance.blogspot.com.es 13
http://plus.url.google.com 7
http://top-performance.blogspot.ru 6
http://top-performance.blogspot.co.uk 6
http://top-performance.blogspot.com.au 5
http://top-performance.blogspot.fr 5
http://top-performance.blogspot.de 5
http://top-performance.blogspot.it 3
http://top-performance.blogspot.hk 3
http://top-performance.blogspot.ro 2
http://top-performance.blogspot.jp 2
http://top-performance.blogspot.com.br 2
http://top-performance.blogspot.cz 2
http://news.google.com 2
http://top-performance.blogspot.kr 2
http://top-performance.blogspot.ie 1
http://top-performance.blogspot.se 1
http://top-performance.blogspot.sg 1
http://top-performance.blogspot.gr 1
http://top-performance.blogspot.ca 1
http://top-performance.blogspot.com.tr 1
http://top-performance.blogspot.com.es.nyud.net 1
http://top-performance.blogspot.ch 1
More...

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Do Big Data and NoSQL Fit Your Needs? Do Big Data and NoSQL Fit Your Needs? Presentation Transcript

  • The VP R&D Open Seminar Big Data Workshop moshe.kaplan@brightaqua.com http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Presentation Objectives http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/ http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Why Do I Care? From 0 to 100 (US mass adaptation) Phone: Radio: TV: Mobile: Internet: Facebook: 100 yrs 40 yrs 30 yrs 20 yrs 10 yrs 2 yrs http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com View slide
  • The Internet Industry http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com View slide
  • The Prime Suspect http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Assumptions… http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Where did it Fail? Get an Answer, Fast and Cheap http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Where did it Fail? I Just Want “Class Persistency Storage” and Changing Schema on Demand http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Where did it Fail? Be Always Available, Even w/ an Old Answer http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Where did it Fail? Get Me Fast and Good Enough Answer http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Where did it Fail? Data is Too Big, and Storage is $$$ But CPU and Network are Even More http://www.powerbyte.com/Isilon.html http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • It is all great, but… I Need to Meet Compliance http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • It is all great, but… I Need a Vendor http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • It is all great, but… I Need Reporting http://www.novell.com/communities/node/5851/get-ready-sentinel-61 http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • It is all great, but… I Need Transactions http://www.novell.com/communities/node/5851/get-ready-sentinel-61 http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • It is all great, but… We Need Training for the Data Analysts db.article.aggregate( { $group : { _id : "$author", < GROUP BY author docsPerAuthor : { $sum : 1 }, < SUM(1) = N viewsPerAuthor : { $sum : "$pageViews" } < SUM(pageViews) }} ); http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • General Architecture Client Server Database http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com Apps
  • The VP R&D Open Seminar CLIENT SIDE http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • It’s a World Made of Pixels http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar SERVER SIDE http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • General Strategies Online In Memory Databases and Q Log files processing http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • In Memory Databases http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • 700 Inserts/Sec In Memory Engine 3000 Inserts/Sec Amazon AWS Standard Large Instance InnoDB Engine 700 Inserts/Sec http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar General Strategies DATA SIDE http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Strategy A - Sharding http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Strategy B – MapReduce http://blogs.microsoft.co.il/blogs/vprnd http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Strategy C - NoSQL insert get multiget remove truncate <Key, Value> http://wiki.apache.org/cassandra/API http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar MongoDB DOCUMENT DATABASES http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • When Should I Choose NoSQL? • • • Eventually Consistent Document Store Key Value http://guyharrison.squarespace.com/blog/tag/nosql http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Same Terminology Database Table Row  Database  Collection  Document http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Same Terminology Database Table Row  Database  Collection  Document http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • A Blog Case Study in RDBMS http://www.slideshare.net/nateabele/building-apps-with-mongodb http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • And as a SW Engineer would like it to be… http://www.slideshare.net/nateabele/building-apps-with-mongodb http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Classic RDBMS Replication http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Auto Selection Using Quorum Selection Methods: • Low Priority • Hidden • (Weighted) Voting http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • MongoDB and Sharding http://www.10gen.com/products/mongodb http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar Cassandra EVENTUALLY CONSISTENT http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Product Architecture http://horicky.blogspot.co.il/2010/10/bigtable-model-with-cassandra-and-hbase.html http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Key Concepts Fast Answer Not Always Right Can Lose Data Autosync Bottom Line:     Use the memory Multiple instances Multiple instances Client timestamp Integrated Memcached + MySQL http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Azure Table Storage: Key Concepts Very Large Tables  Partitioning Get by Key  Portioning Key Sort  Single Sort Key Simple Rows  Basic Types No Joins, No Grouping, No Multiple Sorting Bottom Line: Simple Very Large Tables  LDAP http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • MongoDB and Sharding http://www.10gen.com/products/mongodb http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar Hadoop MAP REDUCE http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Count Pageviews by Date Map The Challenge (Count on every node) Reduce The Answers (Get a Single Answer) http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Word Count function map(String name, String document): // name: document name // document: document contents for each word w in document: emit (w, 1) function reduce(String word, Iterator partialCounts): // word: a word // partialCounts: list of aggregated counts sum = 0 for each pc in partialCounts: sum += ParseInt(pc) emit (word, sum) http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Hadoop Architecture http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Hadoop as a Service http://www.windowsazure.com/en-us/manage/services/hdinsight/get-started-hdinsight/ http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Excel Integration http://www.windowsazure.com/en-us/manage/services/hdinsight/get-started-hdinsight/ http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar COLUMN ORIENTED DATABASES http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Column Oriented Database Databases + INSERT + GROUP BY, SUM … + Compression - Join - DELETE, UPDATE http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Cloud Services http://www.theregister.co.uk/2012/11/28/amazon_aws_redshift_data_warehousing/ http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Google Big Query http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar FEEDBACK SYSTEMS http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Customer Feedback (Kampyle) http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Heatmaps (Clicktale) http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • User Interaction (Totango) http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar MongoDB BUSINESS MONITORING http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Funnel Monitoring http://blog.clicktale.com/2011/01/18/new-clicktale-product-launches-for-2011/ http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Monitoring is not your CPU utilization http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The VP R&D Open Seminar SHARDING IN DEPTH http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Sharding Again http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Vertical Sharding http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Horizontal Sharding Static Hashing Complex growth Simple Mod 10 = 0 Mod 10 = 1 Mod 10 = 2 Mod 10 = 3 Mod 10 = 4 Mod 10 = 5 Mod 10 = 6 Mod 10 = 7 Mod 10 = 8 Mod 10 = 9 http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Horizontal Sharding Key locations are defined in a directory Simple growth Directory is SPOF The Directory Can be Very Large http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Horizontal Sharding Static Hashing with Directory Mapping Simple Growth The Small Directory Can be Cached on Each App Server Mod 1000 = 4 http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • Horizontal Sharding Each key is signed by the DB# on creation Simple growth The Key Store Can be Cached on Each App Server http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
  • The Bottom Line: Grow ∞ Thank you! and Keep Performing! Moshe Kaplan http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com