NoSQL databases and managing big data

NoSQLDatabases &Managing Big Data

Talking aboutWhat is BIG DataNoSQLMongoDBFuture of BIG Data

@spf13 AKASteve Francia15+ years buildingthe internet Father, husband, skateboarderChief Solutions Architect @responsible for drivers,integrations, web & docs

Company behind MongoDBOfﬁces in NYC, Palo Alto, London & Dublin100+ employeesSupport, consulting, trainingMgt: Google/DoubleClick, Oracle, Apple, NetApp, Mark LogicWell Funded: Sequoia, Union Square, Flybridge

What is BIG data ?

2000Google IncToday announced it has releasedthe largest search engine on theInternet.Google’s new index, comprisingmore than 1 billion URLs

2008Our indexing system for processinglinks indicates thatwe now count 1 trillion unique URLs(and the number of individual webpages out there is growing byseveral billion pages per day).

Data Growth 1,0001000 750 500 500 250 250 120 55 4 10 24 1 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 Millions of URLs

An unprecedentedamount of data isbeing created and isaccessible

What good is it ifwe can’t utilize thisdata?

?What isNoSQL

What is NoSQL?Key / Value Column Graph Document

Key-Value StoresA mapping from a key to a valueThe store doesnt know anything about the thekey or valueThe store doesnt know anything about theinsides of the valueOperations :•Set, get, or delete a key-value pair

Column-Oriented StoresLike a relational store, but ﬂipped around: alldata for a column is kept togetherAn index provides a means to get a columnvalue for a recordOperations: •Get, insert, delete records; updating ﬁeldsStreaming column data in and out of Hadoop

Graph DatabasesStores vertex-to-vertex edgesOperations: •Getting and setting edges •Sometimes possible to annotate vertices or edgesQuery languages support ﬁnding pathsbetween vertices, subject to variousconstraints

Document StoresThe store is a container for documentsDocuments are made up of named fields (think object/array/dict/hash...)Can query on any document field(s)Operations:•Insert and delete documents•Update fields within documents

MySQLData Model Columns Key:Value Columns Documents Relational Eventual / Eventual /Consistency Strong Strong Strong Quorum Quorum Multi- Multi- Single Single SingleAvailability Master Master Master Master Master Range orPartitioning Hash Hash Range N/A Hash Thrift, Native Rest, Native Query SQL CQL Drivers (6) Thrift Drivers (12)

Introduction toMongoDB

What do we want in an ideal world?

What do we want in an ideal world?•Horizontal scaling •cloud compatible •works with standard servers•Fast•Development is easy •Features •The Right Data Model •Schema Agility

MongoDB philosophy Keep functionality when we can (key/value stores are great, but we need more) Non-relational (no joins) makes scaling horizontally practical Document data models are good Database technology should run anywhere virtualized, cloud, metal, etc

Under the hoodWritten in C++Runs nearly everywhereData serialized to BSONExtensive use of memory-mapped ﬁlesi.e. read-through write-throughmemory caching.

Database LandscapeScalability & Performance Memcached MongoDB RDBMS Depth of Functionality

“MongoDB has the bestfeatures of key/valuestores, documentdatabases andrelational databasesin one. John Nunemaker

Relational made normalized data look like this Category • Name • Url Article User • Name Tag• Name • Slug • Name• Email Address • Publish date • Url • Text Comment • Comment • Date • Author

Document databases makenormalized data look like this Article • Name • Slug • Publish date User • Text • Name • Author • Email Address Comment[] • Comment • Date • Author Tag[] • Value Category[] • Value

MongoD B

Start with an (or array, hash, dict, eplace1 = { name : "10gen HQ", address : "578 Broadway 7th Floor", city : "New York", zip : "10011", tags : [ "business", "awesome" ]}

Inserting the record Initial Data Load > db.places.insert(place1)> db.places.insert(place1)

Querying{ name : "10gen HQ", address : "134 5th Avenue 3rd Floor", city : "New York", zip : "10011", tags : [ "business", "awesome" ]}> db.places.ﬁndOne({ zip: "10011", tags: "awesome" })> db.places.ﬁnd({tags: "business" })

Nested Documents{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), name : "10gen HQ", address : "578 Broadway 7th Floor", city : "New York", zip : "10011", tags : [ "business", "awesome" ], tips : [{ author : "Fred", date : "Sat Apr 25 2010 20:51:03", text : "Best Place Ever!" }]}

Updating> db.places.update( {name : "10gen HQ"}, { $push : { tips : { author : "nosh", date : 6/26/2011, text : "Office hours are great!" } } })

MongoDBUse Cases

CMS / BlogNeeds:• Business needed modern data store for rapid development and scaleSolution:• Use PHP & MongoDBResults:• Real time statistics• All data, images, etc stored together easy access, easy deployment, easy high availability• No need for complex migrations• Enabled very rapid development and growth

Photo Meta-DataProblem:• Business needed more ﬂexibility than Oracle could deliverSolution:• Use MongoDB instead of OracleResults:• Developed application in one sprint cycle• 500% cost reduction compared to Oracle• 900% performance improvement compared to Oracle

Customer AnalyticsProblem:• Deal with massive data volume across all customer sitesSolution:• Use MongoDB to replace Google Analytics / Omniture optionsResults:• Less than one week to build prototype and prove business case• Rapid deployment of new features

ArchivingWhy MongoDB:• Existing application built on MySQL• Lots of friction with RDBMS based archive storage• Needed more scalable archive storage backendSolution:• Keep MySQL for active data (100mil)• MongoDB for archive (2+ billion)Results:• No more alter table statements taking over 2 months to run• Sharding enabled horizontal scale• Very happily looking at other places to use MongoDB

Online DictionaryProblem:• MySQL could not scale to handle their 5B+ documentsSolution:• Switched from MySQL to MongoDBResults:• Massive simpliﬁcation of code base• Eliminated need for external caching system• 20x performance improvement over MySQL

E-commerceProblem:• Multi-vertical E-commerce impossible to model (efﬁciently) in RDBMSSolution:• Switched from MySQL to MongoDBResults:• Massive simpliﬁcation of code base• Rapidly build, halving time to market (and cost)• Eliminated need for external caching system• 50x+ performance improvement over MySQL

Tons more MongoDB casts a wide net people keep coming up with new and brilliant ways to use it

In Good Company and 1000s more

The Futureof BIGdata

What is BIG? BIG today isnormal tomorrow

Data Growth 9,00090006750 4,4004500 2,1502250 1,000 500 55 120 250 1 4 10 24 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Millions of URLs

2012Generating over250 Millions oftweets per day

MongoDB enablesus to scale withthe redeﬁnitionof BIG.

MongoDB High EasyPerformance Development { author : “steve”, date : new Date(), text : “About MongoDB...”, tags : [“tech”, “database”]} Horizontally Scalable

http://spf13.com http://github.com/s @spf13Question download at mongodb.orgWe’re hiring!! Contact us at jobs@10gen.com

http://www.scoop.it	1442
http://spf13.com	596
http://architects.dzone.com	299
http://css.dzone.com	158
http://itknowledgeexchange.techtarget.com	114
http://feeds.feedburner.com	37
https://si0.twimg.com	5
http://localhost	4
https://twimg0-a.akamaihd.net	2
http://dzone.com	1

NoSQL databases and managing big data

by Steve Francia

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 2,658

Statistics

NoSQL databases and managing big data Presentation Transcript