No sql (for comverse)
Upcoming SlideShare
Loading in...5

No sql (for comverse)



This is the presentation I gave today (14/11/2012) for comverse

This is the presentation I gave today (14/11/2012) for comverse



Total Views
Views on SlideShare
Embed Views



1 Embed 1 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

No sql (for comverse) No sql (for comverse) Presentation Transcript

  • NoSQL (big data) Samuel Dratwa Samuel.dratwa@gmail.comCopyright © 2011 LOGTEL
  • Agenda  How big is BIG ?  The motivation behind NoSQL  CAP theorem  Partitions / fragmentation  The different NoSQL models  Key Value  Column-Based  Document store  Big Table  Map Reduce  Graph  The NoSQL way of thinking (using graphs)Copyright © 2011 LOGTEL
  • NoSQL humor © 2011 LOGTEL
  • Out with the old….Copyright © 2011 LOGTEL
  • How big is BIG ? 10GB ? 10TB ? 10 PB ?Copyright © 2011 LOGTEL
  • NoSQL motivationCopyright © 2011 LOGTEL
  • Parallel Databases and Data Stores  Relational Databases – mainstay of business  Web-based applications caused spikes  Especially true for public-facing e-Commerce sites  Many application servers, one database  Easy to parallelize application servers to 1000s of servers, harder to parallelize databases to same scale  First solution: memcache (in-memory) or other caching mechanisms to reduce database accessCopyright © 2011 LOGTEL
  • Scaling Up  What if the dataset is huge, and very high number of transactions per second  Use multiple servers to host database  „scaling out‟ or „horizontal scaling‟  Parallel databases have been around for a while  But expensive, and designed for decision support not OLTP (Online Transaction Processing)Copyright © 2011 LOGTEL
  • Scaling RDBMS – Master/Slave  Master-Slave  All writes are written to the master. All reads performed against the replicated slave databases  Good for mostly read, very few update applications  Critical reads may be incorrect as writes may not have been propagated down  Large data sets can pose problems as master needs to duplicate data to slavesCopyright © 2011 LOGTEL
  • Scaling RDBMS - Partitioning  Partitioning  Divide the database across many machines  E.g. hash or range partitioning  Handled transparently by parallel databases  but they are expensive  “Sharding”  Divide data amongst many cheap databases (MySQL/PostgreSQL)  Manage parallel access in the application  Scales well for both reads and writes  Not transparent, application needs to be partition-awareCopyright © 2011 LOGTEL
  • What is NoSQL?  Stands for Not Only SQL  Class of non-relational data storage systems  E.g. BigTable, Dynamo, PNUTS/Sherpa, ..  Usually do not require a fixed table schema nor do they use the concept of joins  All NoSQL offerings relax one or more of the ACID properties (will talk about the CAP theorem)  Not a backlash/rebellion against RDBMS  SQL is a rich query language that cannot be rivaled by the current list of NoSQL offeringsCopyright © 2011 LOGTEL
  • Why Now?  Explosion of social media sites (Facebook, Twitter) with large data needs  Explosion of storage needs in large web sites such as Google, Yahoo  Much of the data is not files  Rise of cloud-based solutions such as Amazon S3 (simple storage solution)  Shift to dynamically-typed data with frequent schema changes  Open-source communityCopyright © 2011 LOGTEL
  • NoSQL Data Storage: Classification  NoSQL solutions fall into 4 major areas:  Uninterpreted key/value or „the big hash table‟.  Amazon S3 (Dynamo)  Voldemort  Scalaris  Column-based, with interpreted keys  Cassandra, BigTable, HBase, Sherpa/PNuts  Others  CouchDB (document-based)  Neo4J (graph-based)Copyright © 2011 LOGTEL
  • NoSQL ecosystemCopyright © 2011 LOGTEL
  • Typical NoSQL API  Basic API access:  get(key) -- Extract the value given a key  put(key, value) -- Create or update the value given its key  delete(key) -- Remove the key and its associated value  execute(key, operation, parameters) -- Invoke an operation to the value (given its key) which is a special data structure (e.g. List, Set, Map .... etc).Copyright © 2011 LOGTEL
  • ACID Atomic: Either the whole process of a transaction is done or none is. Consistency: Database constraints (application-specific) are preserved. Isolation: It appears to the user as if only one process executes at a time. (Two concurrent transactions will not see on another‟s transaction while “in flight”.) Durability: The updates made to the database in a committed transaction will be visible to future transactions. (Effects of a process do not get lost if the system crashes.)Copyright © 2011 LOGTEL
  • CAP Theorem  Three properties of a system  Consistency (all copies have same value)  Availability (system can run even if parts have failed)  Partitions (network can break into two or more parts, each with active systems that can‟t talk to other parts)  Brewer‟s CAP “Theorem”: You can have at most two of these three properties for any system  Very large systems will partition at some point  Choose one of consistency or availability  Traditional database choose consistency  Most Web applications choose availability  Except for specific parts such as order processingCopyright © 2011 LOGTEL
  • Availability  Traditionally, thought of as the server/process available five 9‟s (99.999 %).  However, for large node system, at almost any point in time there‟s a good chance that a node is either down or there is a network disruption among the nodes.  Want a system that is resilient in the face of network disruptionCopyright © 2011 LOGTEL
  • Eventual Consistency  When no updates occur for a long period of time, eventually all updates will propagate through the system and all the nodes will be consistent  For a given accepted update and a given node, eventually either the update reaches the node or the node is removed from service  Known as BASE (Basically Available, Soft state, Eventual consistency), as opposed to ACID  Soft state: copies of a data item may be inconsistent  Eventually Consistent – copies becomes consistent at some later time if there are no more updates to that data itemCopyright © 2011 LOGTEL
  • Common Advantages  Cheap, easy to implement (open source)  Data are replicated to multiple nodes (therefore identical and fault-tolerant) and can be partitioned  When data is written, the latest version is on at least one node and then replicated to other nodes  Down nodes easily replaced  No single point of failure  Easy to distribute  Dont require a schemaCopyright © 2011 LOGTEL
  • What am I giving up?  joins  group by  order by  ACID transactions  SQL as a sometimes frustrating but still powerful query language  easy integration with other applications that support SQLCopyright © 2011 LOGTEL
  • Distributed Key-Value Data Stores  Distributed key-value data storage systems allow key-value pairs to be stored (and retrieved on key) in a massively parallel system  E.g. Google BigTable, Yahoo! Sherpa/PNUTS, Amazon Dynamo, ..  Partitioning, high availability etc completely transparent to application  Sharding systems and key-value stores don‟t support many relational features  No join operations (except within partition)  No referential integrity constraints across partitions  etc.Copyright © 2011 LOGTEL
  • Flexible Data Model ColumnFamily: Rockets Key Value Name Value 1 name Rocket-Powered Roller Skates toon Ready, Set, Zoom inventoryQty 5 brakes false 2 Name Value name Little Giant Do-It-Yourself Rocket-Sled Kit toon Beep Prepared inventoryQty 4 brakes false Name Value 3 name Acme Jet Propelled Unicycle toon Hot Rod and Reel inventoryQty 1 wheels 1Copyright © 2011 LOGTEL
  • Data Model Tables are sorted by Row Table schema only define it‟s column families .  Each family consists of any number of columns  Each column consists of any number of versions  Columns only exist when inserted, NULLs are free.  Columns within a family are sorted and stored together Everything except table names are byte[] (Row, Family: Column, Timestamp)  Value Column Family Row keyCopyright © 2011 LOGTEL TimeStamp value
  • PNUTS Data Storage ArchitectureCopyright © 2011 LOGTEL
  • Read Client Query Result Cassandra Cluster Closest replica Result Read repair if digests differ Replica A Digest Query Digest Response Digest Response Replica B Replica CCopyright © 2011 LOGTEL
  • Partitioning And Replication 1 0 h(key1) E A N=3 C h(key2) F B D 1/2Copyright © 2011 LOGTEL 31
  • Should I be using NoSQL Databases?  For almost all of us, regular relational databases are THE correct solution  NoSQL Data storage systems makes sense for applications that need to deal with very very large semi-structured data  Log Analysis  Social Networking FeedsCopyright © 2011 LOGTEL
  • Graph in practice (thanks to Luca Garulli)Copyright © 2011 LOGTEL
  • Lets take a real example - CRM to think «graphically» with one of the most common domains in the enterprise world: The old-classic CRM* domain * today in 99% of the cases a RDBMS is usedCopyright © 2011 LOGTEL
  • Every developer knowsthe Relational Model(?), but who knows the Graph one?Copyright © 2011 LOGTEL
  • Graph Theory crash course Back to school:Copyright © 2011 LOGTEL
  • Basic Graph Sam Likes NoSQL lectureCopyright © 2011 LOGTEL
  • Property Graph Model* Vertices are directed NoSQL Sam Lecture Likes name: Samuel surname: Dratwa since: 2012 company: SADOT editions: [Comverse, Tel-Aviv] Vertices and Edges can have properties * © 2011 LOGTEL
  • Edges - Arcs NoSQL Sam lecture An Edge connects 2 vertices: use multiple vertices to represents 1-N and N-M relationshipsCopyright © 2011 LOGTEL
  • Doron Sam Likes Joins FriendOf NoSQL lecture AvitalCopyright © 2011 LOGTEL
  • Compliments, this is your diploma in «Graph Theory»Copyright © 2011 LOGTEL
  • Domain: minimal CRM Customer AddressRegistry systemOrder system Order StockCopyright © 2011 LOGTEL
  • Customer Address How does Relational DBMSRegistry system manage relationships?Order system Order StockCopyright © 2011 LOGTEL
  • Relational World: 1-1 Relationships Customer Address Id Name Address Id Location 10 Samuel 34 34 Rome, London 11 Katja 44 44 Cologne 34 Sylvia 54 54 Rome 56 Mark 66 66 New Mexico 88 Steve 68 68 Palo Alto JOIN Customer.Address -> Address.IdCopyright © 2011 LOGTEL
  • Relational World: 1-N Relationships Customer Address Id Name Id Customer Location 10 Samuel 24 10 Rome 11 Katja 33 10 London 34 Sylvia 44 34 Rome 56 Mark 66 11 Cologne 88 Steve 68 88 Palo Alto Inverse JOIN Address.Customer -> Customer.IdCopyright © 2011 LOGTEL
  • Relational World: N-M Relationships Customer CustomerAddr Address Id Name ess Id Location 10 Samuel Id Address 24 Rome 11 Katja 10 24 33 London 34 Sylvia 10 33 44 Rome 56 Mark 34 24 66 Cologne 88 Steve 68 Palo Alto Additional table with 2 JOINs (1) CustomerAddress.Id -> Customer.Id and (2) CustomerAddress.Address -> Address.IdCopyright © 2011 LOGTEL
  • What‟s wrong with the Relational Model?Copyright © 2011 LOGTEL
  • The JOIN is the evil! Customer CustomerAddr Address Id Name ess Id Location 10 Samuel Id Address 24 Rome 11 Katja 10 24 33 London 34 Sylvia 10 33 44 Rome 56 Mark 34 24 66 Cologne 88 Steve 68 Palo Alto These are all JOINs executed everytime you traverse a relationship! relationshipCopyright © 2011 LOGTEL
  • Why not JOIN • A JOIN means searching for a key in another table • The first rule to improve performance is indexing all the keys • Index speeds up searches but slows down insert, updates and deletes • So in the best case a JOIN is a lookup into in an index • This is done per single join! • If you traverse hundreds of relationships you‟re executing hundreds of JOINsCopyright © 2011 LOGTEL
  • Index Lookup it is really that fast?Copyright © 2011 LOGTEL
  • Index Lookup: how does it works? Think to an Address Book A-Z where we have to find A-L M-Z Samuel‟s phone number A-L M-Z A-D E-L M-R S-Z A-D E-L A-B C-D E-G H-L E-G H-L Index algorithms are all E-F G H-J K-L similar and based on balanced treesCopyright © 2011 LOGTEL
  • A-Z A-L M-Z A-L M-Z Found! A-D E-L M-R S-Z Each lookup takes A-D E-L X steps, where X A-B C-D E-G H-L grows with the E-G H-L index size! E-F G H-J K-LCopyright © 2011 LOGTEL
  • An index lookup is executed for each JOIN Querying more tables can easilyproduce millions of JOINs/Lookups! Here the rule: more entries = more lookup steps = slower JOINCopyright © 2011 LOGTEL
  • Is there a better way to manage relationships?Copyright © 2011 LOGTEL
  • How does GraphDB manage index-free relationships?Copyright © 2011 LOGTEL
  • an Open Source (Apache 2) document-graph NoSQL dbmssupports: transactions, extended-SQL, Multi-Master replication, etcCopyright © 2011 LOGTEL
  • OrientDB: traverse a relationship The Record ID (RID) is a Physical position RID = RID = #13:35 #13:100 RID = #14:54 Lives Sam Rome out: [#13:35] in: [#13:100] Label : ‘Lives’ out : [#14:54] in: [#14:54] label : ‘Customer’ label = ‘Address’ name : ‘Sam’ name = ‘Rome’Copyright © 2011 LOGTEL
  • GraphDB handles relationships as a physical LINK to the record assigned when the edge is created on the other side RDBMS computes the relationship every time you query a database Is not that crazy?!Copyright © 2011 LOGTEL
  • This means jumping from a O(log N) algorithm to a near O(1) traversing cost is not more affected by database size! This is huge in the BigData ageCopyright © 2011 LOGTEL
  • Create the graph in SQL $luca> cd bin $luca> ./ OrientDB console v.1.2.0-SNAPSHOT ( Type help to display all the commands supported. orientdb> create vertex V set name = „Sam‟, label = „Customer‟ Created vertex #13:35 in 0.03 secs orientdb> create vertex V set name = „Rome‟, label = „Address‟ Created vertex #13:100 in 0.02 secs orientdb> create edge E from #13:35 to #13:100 set label = „Lives‟ Created edge #14:54 in 0.02 secsCopyright © 2011 LOGTEL
  • Create the graph in Java OGraphDatabase graph = new OGraphDatabase("local:/tmp/db/graph”); ODocument sam= graph.createVertex(); sam.field(“name", “Sam"); sam.field(“label", “Customer"); ODocument rome = graph.createVertex(); rome.field(“name", “Rome”); rome.field(“label", “Address”); ODocument edge = graph.createEdge(sam, rome).field(“label”, “Lives”);; graph.close();Copyright © 2011 LOGTEL
  • Query the graph in SQLorientdb> select in[label=„Lives‟].out from V wherelabel = „Address‟ and name = „Rome‟---+--------+--------------------+--------------------+--------------------+ #| REC ID |label |out |in |---+--------+--------------------+--------------------+--------------------+ 0| 13:35|Sam |[#14:54] | |---+--------+--------------------+--------------------+--------------------+1 item(s) found. Query executed in 0.007 sec(s).orientdb> select * from V where label = „Address‟ AND in[label=„Lives‟].size() > 0---+--------+--------------------+--------------------+--------------------+ #| REC ID |label |out |in |---+--------+--------------------+--------------------+--------------------+ 0| 13:100| Rome | |[#14:54] |---+--------+--------------------+--------------------+--------------------+1 item(s) found. Query executed in 0.007 sec(s).Copyright © 2011 LOGTEL
  • Query the graph in Java OGraphDatabase graph = new OGraphDatabase("local:/tmp/db/graph”); // GET ALL THE THE CUSTOMER FROM ROME, ITALY List<ODocument> result = graph.command( new OCommandSQL ( “select in[label=„Lives‟].out from V where label = „Address‟ and name = ?”) ).execute( “Rome”); for( ODocument v : result ) { System.out.println(“Result: “ + v.field(“label”) ); } ----------------------------------------------------------------------------------- ----Result: SamCopyright © 2011 LOGTEL
  • Query vs. traversal  Once you‟ve a well connected database in the form of a Super Graph you can cross records instead of query them!  All you need is some root vertices where to start to traverseCopyright © 2011 LOGTEL
  • Query vs traversal Special Customers Stocks Customers Sam John Sylvia White This is a Soap root Order Order vertex 2332 8834Copyright © 2011 LOGTEL
  • Query the graph in SQL Customers Supposing that the root node #30:0 links all the #30:0 Customer vertices Get all the customers: orientdb> select from #30:0 Get all the customers who bought at least one „White Soap‟ product: orientdb> select * from ( select from #30:0) where[label=„Bought‟] = „White Soap‟Copyright © 2011 LOGTEL
  • Demo time!Copyright © 2011 LOGTEL
  • Should I be using NoSQL Databases?  For almost all of us, regular relational databases are THE correct solution  NoSQL Data storage systems makes sense for applications that need to deal with very very large semi-structured data  Log Analysis  Social Networking FeedsCopyright © 2011 LOGTEL