Data SLA in the public cloud


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Data SLA in the public cloud

  1. 1. Data SLA in the cloud<br />
  2. 2. About Us<br />ScaleBase is a new startup targeting the database-as-a-service market (DBaaS)<br />We offer unlimited database scalability and availability using our Database Load Balancer<br />We launch in September, 2010. Stay tuned at our site.<br />
  3. 3. Agenda<br />The requirements for data SLA in public cloud environments<br />Achieving data SLA with NOSQL<br />Achieving data SLA with relational databases<br />
  4. 4. The requirements for data SLA in public cloud environments<br />
  5. 5. What We Need<br />Availability<br />Consistency<br />Scalability<br />
  6. 6. Brewer's (CAP) Theorem<br />It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:<br />Consistency (all nodes see the same data at the same time)<br />Availability (node failures do not prevent survivors from continuing to operate)<br />Partition Tolerance (the system continues to operate despite arbitrary message loss)<br /><br />
  7. 7. What It Means<br /><br />
  8. 8. Dealing With CAP<br />Drop Partition Tolerance<br />Run everything on one machine.<br />This is, of course, not very scalable.<br />
  9. 9. Dealing With CAP<br />Drop Availability<br />If a partition fail, everything waits until the data is consistent again. <br />This can be very complex to handle over a large number of nodes.<br />
  10. 10. Dealing With CAP<br />Drop Consistency<br />Welcome to the “Eventually Consistent” term.<br />At the end – everything will work out just fine - And hi, sometimes this is a good enough solution<br />When no updates occur for a long period of time, eventually all updates will propagate through the system and all the nodes will be consistent<br />For a given accepted update and a given node, eventually either the update reaches the node or the node is removed from service<br />Known as BASE (Basically Available, Soft state, Eventual consistency), as opposed to ACID<br />
  11. 11. Reading More On CAP<br />This is an excellent read, and some of my samples are from this blog<br /><br />
  12. 12. Achieving data SLA with relational databases<br />
  13. 13. Databases And CAP<br />ACID – Consistency<br />Availability – tons of solutions, most of them not cloud oriented<br />Oracle RAC<br />MySQL Proxy<br />Etc.<br />Replication based solutions can solve at least read availability and scalability (see Azure SQL)<br />
  14. 14. Database Cloud Solutions<br />Amazon RDS<br />NaviSite Oracle RAC<br />Not that popular<br />Costs to cloud providers (complexity, not standard)<br />
  15. 15. So Where Is The Problem?<br />Partition Tolerance just doesn’t work<br />Scaling problems (usually write but also read)<br />BigData problems<br />
  16. 16. Scaling Up<br />Issues with scaling up when the dataset is just too big<br />RDBMS were not designed to be distributed<br />Began to look at multi-node database solutions<br />Known as ‘scaling out’ or ‘horizontal scaling’<br />Different approaches include:<br />Master-slave<br />Sharding<br />
  17. 17. Scaling RDBMS – Master/Slave<br />Master-Slave<br />All writes are written to the master. All reads performed against the replicated slave databases<br />Critical reads may be incorrect as writes may not have been propagated down<br />Large data sets can pose problems as master needs to duplicate data to slaves<br />
  18. 18. Scaling RDBMS - Sharding<br />Partition or sharding<br />Scales well for both reads and writes<br />Not transparent, application needs to be partition-aware<br />Can no longer have relationships/joins across partitions<br />Loss of referential integrity across shards<br />
  19. 19. Other ways to scale RDBMS<br />Multi-Master replication<br />INSERT only, not UPDATES/DELETES<br />No JOINs, thereby reducing query time<br />This involves de-normalizing data<br />In-memory databases<br />
  20. 20. Achieving data SLA with NOSQL<br />
  21. 21. NoSQL<br />A term used to designate databases which differ from classic relational databases in some way. These data stores may not require fixed table schemas, and usually avoid join operations and typically scale horizontally. Academics and papers typically refer to these databases as structured storage, a term which would include classic relational databases as a subset.<br /><br />
  22. 22. NoSQL Types<br />Key/Value<br />A big hash table<br />Examples: Voldemort, Amazon Dynamo<br />Big Table<br />Big table, column families<br />Examples: Hbase, Cassandra<br />Document based<br />Collections of collections<br />Examples: CouchDB, MongoDB<br />Graph databases<br />Based on graph theory<br />Examples: Neo4J<br />Each solves a different problem<br />
  23. 23. NO-SQL<br /><br />
  24. 24. Pros/Cons<br />Pros:<br />Performance<br />BigData<br />Most solutions are open source<br />Data is replicated to nodes and is therefore fault-tolerant (partitioning)<br />Don't require a schema<br />Can scale up and down<br />Cons:<br />Code change<br />No framework support<br />Not ACID<br />Eco system (BI, Backup)<br />There is always a database at the backend<br />Some API is just too simple<br />
  25. 25. Amazon S3 Code Sample<br />AWSAuthConnection conn = new AWSAuthConnection(awsAccessKeyId, awsSecretAccessKey, secure, server, format);<br />Response response = conn.createBucket(bucketName, location, null);<br />final String text = "this is a test";<br />response = conn.put(bucketName, key, new S3Object(text.getBytes(), null), null);<br />
  26. 26. Cassandra Code Sample<br />CassandraClient cl = pool.getClient() ;<br />KeySpaceks = cl.getKeySpace("Keyspace1") ;<br />// insert value<br />ColumnPathcp = new ColumnPath("Standard1" , null, "testInsertAndGetAndRemove".getBytes("utf-8")); <br />for(int i = 0 ; i < 100 ; i++){<br />ks.insert("testInsertAndGetAndRemove_"+i, cp , ("testInsertAndGetAndRemove_value_"+i).getBytes("utf-8"));<br />}<br />//get value<br />for(inti = 0 ; i < 100 ; i++){<br /> Column col = ks.getColumn("testInsertAndGetAndRemove_"+i, cp);<br /> String value = new String(col.getValue(),"utf-8") ;<br />}<br />//remove value<br />for(int i = 0 ; i < 100 ; i++){<br />ks.remove("testInsertAndGetAndRemove_"+i, cp);<br />}<br />
  27. 27. Cassandra Code Sample – Cont’<br />try{<br />ks.remove("testInsertAndGetAndRemove_not_exist", cp);<br />}catch(Exception e){<br /> fail("remove not exist row should not throw exceptions");<br />}<br />//get already removed value<br />for(int i = 0 ; i < 100 ; i++){<br />try{<br /> Column col = ks.getColumn("testInsertAndGetAndRemove_"+i, cp);<br /> fail("the value should already being deleted");<br />}catch(NotFoundException e){<br />}catch(Exception e){<br /> fail("throw out other exception, should be NotFoundException." + e.toString() );<br /> }<br />}<br />pool.releaseClient(cl) ;<br />pool.close() ;<br />
  28. 28. Cassandra Statistics<br />Facebook Search<br />MySQL > 50 GB Data<br />Writes Average : ~300 ms<br />Reads Average : ~350 ms<br />Rewritten with Cassandra > 50 GB Data<br />Writes Average : 0.12 ms<br />Reads Average : 15 ms<br />
  29. 29. MongoDB<br />Mongo m = new Mongo();<br />DB db = m.getDB( "mydb" );<br />Set<String> colls = db.getCollectionNames();<br />for (String s : colls) {<br />System.out.println(s);<br />}<br />
  30. 30. MongoDB – Cont’<br />BasicDBObjectdoc = new BasicDBObject();<br />doc.put("name", "MongoDB");<br />doc.put("type", "database");<br />doc.put("count", 1);<br />BasicDBObject info = new BasicDBObject();<br />info.put("x", 203);<br />info.put("y", 102);<br />doc.put("info", info);<br />coll.insert(doc);<br />
  31. 31. Neo4J<br />GraphDatabaseServicegraphDb = new EmbeddedGraphDatabase("var/base");<br />Transaction tx = graphDb.beginTx();<br />try {<br /> Node firstNode = graphDb.createNode();<br /> Node secondNode = graphDb.createNode();<br /> Relationship relationship = firstNode.createRelationshipTo(secondNode, MyRelationshipTypes.KNOWS);<br />firstNode.setProperty("message", "Hello, ");<br />secondNode.setProperty("message", "world!");<br />relationship.setProperty("message", "brave Neo4j ");<br />tx.success();<br />System.out.print(firstNode.getProperty("message"));<br />System.out.print(relationship.getProperty("message"));<br />System.out.print(secondNode.getProperty("message"));<br />}<br />finally {<br />tx.finish();<br />graphDb.shutdown();<br />}<br />
  32. 32. The Bottom Line<br />
  33. 33. Data SLA<br />There is no golden hammer<br />Choose your tool wisely, based on what you need<br />Usually<br />Start with RDBMS (shortest TTM, which is what we really care about)<br />When scale issues occur – start moving to NoSQL based on your needs<br />You can get Data SLA in the cloud – just think before you code!!!<br />