NewSQL Database Overview

3,572 views

Published on

1 Comment
5 Likes
Statistics
Notes
  • This is a great Thanks for the mention! Clustrix is also available as a service: http://www.clustrix.com/clustrix-database-as-a-service/ and recently posted some more info about the architecture they use: http://docs.clustrix.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
3,572
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
233
Comments
1
Likes
5
Embeds 0
No embeds

No notes for slide

NewSQL Database Overview

  1. 1. NewSQL Database Overview 민형기 (S-Core) hg.min@samsung.com 2013. 2. 22.
  2. 2. ContentsI. Why NewSQL?II. NewSQL 기본 개념III. NewSQL 종류IV.NewSQL 정리 1
  3. 3. Why NewSQL? 2
  4. 4. Thinking – Extreme DataQcon London 2012 3
  5. 5. Thinking - Traffic Explosion출처 : Netflix in the Cloud (http://www.slideshare.net/adrianco/netflix-in-the-cloud-2011) 4
  6. 6. Organizations need deeper insightsQcon London 2012 5
  7. 7. Solutions□Buy High end Technology□Higher more developers□Using NoSQL□Using NewSQL 6
  8. 8. Solution – Buy High End TechnologyOracle, IBM 7
  9. 9. Solution – Higher more developers □Application Level Sharding □Build your replication middleware □…http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_4_series/madone_4_5 8
  10. 10. Solutions – Use NoSQL □새로운 비 관계형 데이터 베이스 □분산 아키텍처 □수평 확장성 □고정된 테이블 스키마가 없음 □Join, UPDATE, DELETE 연산이 없음 □트랜잭션이 없음 □SQL 지원이 없음 9
  11. 11. NoSQL Ecosystems451 group 10
  12. 12. MongoDB □Document-oriented database  JSON-style documents: Lists, Maps, primitives  Schema-less □Transaction = update of a single document □Rich query language for dynamic queries □Tunable writes: speed reliability □Highly scalable and available 11
  13. 13. MongoDB 사용예□Use cases  High volume writes  Complex data  Semi-structured data□주요 고객  Foursquare  Bit.ly Intuit  SourceForge, NY Times  GILT Groupe, Evite,  SugarCRM 12
  14. 14. Apache Cassandra □Column-oriented database/Extensible row store  Think Row ~= java.util.SortedMap □Transaction = update of a row □Fast writes = append to a log □Tunable reads/writes: consistency / availability □Extremely scalable  Transparent and dynamic clustering  Rack and datacenter aware data replication □CQL = “SQL”-like DDL and DML 13
  15. 15. Apache Cassandra 사용 예 □사용 예  Big data  Multiple Data Center distributed database  Persistent cache  (Write intensive) Logging  High-availability (writes) □주요 고객  Digg, Facebook, Twitter, Reddit, Rackspace  Cloudkick, Cisco, SimpleGeo, Ooyala, OpenX  The largest production cluster has over 100 TB of data in over 150 machines.“ – Casssandra web site 14
  16. 16. Solutions – Use NewSQL □새로운 관계형 데이터베이스 □SQL과 ACID 트랜잭션을 유지 □새롭고 개선된 분산 아키텍처 □뛰어난 확장성과 성능을 지원 □NewSQL vendors: ScaleDB, NimbusDB, ..., VoltDB 15
  17. 17. http://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 16
  18. 18. NewSQL 정의 – Wikipedia NewSQL is a class of modern relational database management systems that seek to provide the same scalable performance of NoSQL systems for OLTP workloads while still maintaining the ACID guarantees of a traditional single-node database systemhttp://en.wikipedia.org/wiki/NewSQL 17
  19. 19. NewSQL 정의 – 451 Group A DBMS that delivers the scalability and flexibility promised by NoSQL while retaining the support for SQL queries and/or ACID, or to improve performance for appropriate workloads.http://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 18
  20. 20. NewSQL 정의 – Stonbraker  SQL as the primary interface.  ACID support for transactions  Non-locking concurrency control.  High per-node performance.  Parallel, shared-nothing architecture.http://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 19
  21. 21. NewSQL Category New Database New MySQL Storage Engines Transparent Clustering 20
  22. 22. The evolving database landscapeOSBC 21
  23. 23. MySQL Ecosystem 22
  24. 24. NewSQL Ecosystem 23
  25. 25. New Database □ Newly designed from scratch to achieve scalability and performance  One of the key considerations in improving the performance is making non-disk (memory) or new kinds of disks (flash/SSD) the primary data store.  some (hopefully minor) changes to the code will be required and data migration is still needed. □Solutions  Software-Only: VoltDB, NuoDB, Drizzle, Google Spanner  Supported as an appliance: Clustrix, Translattice.http://www.linuxforu.com/2012/01/newsql-handle-big-data/ 24
  26. 26. New MySQL Storage Engines □Highly optimized storage engines for MySQL □Scale better than built-in engines, such as InnoDB.  Good part: the usage of the MySQL interface  Downside part: data migration from other databases □Solutions  TokuDB, MemSQL, Xeround, Akiban, NDBhttp://www.linuxforu.com/2012/01/newsql-handle-big-data/ 25
  27. 27. Transparent Clustering □Retain the OLTP databases in their original format, but provide a pluggable feature  Cluster transparently  Ensure Scalability □Avoid the rewrite code or perform any data migration □Solutions  Cluster transparently: Schooner MySQL, Continuent Tungsten, ScalArc  Ensure Scalability: ScaleBase, dbShardshttp://www.linuxforu.com/2012/01/newsql-handle-big-data/ 26
  28. 28. NewSQL Products VoltDB Google Spanner 27
  29. 29. VoltDB □ VoltDB, 2010, GPL/VoltDB Proprietary License, Java/C++ □ Type: NewSQL, New Database □ Main Point: In-memory Database, Java Stored Procedure, VoltDB implements the design of the academic H-Store project □ Protocol: SQL □ Transaction: Yes □ Data Storage: Memory □ Features □ in-memory relational database □ Durability thru replication, snapshots, logging □ Transparent partitioning □ ACID-level consistency □ Synchronous multi-master replication □ Database Replicationhttp://voltdb.com/products-services/products, http://www.slideshare.net/chris.e.richardson/polygot-persistenceforjavadevs-jfokus2012reorgpptx 28
  30. 30. VoltDB- Technical Overview  “OLTP Through the Looking Glass” http://cs-www.cs.yale.edu/homes/dna/papers/oltpperf-sigmod08.pdf  VoltDB avoids the overhead of traditional databases K-safety for fault tolerance • no logging In memory operation for maximum throughput • no buffer management Partitions operate autonomously X X and single-threaded • no latching or locking X  Built to horizontally scale X 29 29
  31. 31. VoltDB - Partitions (1/3)  1 partition per physical CPU core – Each physical server has multiple VoltDB partitions  Data - Two types of tables – Partitioned Single column serves as partitioning key Rows are spread across all VoltDB partitions by partition column X X Transactional data (high frequency of modification) – Replicated All rows exist within all VoltDB partitions Relatively static data (low frequency of modification)  Code - Two types of work – both ACID – Single-Partition X All insert/update/delete operations within single partition X X Majority of transactional workload – Multi-Partition CRUD against partitioned tables across multiple partitions Insert/update/delete on replicated tables 30
  32. 32. VoltDB - Partitions (2/3)  Single-partition vs. Multi-partition select count(*) from orders where customer_id = 5 single-partition select count(*) from orders where product_id = 3 multi-partition insert into orders (customer_id, order_id, product_id) values (3,303,2) single-partition update products set product_name = ‘spork’ where product_id = 3 multi-partition Partition 1 Partition 2 Partition 3 1 101 2 2 201 1 3 201 1 table orders : customer_id (partition key) 1 101 3 5 501 3 6 601 1 (partitioned) order_id 4 401 2 5 502 2 6 601 2 product_id 1 knife 1 knife 1 knife table products : product_id 2 spoon 2 spoon 2 spoon (replicated) product_name 3 fork 3 fork 3 fork 31
  33. 33. VoltDB - Partitions (3/3)  Looking inside a VoltDB partition… – Each partition contains data and an execution engine. – The execution engine contains a queue for transaction requests. Work – Requests are executed sequentially (single threaded). Queue execution engine Table Data Index Data - Complete copy of all replicated tables - Portion of rows (about 1/partitions) of all partitioned tables 32
  34. 34. VoltDB - Compiling Schema Stored Procedures  The database is constructed from CREATE TABLE HELLOWORLD ( import org.voltdb. * ; import org.voltdb. * ; HELLO CHAR(15), @ProcInfo( org.voltdb. * ; import @ProcInfo( WORLD CHAR(15), partitionInfo = "HELLOWORLD.DIA – The schema (DDL) DIALECT CHAR(15), partitionInfo true "HE singlePartition = = @ProcInfo( partitionInfo = "HELLOWORLD.DIA )singlePartition = t PRIMARY KEY (DIALECT) singlePartition = true ); ) public class Insert extends VoltPr – The work load (Java stored procedures) public final SQLStmt public final SQLStmt sql = public class Insert extends VoltPr new SQLStmt("INSERT INTO HELLO public VoltTable[] sql = public final SQLStmt run new SQLStmt("INSERT INTO HELLO public VoltTable[] run( String hel – The Project (users, groups, partitioning) public VoltTable[] run( String hel  VoltCompiler creates application catalog Project.xml – Copy to servers along with 1 .jar and <?xml version="1.0"?> <project> 1 .so <database name=data <schema path=ddl. <partition table=‘ – Start servers </database> </project> 33
  35. 35. VoltDB - Transactions  All access to VoltDB is via Java stored procedures (Java + SQL)  A single invocation of a stored procedure is a transaction (committed on success) SQL  Limits round trips between DBMS and application  High performance client applications communicate asynchronously with VoltDB 34
  36. 36. VoltDB - Clusters/Durability  Scalability – Increase RAM in servers to add capacity – Add servers to increase performance / capacity – Consistently measuring 90% of single-node performance increase per additional node  High availability – K-safety for redundancy  Snapshots – Scheduled, continuous, on demand  Spooling to data warehouse  Disaster Recovery/WAN replication (Future) – Asynchronous replication 35
  37. 37. Google Spanner □ Google, 2012, Paper, C++ □ Type: NewSQL, New Database □ Main Point: Googles scalable, multi-version, globally-distributed, and synchronously-replicated database □ Distributed multiversion database  General-purpose transactions (ACID)  SQL query language  Schematized tables  Semi-relational data model □ Running in production  Storage for Google’s ad data  Replaced a sharded MySQL databasehttp://research.google.com/archive/spanner.html 36
  38. 38. Google Spanner Overview □Feature: Lock-free distributed read transactions □Property: External consistency of distributed transactions □First system at global scale □Implementation: Integration of concurrency control, replication, and 2PC □Correctness and performance □Enabling technology: TrueTime □Interval-based global timehttp://research.google.com/archive/spanner.html 37
  39. 39. Design Goals for Spannerhttp://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf 38
  40. 40. MySQL Cluster – NDB Architecturehttp://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-overview.html 39
  41. 41. Schooner MySQL Active Clusterhttp://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-overview.html 40
  42. 42. dbShards Architecturehttp://www.linuxforu.com/2012/01/newsql-handle-big-data/ 41
  43. 43. NewSQL 정리 42
  44. 44. Database 업계의 3가지 Trends□ NoSQL 데이터베이스:  분산 아키텍처의 확장성 등의 요구 사항을 충족하며, 스키마 없는 데이터 관리 요구 사항에 부합하도록 설계됨.□ NewSQL 데이터베이스:  분산 아키텍처의 확장성 등의 요구 사항을 충족하거나 혹은 수평 확장을 필요로하지 않지만 성능을 개선은 되도록 설계됨.□ Data Grid/Cache 제품:  응용 프로그램 및 데이터베이스 성능을 높이기 위해 메모리에 데이터를 저장하도록 설계됨. 43
  45. 45. 결론□ 데이터 저장을 위한 많은 솔루션이 존재 □ Oracle, MySQL만 있다는 생각은 버려야 함 □ 먼저 시스템의 데이터 속성과 요구사항을 파악(CAP, ACID/BASE) □ 한 시스템에 여러 솔루션을 적용  소규모/복잡한 관계 데이터: RDBMS  대규모 실시간 처리 데이터: NoSQL, NewSQL  대규모 저장용 데이터: Hadoop 등□ 적절한 솔루션 선택 □ 반드시 운영 중 발생할 수 있는 이슈에 대해 검증 후 도입 필요 □ 대부분의 NewSQL 솔루션은 베타 상태(섣부른 선택은 독이 될 수 있음) □ 솔루션의 프로그램 코드 수준으로 검증 필요□ NewSQL 솔루션에 대한 안정성 확보 □ 솔루션 자체의 안정성은 검증이 필요하며 현재의 DBMS 수준의 안정성은 지원하 지 않음 □ 반드시 안정적인 데이터 저장 방안 확보 후 적용 필요 □ 운영 및 개발 경험을 가진 개발자 확보 어려움 □ 요구사항에 부합되는 NewSQL 선정 필요□ 처음부터 중요 시스템에 적용하기 보다는 시범 적용 필요 □ 선정된 솔루션 검증, 기술력 내재화 44
  46. 46. 감사합니다. 45
  47. 47. Appendix. 46
  48. 48. Early – 2000s □All the big players were heavyweight and expensive.  Oracle, DB2, Sybase, SQL Server, etc. □Open-source databases were missing important features.  Postgres, mSQL, and MySQL.http://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 47
  49. 49. Early – 2000s : eBay Architecturehttp://highscalability.com/ebay-architecture 48
  50. 50. Early – 2000s : eBay Architecture  Push functionality to application:  Joins  Referential integrity  Sorting done  No distributed transactionshttp://highscalability.com/ebay-architecture 49
  51. 51. Mid– 2000s □MySQL + InnoDB is widely adopted by new web companies:  Supported transactions, replication, recovery.  Still must use custom middleware to scale out across multiple machines.  Memcache for caching queries.http://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 50
  52. 52. Mid – 2000s : Facebook Architecturehttp://www.techthebest.com/2011/11/29/technology-used-in-facebook/ 51
  53. 53. Mid – 2000s : Facebook Architecture  Scale out using custom middleware.  Store ~75% of database in Memcache.  No distributed transactions.http://www.techthebest.com/2011/11/29/technology-used-in-facebook/ 52
  54. 54. Late – 2000s □MySQL + InnoDB is widely adopted by new web companies:  Supported transactions, replication, recovery.  Still must use custom middleware to scale out across multiple machines.  Memcache for caching queries.http://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 53
  55. 55. Late – 2000s : MongoDB Architecturehttp://sett.ociweb.com/sett/settAug2011.html 54
  56. 56. Late – 2000s : MongoDB Architecture  Easy to use.  Becoming more like a DBMS over time.  No transactions.http://sett.ociweb.com/sett/settAug2011.html 55
  57. 57. Early – 2010s □New DBMSs that can scale across multiple machines natively and provide ACID guarantees.  MySQL Middleware  Brand New Architectureshttp://www.cs.brown.edu/courses/cs227/slides/newsql/newsql-intro.pdf 56
  58. 58. Database SPRAIN 57
  59. 59. Database SPRAIN □“An injury to ligaments... caused by being stretched beyond normal capacity” □Six key drivers for NoSQL/NewSQL/DDG adoption  Scalability  Performance  Relaxed consistency  Agility  Intricacy  Necessity 58
  60. 60. Database SPRAIN - Scalability □Associated sub-driver: Hardware economics  Scale-out across clusters of commodity servers □Example project/service/vendor  BigTable HBase Riak MongoDB Couchbase, Hadoop  Amazon RDS, Xeround, SQL Azure, NimbusDB  Data grid/cache □Associated use case:  Large-scale distributed data storage  Analysis of continuously updated data  Multi-tenant PaaS data layer 59
  61. 61. Database SPRAIN - Scalability □User: StumbleUpon □Problem:  Scaling problems with recommendation engine on MySQL □Solution: HBase  Started using Apache HBase to provide real-time analytics on Su.pr  MySQL lacked the performance headroom and scale  Multiple benefits including avoiding declaring schema  Enables the data to be used for multiple applications and use cases 60
  62. 62. Database SPRAIN - Performance □Associated sub-driver: MySQL limitations  Inability to perform consistently at scale □Example project/service/vendor  Hypertable Couchbase Membrain MongoDB Redis  Data grid/cache  VoltDB, Clustrix □Associated use case:  Real time data processing of mixed read/write workloads  Data caching  Large-scale data ingestion 61
  63. 63. Database SPRAIN - Performance □User: AOL Advertising □Problem:  Real-time data processing to support targeted advertising □Solution: Membase Server  Segmentation analysis runs in CDH, results passed into Membase  Make use of its sub-millisecond data delivery  More time for analysis as part of a 40ms targeted and response time  Also real time log and event management 62
  64. 64. Database SPRAIN – Relaxed Consistency □Associated sub-driver: CAP theorem  The need to relax consistency in order to maintain availability □Example project/service/vendor:  Dynamo, Voldemort, Cassandra  Amazon SimpleDB □Associated use case:  Multi-data center replication  Service availability  Non-transactional data off-load 63
  65. 65. Database SPRAIN – Relaxed Consistency □User: Wordnik □Problem:  MySQL too consistent –blocked access to data during inserts and created numerous temp files to stay consistent. □Solution: MongoDB  Single word definition contains multiple data items from various sources  MongoDB stores data as a complete document  Reduced the complexity of data storage 64
  66. 66. Database SPRAIN – Agility □ Associated sub-driver: Polyglot persistence  Choose most appropriate storage technology for app in development □Example project/service/vendor  MongoDB, CouchDB, Cassandra  Google App Engine, SimpleDB, SQL Azure □Associated use case:  Mobile/remote device synchronization  Agile development  Data caching 65
  67. 67. Database SPRAIN – Agility □ User: Dimagi BHOMA (Better Health Outcomes through Mentoring and Assessments) project □Problem:  Deliver patient information to clinics despite a lack of reliable Internet connections □Solution: Apache CouchDB  Replicates data from regional to national database  When Internet connection, and power, is available  Upload patient data from cell phones to local clinic 66
  68. 68. Database SPRAIN – Intricacy □ Associated sub-driver: Big data, total data  Rising data volume, variety and velocity □Example project/service/vendor  Neo4j GraphDB, InfiniteGraph  Apache Cassandra, Hadoop,  VoltDB, Clustrix □Associated use case:  Social networking applications  Geo-locational applications  Configuration management database 67
  69. 69. Database SPRAIN – Intricacy □ User: Evident Software □Problem:  Mapping infrastructure dependencies for application performance management □Solution: Neo4j  Apache Cassandra stores performance data  Neo4j used to map the correlations between different elements  Enables users to follow relationships between resources while investigating issues 68
  70. 70. Database SPRAIN – Necessity □ Associated sub-driver: Open source  The failure of existing suppliers to address the performance, scalability and flexibility requirements of large-scale data processing □ Example project/service/vendor  BigTable, Dynamo, MapReduce, Memcached  Hadoop HBase, Hypertable, Cassandra, Membase  Voldemort, Riak, BigCouch  MongoDB, Redis, CouchDB, Neo4J □Associated use case:  All of the above 69
  71. 71. Database SPRAIN – Necessity □BigTable: Google □Dynamo: Amazon □Cassandra: Facebook □HBase: Powerset □Voldemort: LinkedIn □Hypertable: Zvents □Neo4j: Windh Technologies  Yahoo: Apache Hadoop and Apache HBase  Digg: Apache Cassandra  Twitter: Apache Cassandra, Apache Hadoop and FlockDB 70

×