http://www.coordguru.com




MapReduce Debates and Schema-Free
 - Big Data, MapReduce, RDBMS+MapReduce, Non-Relational DB
...
http://www.coordguru.com




The Advent of Big Data
http://www.coordguru.com


Noah’s Ark Problem
• Did Noah take dinosaurs on the Ark?
    • The Ark was a very large ship de...
http://www.coordguru.com


Perspectives of Big Data

  •SAN                                               •SQL
  •HDFS    ...
http://www.coordguru.com


How to deal with “Big Data”
 Struggling to STORE and ANALYZE “Big Data”
http://www.coordguru.com


Case Study: User Credit Analysis
    A User Credit Model
                                      ...
http://www.coordguru.com


Case Study: User Credit Analysis
  Preprocessing Blog Data for Analyzing User Credit
          ...
http://www.coordguru.com


New Changes surrounding Data Storages
  • Volume
      • Data volumes have grown from tens of g...
http://www.coordguru.com




Hadoop Revolution
http://www.coordguru.com


Best Practice in Hadoop
• Software Stack in Google/Hadoop        • Cookbook for ‚Big Data‛




...
http://www.coordguru.com


Hadoop is changing the Game
• Hadoop, DW, and BI
http://www.coordguru.com


“Big Data” goes well with Hadoop
• Parallelize Relational Algebra Operations using MapReduce
http://www.coordguru.com


Case Study: Parallel Join
• A Parallel Join Example using MapReduce
http://www.coordguru.com


Case Study: Further Study in Parallel Join
  Problems
  • Need to sort
  • Move the partitioned...
http://www.coordguru.com


Case Study: Improvements in Parallel Join
  Map-Side Join
  • Replicate a relatively smaller in...
http://www.coordguru.com




MapReduce Debates
http://www.coordguru.com


MapReduce is just A Major Step Backwards!!!
                                                   ...
http://www.coordguru.com


MapReduce is just A Major Step Backwards!!! (cont’d)
                                          ...
http://www.coordguru.com


MapReduce is just A Major Step Backwards!!! (cont’d)
                                          ...
http://www.coordguru.com


MapReduce is just A Major Step Backwards!!! (cont’d)
                                          ...
http://www.coordguru.com




What the !@# MapReduce?
http://www.coordguru.com


RDB experts Jump the MR Shark
                                                                 ...
http://www.coordguru.com


DBs are hammers; MR is a screwdriver
                                                        Ma...
http://www.coordguru.com


MR is a Step Backwards, but some Steps Forward
                                                ...
http://www.coordguru.com
http://www.coordguru.com


Lessons Learned from the Debates
  Who Moved My Cheese?
http://www.coordguru.com




Hybrids of MapReduce and RDBMS
http://www.coordguru.com


Integrate MapReduce into RDBMS


                                                Vertica+Hadoop...
http://www.coordguru.com


HadoopDB Details
  HadoopDB Architecture




                          Connection parameters
  ...
http://www.coordguru.com


An Interesting Friendship of RDBMS and MapReduce
  RDBMS vs. MapReduce
                        ...
http://www.coordguru.com


 In-Database MapReduce vs. File-only MapReduce
      • In-Database MapReduce
           • Green...
http://www.coordguru.com




Why Non-Relational?
http://www.coordguru.com


Challenges in Traditional RDBMS
  • Volume
     • Data volumes have grown from tens of gigabyte...
http://www.coordguru.com


Challenges in Traditional RDBMS (cont’d)
  • Scale Out
      • Is it possible to achieve a larg...
http://www.coordguru.com


The New Faces of Data
  • Scale out
      • CAP Theorem
           •    CAP theorem simply stat...
http://www.coordguru.com


The New Faces of Data (cont’d)
  • Sparsity
      • Some data have sparse attributes
          ...
http://www.coordguru.com




Non-Relational Databases
http://www.coordguru.com


Trends of Emergent Data Stores

                                                               ...
http://www.coordguru.com


Emergent Data Stores in CAP Dimension
  CAP Dimension
http://www.coordguru.com


Key Features of Non-Relational Databases
  • Common Features
     • A call level interface (in ...
http://www.coordguru.com


Data Models of Non-Relational Databases
  • Data Models
      • Tuple
           •    A set of ...
http://www.coordguru.com


Classes of Non-Relational Databases
  • Classification by Data Model
      • Key-value Stores
 ...
http://www.coordguru.com


A Comparison of Non-Relational Databases
             Langu Replicatio                         ...
http://www.coordguru.com


Document-oriented vs. RDBMS
                                 CouchDB                           ...
http://www.coordguru.com




Thank you.
http://www.coordguru.com


Appendix: What is Coord?
  Architectural Comparison
  • dust: a distributed file system based o...
http://www.coordguru.com


Appendix: Coord Internals
 A space-based architecture built on distributed hash tables
    SB...
Upcoming SlideShare
Loading in …5
×

MapReduce Debates and Schema-Free

9,626 views

Published on

Published in: Technology
3 Comments
15 Likes
Statistics
Notes
  • Great stuff! Thanks!!!!!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • What a silly discussion. Greenplum, Aster, Teradata, DB2, and others can scale to petabytes and can outperform Hadoop on hundreds of commonly needed queries. Go learn about indexing, data skew, and preintegrated data. No, you cannot throw hardware at all the problems.

    Its common for new technologists to challenge the status quo and denigrate it. But this is not how Hadoop will survive and succeed. Hadoop's value is in processing not data management. Some of the PPT is correct -- there are workloads and tasks unsuitable for SQL in the first place.

    Don't get hung up on HDFS -- its just ONE of many data sources for Hadoop. The Data Warehouse is another, flat files, real time streams, etc. HDFS is drastically inferior to relational and YES some RDBMS scale to petabytes. If cost is the only benefit to Hadoop, you lose. Hadoop is free like a puppy is free.

    Hadoop parallel PROCESSING is really cool. I'd rather learn from someone with a balanced view of how to use each technology -- when to use which. Large data warehouses are not a competitor to Hadoop.

    Focus on the benefits of Hadoop processing, not this silly debate of RDBMS versus Hadoop.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • that's right
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
9,626
On SlideShare
0
From Embeds
0
Number of Embeds
66
Actions
Shares
0
Downloads
638
Comments
3
Likes
15
Embeds 0
No embeds

No notes for slide

MapReduce Debates and Schema-Free

  1. 1. http://www.coordguru.com MapReduce Debates and Schema-Free - Big Data, MapReduce, RDBMS+MapReduce, Non-Relational DB Woohyun Kim The creator of open source “Coord” (http://www.coordguru.com) 2010-03-03
  2. 2. http://www.coordguru.com The Advent of Big Data
  3. 3. http://www.coordguru.com Noah’s Ark Problem • Did Noah take dinosaurs on the Ark? • The Ark was a very large ship designed especially for its important purpose • It was so large and complex that it took Noah 120 years to build • How to put such a big thing • Diet or DNA? • Differentiate, Put, and Integrate • Larger? • More? • ‚Big Data‛ problem is just like that • Compression or Reduction • gzip, Fingerprint, DNA, MD5, … • Scale Up • Scale Out
  4. 4. http://www.coordguru.com Perspectives of Big Data •SAN •SQL •HDFS •MapReduce •Hbase, Voldemort, MongoDB, •Pig Cassandra •Hive, CloudBase •HadoopDB Store Process Analyze Retrieve •OLAP •SQL •Text/Data Mining •MapReduce •Social/Semantic Analysis •Key-Value •Visualization •RESTFul •Reporting
  5. 5. http://www.coordguru.com How to deal with “Big Data” Struggling to STORE and ANALYZE “Big Data”
  6. 6. http://www.coordguru.com Case Study: User Credit Analysis A User Credit Model User Credit 0.5 ∑ 0.5 amount quality ∑ 0.3 ∑ 0.1 0.7 0.3 0.6 Open100_write Answer_ Question_cn confidence popularity _cnt cnt t ∑ ∑ -0.5 0.5 -0.2 0.8 Confidence_negative Confidence_positive Popularity_negative popularity_positive ∑ ∑ ∑ 0.5 0.5 0.5 0.5 1.0 0.7 0.3 Confidence_positive_ Confidence_negativ best_answer_ Total_kinup_poi Penalty_cnt Admin_delete content e_user Report_cnt _cnt cnt nt ∑ 1.0 0.3 0.3 0.4 Aha_best_cnt Is_honor Dredt_level Is_sponsor ETL
  7. 7. http://www.coordguru.com Case Study: User Credit Analysis Preprocessing Blog Data for Analyzing User Credit Post * Attachment pt_log1.csv make_blog_ post_info.cpp pt_attachfile1.csv Post/Attachment * Buddy/Count/PowerBlogger/Commen Blog Post t att_pt_log.cpp Buddy pt_buddy.csv cal_buddy_ cnt.cpp Buddy * Count pt_count.csv att_visit_ count.cpp Buddy/Count * PowerBlogger pt_power_blog1.csv att_is_power blogger.cpp Buddy/Count/PowerBlogger * Comment pt_comment1.csv att_commenting.cpp Blogger
  8. 8. http://www.coordguru.com New Changes surrounding Data Storages • Volume • Data volumes have grown from tens of gigabytes in the 1990s to hundreds of terabytes and often petabytes in recent years • Scale Out • Relational databases are hard to scale • Partitioning(for scalability) ‚Relations‛ get broken • Replication(for availability) • Speed • The seek times of physical storage is not keeping pace with improvements in network speeds ‚New Relations‛ • Integration • Today’s data processing tasks increasingly have to access and combine data from many different non-relational sources, often over a network
  9. 9. http://www.coordguru.com Hadoop Revolution
  10. 10. http://www.coordguru.com Best Practice in Hadoop • Software Stack in Google/Hadoop • Cookbook for ‚Big Data‛ • Structured Data Storage for ‚Big Data‛ Row key Column key Row Structured Data Time Column Column stamp Family Family
  11. 11. http://www.coordguru.com Hadoop is changing the Game • Hadoop, DW, and BI
  12. 12. http://www.coordguru.com “Big Data” goes well with Hadoop • Parallelize Relational Algebra Operations using MapReduce
  13. 13. http://www.coordguru.com Case Study: Parallel Join • A Parallel Join Example using MapReduce
  14. 14. http://www.coordguru.com Case Study: Further Study in Parallel Join Problems • Need to sort • Move the partitioned data across the network • Due to shuffling, must send the whole data • Skewed by popular keys • All records for a particular key are sent to the same reducer • Overhead by tagging Alternatives • Map-side Join • Mapper-only job to avoid sort and to reduce data movement across the network • Semi-Join • Shrink data size through semi-join(by preprocessing)
  15. 15. http://www.coordguru.com Case Study: Improvements in Parallel Join Map-Side Join • Replicate a relatively smaller input source to the cluster • Put the replicated dataset into a local hash table • Join – a relatively larger input source with each local hash table • Mapper: do Mapper-side Join Semi-Join • Extract – unique IDs referenced in a larger input source(A) • Mapper: extract Movie IDs from Ratings records • Reducer: accumulate all unique Movie IDs • Filter – the other larger input source(B) with the referenced unique IDs • Mapper: filter the referenced Movie IDs from full Movie dataset • Join - a larger input source(A) with the filtered datasets • Mapper: do Mapper-side Join • Ratings records & the filtered movie IDs dataset
  16. 16. http://www.coordguru.com MapReduce Debates
  17. 17. http://www.coordguru.com MapReduce is just A Major Step Backwards!!! Dewitt and StoneBraker in January 17, 2008 • A giant step backward in the programming paradigm for large-scale data intensive applications • Schema are good • Type check in runtime, so no garbage • Separation of the schema from the application is good • Schema is stored in catalogs, so can be queried(in SQL) • High-level access languages are good • Present what you want rather than an algorithm for how to get it • No schema??! • At least one data field by specifying the key as input • For Bigtable/Hbase, different tuples within the same table can actually have different schemas • Even there is no support for logical schema changes such as views
  18. 18. http://www.coordguru.com MapReduce is just A Major Step Backwards!!! (cont’d) Dewitt and StoneBraker in January 17, 2008 • A sub-optimal implementation, in that it uses brute force instead of indexing • Indexing • All modern DBMSs use hash or B-tree indexes to accelerate access to data • In addition, there is a query optimizer to decide whether to use an index or perform a brute-force sequential search • However, MapReduce has no indexes, so processes only in brute force fashion • Automatic parallel execution • In the 1980s, DBMS research community explored it such as Gamma, Bubba, Grace, even commercial Teradata • Skew • The distribution of records with the same key causes is skewed in the map phase, so it causes some reduce to take much longer than others • Intermediate data pulling • In the reduce phase, two or more reduce attempt to read input files form the same map node simultaneously
  19. 19. http://www.coordguru.com MapReduce is just A Major Step Backwards!!! (cont’d) Dewitt and StoneBraker in January 17, 2008 • Not novel at all – it represents a specific implementation of well known techniques developed nearly 25 years ago • Partitioning for join • Application of Hash to Data Base Machine and its Architecture, 1983 • Joins in parallel on a shared-nothing • Multiprocessor Hash-based Join Algorithms, 1985 • The Case for Shared-Nothing, 1986 • Aggregates in parallel • The Gamma Database Machine Project, 1990 • Parallel Database System: The Future of High Performance Database Systems, 1992 • Adaptive Parallel Aggregation Algorithms, 1995 • Teradata has been selling a commercial DBMS utilizing all of these techniques for more than 20 years • PostgreSQL supported user-defined functions and user-defined aggregates in the mid 1980s
  20. 20. http://www.coordguru.com MapReduce is just A Major Step Backwards!!! (cont’d) Dewitt and StoneBraker in January 17, 2008 • Missing most of the features that are routinely included in current DBMS • MapReduce provides only a sliver of the functionality found in modern DBMSs • Bulk loader – transform input data in files into a desired format and load it into a DBMS • Indexing – hash or B-Tree indexes • Updates – change the data in the data base • Transactions – support parallel update and recovery from failures during update • integrity constraints – help keep garbage out of the data base • referential integrity – again, help keep garbage out of the data base • Views – so the schema can change without having to rewrite the application program • Incompatible with all of the tools DBMS users have come to depend on • MapReduce cannot use the tools available in a modern SQL DBMS, and has none of its own • Report writers(Crystal reports) • Prepare reports for human visualization • business intelligence tools(Business Objects or Cognos) • Enable ad-hoc querying of large data warehouses • data mining tools(Oracle Data Mining or IBM DB2 Intelligent Miner) • Allow a user to discover structure in large data sets • replication tools(Golden Gate) • Allow a user to replicate data from on DBMS to another • database design tools(Embarcadero) • Assist the user in constructing a data base
  21. 21. http://www.coordguru.com What the !@# MapReduce?
  22. 22. http://www.coordguru.com RDB experts Jump the MR Shark Greg Jorgensen in January 17, 2008 • Arg1: MapReduce is a step backwards in database access • MapReduce is not a database, a data storage, or management system • MapReduce is an algorithmic technique for the distributed processing of large amounts of data • Arg2: MapReduce is a poor implementation • MapReduce is one way to generate indexes from a large volume of data, but it’s not a data storage and retrieval system • Arg3: MapReduce is not novel • Hashing, parallel processing, data partitioning, and user-defined functions are all old hat in the RDBMS world, but so what? • The big innovation MapReduce enables is distributing data processing across a network of cheap and possibly unreliable computers • Arg4: MapReduce is missing features • Arg5: MapReduce is incompatible with the DBMS tools • The ability to process a huge volume of data quickly such as web crawling and log analysis is more important than guaranteeing 100% data integrity and completeness
  23. 23. http://www.coordguru.com DBs are hammers; MR is a screwdriver Mark C. Chu-Carroll • RDBs don’t parallelize very well • How many RDBs do you know that can efficiently split a task among 1,000 cheap computers? • RDBs don’t handle non-tabular data well • RDBs are notorious for doing a poor job on recursive data structures • MapReduce isn’t intended to replace relational databases • It’s intended to provide a lightweight way of programming things so that they can run fast by running in parallel on a lot of machines
  24. 24. http://www.coordguru.com MR is a Step Backwards, but some Steps Forward Eugene Shekita • Arg1: Data Models, Schemas, and Query Languages • Semi-structured data model and high level of parallel data flow query language is built on top of MapReduce • Pig, Hive, Jaql, Cascading, Cloudbase • Hadoop will eventually have a real data model, schema, catalogs, and query language • Moreover, Pig, Jaql, and Cascading are some steps forward • Support semi-structured data • Support more high level-like parallel data flow languages than declarative query languages • Greenplum and Aster Data support MapReduce, but look more limited than Pig, Jaql, Cascading • The calls to MapReduce functions wrapped in SQL queries will make it difficult to work with semi-structured data and program multi-step dataflows • Arg3: Novelty • Teradata was doing parallel group-by 20 years ago • UDAs and UDFs appeared in PostgreSQL in the mid 80s • And yet, MapReduce is much more flexible, and fault-tolerant • Support semi-structured data types, customizable partitioning
  25. 25. http://www.coordguru.com
  26. 26. http://www.coordguru.com Lessons Learned from the Debates Who Moved My Cheese?
  27. 27. http://www.coordguru.com Hybrids of MapReduce and RDBMS
  28. 28. http://www.coordguru.com Integrate MapReduce into RDBMS Vertica+Hadoop Oracle+Hadoop Sybase IQ Netezza+MapReduce Teradata+MapReduce HadoopDB Greenplum Aster Data
  29. 29. http://www.coordguru.com HadoopDB Details HadoopDB Architecture Connection parameters - database location - driver class - credentials Metadata - dataset - replica locations - data partitioning
  30. 30. http://www.coordguru.com An Interesting Friendship of RDBMS and MapReduce RDBMS vs. MapReduce RDBMS MapReduce Data size Gigabytes Petabytes Updates Read and write(Mutable) Write once, read many times(Immutable) Latency Low High Access Interactive(point query) and batch Batch(ad-hoc query in brute-force) Structure Fixed schema Semi-structured schema Language SQL Procedural (Java, C++, etc) Integrity High Low Scaling Nonlinear Linear RDBMS + MapReduce Greenplum, Pig, Hive, Aster Data, CloudBase Scalability, Fault HadoopDB tolerance, Flexibility SQL or Script MapReduce Performance, Efficiency MapReduce RDBMS
  31. 31. http://www.coordguru.com In-Database MapReduce vs. File-only MapReduce • In-Database MapReduce • Greenplum, Aster Data, HadoopDB • File-only MapReduce • Pig, Hive, Cloudbase In-Database MapReduce File-Only MapReduce Target User Analyst, DBA, Data Miner Computer Science Engineer Scale & Performance High High Hardware Costs Low Low Analytical Insights High High Failover & Recovery High High Use: Ad-Hoc Queries Easy (seamless) Harder (custom) Use: UI, Client Tools BI Tool (GUI), SQL (CLI) Developer Tool (Java) Use: Ecosystem High (JDBC, ODBC) Lower (custom) Protect: Data Integrity High (ACID, schema) Lower (no transaction guarantees) Protect: Security High (roles, privileges) Lower (custom) Protect: Backup & DR High (database backup/DR) Lower (custom) Performance: Mixed Workloads High (workload/QoS mgmt) Lower (limited concurrency) Performance: Network Bottleneck No (optimized partitioning) Higher (network inefficient) Operational Cost Low (1 DBA) Higher (several engineers)
  32. 32. http://www.coordguru.com Why Non-Relational?
  33. 33. http://www.coordguru.com Challenges in Traditional RDBMS • Volume • Data volumes have grown from tens of gigabytes in the 1990s to hundreds of terabytes and often petabytes in recent years • Speed • The seek times of physical storage is not keeping pace with improvements in network speeds ‚New Relations‛
  34. 34. http://www.coordguru.com Challenges in Traditional RDBMS (cont’d) • Scale Out • Is it possible to achieve a large number of simple read/write operations per second? • Traditional RDBMSs have not provided good horizontal scaling for OLTP • Partitioning(for scalability) ‚Relations‛ get broken • Replication(for availability) • Data warehousing RDBMSs provide horizontal scaling of complex joins and queries • Most of them are read-only or read-mostly • Integration • Today’s data processing tasks increasingly have to access and combine data from many different non-relational sources, often over a network
  35. 35. http://www.coordguru.com The New Faces of Data • Scale out • CAP Theorem • CAP theorem simply states that any distributed data system can only achieve two of these three at any given time • Hence when building distributed systems, Just Pick 2/3 • Design Issues • ACID • BASE Atomicity Consistency Isolation Durability Basically Available Soft-state Eventual Consistency v0
  36. 36. http://www.coordguru.com The New Faces of Data (cont’d) • Sparsity • Some data have sparse attributes • document-term vector Schema-Free • user-item matrix • semantic or social relations • Some data do not need ‘relational’ property, or complex join queries • log-structured data • stacking or streamed data • e.g. Facebook, Server Density(MySQL -> MongoDB) • Immutable • Do not need update and delete data, only insert it with versions • tracking history • lock-free • atomicity is based on just a key
  37. 37. http://www.coordguru.com Non-Relational Databases
  38. 38. http://www.coordguru.com Trends of Emergent Data Stores Trend Google(Jan.) 2500 2000 1500 1000 500 0 Voldemort Sclaris Cassandra CouchDB MongoDB ScaleDB Drizzle VoltDB Tokyo Hbase HyperTable Bigtable SimpleDB Riak Redis MySQL Cluster On-going classification by Woohyun Kim
  39. 39. http://www.coordguru.com Emergent Data Stores in CAP Dimension CAP Dimension
  40. 40. http://www.coordguru.com Key Features of Non-Relational Databases • Common Features • A call level interface (in contrast to a SQL binding) • HTTP/REST or easy to program APIs • Fast indexes on large amounts of data • Lookups by one and more keys(key-value or document) • Ability to horizontally scale throughput over many servers • Automatic sharding or client-side manual sharding • Built-in replication(sync or async) • Eventual Consistency • Ability to dynamically define attributes or data schema • Key-Value, Column, or Document • Support for MapReduce
  41. 41. http://www.coordguru.com Data Models of Non-Relational Databases • Data Models • Tuple • A set of attribute-value pairs • Attribute names are defined in a schema • Values must be scalar(like numbers and strings), not BLOBs • The values are referenced by attribute name, not by ordinal position • Document • A set of attribute-value pairs • Attribute names are dynamically defined for each document at runtime • Unlike Tuple, there is no global schema for attributes • Values may be complex values or nested values • Multiple indexes are supported • Extensible Record • A hybrid between Tuple and Document • Families of attributes are defined in a schema • New attributes can be defined (within an attribute family) on a per-record basis • Object • A set of attribute-value pairs • Values may be complex values or pointers to other objects
  42. 42. http://www.coordguru.com Classes of Non-Relational Databases • Classification by Data Model • Key-value Stores • Store values and an index to find them • Provide replication, versioning, locking, transactions, sorting, and etc. • Document Stores • Store indexed documents(with multiple indexes) • Not support locking, synchronous replication, and ACID transactions • Instead of ACID, support BASE for much higher performance and scalability • Provide some simple query mechanisms • Extensible Record Stores(=Column-oriented Stores) • Store extensible records that can be horizontally and vertically partitioned across nodes • Both rows and columns are splitted over multiple nodes • Rows are split across nodes by range partitioning • Columns of a table are distributed over multiple nodes by using ‚column groups‛ • Relational Databases • Store, index, and query tuples • Some new RDBMSs provide horizontal scaling
  43. 43. http://www.coordguru.com A Comparison of Non-Relational Databases Langu Replicatio Consistency & Data mode Doc Project Partitioning Persistence Client Protocol Community age n Transaction l s Lock + limited ACID tr Bigtable C++ Sync(GFS) Range Memtable/SSTable on GFS ansactions Custom API Column A Google, no Lock + limited ACID tr Hbase Java Sync(HDFS) Range Memtable/SSTable on HDFS ansactions Custom API, Thrift, Rest Column A Apache, yes Lock + limited ACID tr Hypertable C++ Sync(FS) Range CellCache/CellStore on any FS ansactions Thrift, other Column A Zvents, Baidu, yes MVCC + limited ACID t Column & Key Cassandra Java Async Hash On-disk ransactions Thrift -Value B Facebook, no Key-Value or Sync(on clie Hash (on client Custom API(python, php,jav Coord C++ nt-side) -side) Pluggable: in-memory, Lucene no a, c++) Document(jso A NHN, yes n) Dynamo ? Yes Yes ? Custom API Key-Value A Amazon, no Key-Value(bl Voldemort Java Async Hash Pluggable: BerkleyDB, Mysql MVCC Java API ob/text) A Linkedin, no Hash (on client In-memory with background sna Redis C Sync -side) pshots lock Custom API(Collection) Key-Value C some In-memory or on-disk(hash , b-t Manual shardin lock + limited Tokyo Tyrant C Async g ree, fixed-size/variable-length r ACID transactions Key-Value C ecord tables) lock + limited ACID tr Key-Value(bl Scalaris Erlang Sync Range Only in-memory ansactions Erlang, Java, HTTP ob) B OnScale, no Key-Value(bl Kai Erlang ? Yes On-disk Dets file Memcached ob) C no Key-Value(bl Dynomite Erlang Yes Yes Pluggable: couch, dets Custom ascii, Thrift ob) D+ Powerset, no Key-Value(bl MemcacheDB C Yes No BerkleyDB Memcached ob) B some Pluggable: in-memory, ets, dets, Key-Value & Riak Erlang Async Hash osmos tables (no indices on 2nd MVCC Rest(json-based) Document B no key fields) No automated s SimpleDB ? Async harding S3 no Custom API Document B Amazon, no Pluggable: BerkleyDB, Custom, ThruDB C++ Yes No Mysql, S3 Thrift Document C+ Third rail, unsure No automated s On-disk with append-only B-tre HTTP, json, Custom API(ma Document(jso CouchDB Erlang Async harding e MVCC p/reduce views) n) A Apache, yes HTTP, bson, Custom API(Cur Document(bs MongoDB C++ Async Sharding new On-disk with B-tree Filed-level sor) on) A 10gen, yes Neo4J On-disk linked lists Custom API(Graph) Graph On-going classification by Woohyun Kim
  44. 44. http://www.coordguru.com Document-oriented vs. RDBMS CouchDB MongoDB MySQL Terminology Document, Field, Database Document, Key, Collection Data Model Document-Oriented (JSON) Document-Oriented (BSON) Relational string, int, double, boolean, date, bytea Data Types Text, numeric, boolean, and list Link rray, object, array, others Large Objects (Files) Yes (attachments) Yes (GridFS) no??? Master-master (with developer sup Replication Master-slave Master-slave plied conflict resolution) Object(row) Storage One large repository Collection based Table based Map/reduce of javascript functions Dynamic; object-based query language Query Method Dynamic; SQL to lazily build an index per query Secondary Indexes Yes Yes Yes Atomicity Single document Single document Yes – advanced Interface REST Native drivers Native drivers Server-side batch dat Yes, via javascript(thru. map/reduce Yes, via javascript Yes (SQL) a manipulation views) Written in Erlang C++ C Concurrency Control MVCC Update in Place Update in Place
  45. 45. http://www.coordguru.com Thank you.
  46. 46. http://www.coordguru.com Appendix: What is Coord? Architectural Comparison • dust: a distributed file system based on DHT • coord spaces: a resource sharable store system based on SBA • coord mapreduce: a simplified large-scale data processing framework • warp: a scalable remote/parallel execution system • graph: a large-scale distributed graph search system
  47. 47. http://www.coordguru.com Appendix: Coord Internals  A space-based architecture built on distributed hash tables  SBA(Space-based Architecture)  processes communicate with others thru. only spaces  DHT(Distributed Hash Tables)  data identified by hash functions are placed on numerically near nodes  A computing platform to project a single address space on distributed memories  As if users worked in a single computing environment App take write read 2m-1 0 node 1 node 2 node 3 node n

×