NoSQL with Hadoop and HBase

  • 8,934 views
Uploaded on

NoSQL overview presentation for BeJUG - 17/6/2010

NoSQL overview presentation for BeJUG - 17/6/2010

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,934
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
587
Comments
0
Likes
21

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. NoSQL with HBase and Hadoop BeJUG - 17/6/2010 http://www.flickr.com/photos/wolfgangstaudt/2215246206/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 2. Who am I » Steven Noels - stevenn@outerthought.org » Outerthought : scalable content applications » makers of Daisy, Lily and Kauri : open source internet/Java/REST/content apps IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 2
  • 3. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 4. An evolution driven by pain. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 5. History 2. simplification 1. standardization hierarchical databases IMS XMLDB RDBMS OODBMS IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 5
  • 6. History 4. rethinking the problem RDBMS NOSQL caching denormalisation sharding replication ... 3. pain IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 6
  • 7. Four Trends » Trend 1 : Data Size » Trend 2 : Connectedness » Trend 3 : Semi-structure » Trend 4 : Architecture IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 7
  • 8. Trend 1: Data size ExaBytes (10!") of data stored per year 988 1000 Each year more and more digital data is created. Over t wo 750 years we create more digital data than all 623 the data created in history before that. 500 397 253 250 161 0 2006 2007 2008 2009 2010 Data source: IDC 2007 3 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 8
  • 9. Trend 2: Connectedness Giant Global Graph (GGG) Over time data has evolved to Ontologies be more and more interlinked and connected. RDF Hypertext has links, Blogs have pingback, Tagging groups all related data Folksonomies Information connectivity Tagging Wikis User-generated content Blogs RSS Hypertext Text documents web 1.0 web 2.0 “web 3.0” 1990 2000 2010 2020 4 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 9
  • 10. Trend 3: Semi-structure ! Individualization of content • In the salary lists of the 1970s, all elements had exactly one job • In Or 15? lists of the 2000s, we need 5 job columns! Or 8? the salary ! All encompassing “entire world views” • Store more data about each entity ! Trend accelerated by the decentralization of content generation that is the hallmark of the age of participation (“web 2.0”) 5 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 10
  • 11. Trend 4: Architecture 1980s: Mainframe applications Application DB 6 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 11
  • 12. Trend 4: Architecture 1990s: Database as integration hub Application Application Application DB 7 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 12
  • 13. Trend 4: Architecture 2000s: (moving towards) Decoupled services with their own backend Application Application Application DB DB DB 8 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 13
  • 14. http://bigdatamatters.com/bigdatamatters/2010/04/high-availability-with-oracle.html IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 14
  • 15. Enter NoSQL IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 16. It’s a Cambrian Explosion Cassandra NoSQL neo4j IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 16
  • 17. Buzz-oriented development ? IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 17
  • 18. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 18
  • 19. Common themes » SCALE SCALE SCALE » new datamodels » devops » N-O-SQL » The Cloud : technology is of no interest anymore IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 19
  • 20. New Data » sparse structures » weak schemas » graphs » semi-structured » document-oriented IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 20
  • 21. NoSQL » Not a movement. » Not ANSI NoSQL-2010. » Not one-size-fits-all. » Not (necessarily) anti-RDBMS. » No silver bullet. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 21
  • 22. NoSQL = pro Choice IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 22
  • 23. NoSQL = toolbox IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 23
  • 24. NOSQL, if you need ... » horizontal scaling (out rather than up) » unusually common data (aka free-structured) » speed (especially for writes) » the bleeding edge IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 24
  • 25. SQL/RDBMS, if you need ... » SQL » ACID » normalisation » a defined liability IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25
  • 26. Some RDBMS bashing » sparse and dynamic tables IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 26
  • 27. Some RDBMS bashing » solution mysql> desc thefields; +---------------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------------+---------------+------+-----+---------+-------+ | doc_id | bigint(20) | NO | PRI | NULL | | ... | fieldtype_id | bigint(20) | NO | PRI | NULL | | ... | stringvalue | varchar(255) | YES | MUL | NULL | | | datevalue | datetime | YES | MUL | NULL | | | datetimevalue | datetime | YES | MUL | NULL | | | integervalue | bigint(20) | YES | MUL | NULL | | | floatvalue | double | YES | MUL | NULL | | | decimalvalue | decimal(10,5) | YES | MUL | NULL | | | booleanvalue | char(1) | YES | MUL | NULL | | ... +---------------------+---------------+------+-----+---------+-------+ 25 rows in set (0.00 sec) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 27
  • 28. More RDBMS bashing » replication and failure recovery » (when working on a budget) » application-level partitioning logic IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 28
  • 29. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 30. Academic background » Amazon Dynamo » Google BigTable » Eric Brewer CAP theorem IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 30
  • 31. Shameless plug nosqlsummer.org IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 31
  • 32. Shameless plug nosqlsummer.org IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 31
  • 33. Amazon Dynamo » coined the term ‘eventual consistency’ » consistent hashing IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 32
  • 34. Eventual Consistency Gone Wild IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 33
  • 35. 1. replica server 1 2. server 2 1. update ACL: disallow mother from folder ‘spring break’ 2. upload spring break pictures how is my boy doing on his spring break? IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 34
  • 36. Consistent hashing » a solution for naive mod n distributions » specifically in the case of adding or deleting nodes IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 35
  • 37. Consistent hashing N value 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2 (c) Basho/Riak 5 Tuesday, November 17, 2009 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 36
  • 38. Google BigTable » multi-dimensional column-oriented database » on top of GoogleFileSystem » object versioning IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 37
  • 39. CAP theorem strong high consistency availability partition- tolerance IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 38
  • 40. CAP » Strong Consistency: all clients see the same view, even in the presence of updates » High Availability: all clients can find some replica of the data, even in the presence of failures » Partition-tolerance: the system properties hold even when the system is partitioned IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 39
  • 41. Culture Clash » ACID » BASE » highest priority: strong » availability and scaling consistency for highest priorities transactions » weak consistency » availability less important » optimistic » pessimistic » best effort » rigorous analysis » simple and fast » complex mechanisms spectrum IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 40
  • 42. Availability ≠ total async ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 43. ✘ The Enterprise Service Bus bus = congestion IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 42
  • 44. Bus systems » objects don’t fit in a pipe » object ➙ message » serialization / de-serialization cost » message size » queuing = cost IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 43
  • 45. Use a mixture of both »async + sync stuff which matters ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 44
  • 46. 2.1 Interlude IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 47. 2.1 Interlude IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 48. 2.1 Interlude IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 49. Processing large datasets : Hadoop + Map/Reduce IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 50. Hadoop: HDFS + MapReduce » single filesystem + single execution-space IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 47
  • 51. M/R Execution (c) Google IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 48
  • 52. MapReduce example: WordCount IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 49
  • 53. Hadoop ecosystem » Hadoop Common » Subprojects » Chukwa: A data collection system for managing large distributed systems. » HBase: A scalable, distributed database that supports structured data storage for large tables. » HDFS: A distributed file system that provides high throughput access to application data. » Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying. » MapReduce: A software framework for distributed processing of large data sets on compute clusters. » Pig: A high-level data-flow language and execution framework for parallel computation. » ZooKeeper: A high-performance coordination service for distributed applications. » Mahout: machine learning libraries IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 50
  • 54. Processing large datasets with MR » Benefit from parallellisation » Less modelling upfront (ad-hoc processing) » Compartmentalized approach reduces operational risks » AsterData et al. have SQL/MR hybrids for huge-scale BI IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 51
  • 55. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 56. We welcome the Polyglot Persistence overlords. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 57. The NOSQL footprint free-structured or sparse data NOSQL MongoDB CouchDB neo4j Cassandra available (complexity) simple operational HBase highly scalable and constraints ACID, SQL referential integrity, typed data (c) me! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 54
  • 58. Categories » key-value stores » column stores » document stores » graph databases IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 55
  • 59. Key-value stores » Focus on scaling to huge amounts of data » Designed to handle big loads » Often: cfr. Amazon Dynamo » ring partitioning and replication » Data model: key/value pairs IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 56
  • 60. Key-value stores » Redis » Voldemort » Tokyo Cabinet IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 57
  • 61. Redis » REmote DIctionary Server » http://code.google.com/p/redis/ » vmware IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 58
  • 62. Redis Features » persisted memcache, ‘awesome’ » RAM-based + persistable » key ➙ values: string, list, set » higher-level ops » i.e. push/pop and sort for lists » fast (very) » configurable durability » client-managed sharding IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 59
  • 63. Voldemort » http://project-voldemort.com/ » LinkedIn IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 60
  • 64. Voldemort » persistent » distributed » fault-tolerant » hash table IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 61
  • 65. Voldemort API: GET, PUT, DELETE IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 62
  • 66. Voldemort routing logic moving up the stack, smaller latency IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 63
  • 67. Column stores » BigTable clones » Sparseness! » Data model: columns ➙ column families ➙ cells » Datums keyed by: row, column, time, index » Row-range ➙ tablet ➙ distribution IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 64
  • 68. Column stores » BigTable » HBase » Cassandra IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 65
  • 69. BigTable » http://labs.google.com/papers/bigtable.html » Google » layered on top of GFS IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 66
  • 70. HBase » http://hadoop.apache.org/hbase/ » StumbleUpon / Adobe / Cloudera IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 67
  • 71. HBase » sorted » persisted » distributed » storage system » column-oriented » multi-dimensional » highly-available » adds random access » high-performance reads and writes atop HDFS IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 68
  • 72. HBase data model » Distributed multi-dimensional sparse map » Multi-dimensional keys: (table, row, family:column, timestamp) → value » Keys are arbitrary strings » Access to row data is atomic IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 69
  • 73. Sample schema !"#$%&'(%)*+#,-% )#&*+,-./%& 0122345322!12 )#&*+,-./%& 0122345322!14 !"#$$%&'( (c) eCircle IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 70
  • 74. Storage architecture © lars george IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 71
  • 75. Cassandra » http://cassandra.apache.org/ » Rackspace / Facebook IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 72
  • 76. Cassandra » Key-value store (with added structure) » Reliability (identical nodes) » Eventual consistent » Distributed A C » Tunable » Partitioning P » Replication IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 73
  • 77. Cassandra applicability FIT NO FIT » Scalable reliability » Flexible indexing (through identical » Only PK-based nodes) querying » Linear scaling » Big Binary Data » Write throughput » 1 Row must fit in » Large Data Sets RAM entirely IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 74
  • 78. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 75
  • 79. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 76
  • 80. Document databases » ≈ K/V stores, but DB knows what the Value is » Lotus Notes heritage » Data model: collections of K/V collections » Documents often versioned IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 77
  • 81. Document stores » CouchDB » MongoDB » Riak IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 78
  • 82. CouchDB » http://couchdb.apache.org/ » couch.io IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 79
  • 83. CouchDB » fault-tolerant » schema-free » document-oriented » accessible via a RESTful HTTP/JSON API IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 80
  • 84. CouchDB documents { “_id”: ”BCCD12CBB”, “_rev”: ”AB764C”, “type”: ”person”, “name”: ”Darth Vader”, “age”: 63, “headware”: [“Helmet”, “Sombrero”], “dark_side”: true } IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 81
  • 85. CouchDB REST API » HTTP » PUT /db/docid » GET /db/docid » POST /db/docid » DELETE /db/docid IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 82
  • 86. CouchDB Views » MapReduce-based » Filter, Collate, Aggregate » Javascript map reduce function (doc) { function (Key, Values) { for(var i in doc.tags) var sum = 0; emit(doc.tags[i], 1); for(var i in Values) } sum += Values[i]; return sum; } IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 83
  • 87. CouchDB » be careful on semantics » replication ≠ partioning/sharding ! » distributed database = distributable database » sharded / distributed deployment requires proxy layer IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 84
  • 88. MongoDB » http://www.mongodb.org/ » 10gen IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 85
  • 89. MongoDB » cfr. CouchDB, really » except for: » C++ » performance focus » runtime queries (mapreduce still available) » native drivers (no REST/HTTP layering) » no MVCC: update-in-place » auto sharding (alpha) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 86
  • 90. Graph databases » Focus on modeling structure of data - interconnectivity » Scale, but only to the complexity of data » Data model: property graphs IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 87
  • 91. Graph databases » Neo4j » AllegroGraph (RDF) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 88
  • 92. Neo4j » http://neo4j.org/ » Neo Technology IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 89
  • 93. Neo4j » data = nodes + relationships + key/value properties IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 90
  • 94. Neo4j » many language bindings, little remoting » ‘whiteboard’ friendly » scaling to complexity (rather than volume?) » lots of focus on domain modelling » SPARQL/SAIL impl for triple geeks » mostly RAM centric (with disk swapping & persistence) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 91
  • 95. Bandwagonjumpers » JCR / Jackrabbit » GT/M » RDF stores IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 92
  • 96. Market maturization IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 97. Rise of integrators » Cloudera (H-stack) » Riptano (Cassandra) » Cloudant (hosted CouchDB) » (Outerthought: HBase) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 94
  • 98. VC capital » Cloudera » couch.io » Neo » 10gen » many others IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 95
  • 99. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  • 100. the fireside conversations http://www.flickr.com/photos/52641994@N00/516394238/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 97
  • 101. NOSQL applicability » Horizontal scaling » Multi-Master » Data representation » search of simplicity » data that doesn’t fit the E-R model (graphs, trees, versions) » Speed IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 98
  • 102. Tool selection » be careful with the marketeese: smoke and mirrors beware! » monitor dev list, IRC, Twitter, blogs » monitor project ‘sponsors’ » mix-and-match: polyglot persistency » DON’T NOSQL WITHOUT INTERNAL SYS ARCHS & DEV(OP)S ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 99
  • 103. Our Context: Lily » cloud-scalable content store and search repository » successor (in many ways) of Daisy IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 100
  • 104. Complexity complexity software architecture 3.0 2.0 1.0 age IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 101
  • 105. Complexity complexity user interest 3.0 2.0 1.0 age IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 102
  • 106. Business Development 101 user interest budget IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 103
  • 107. Solution sophistication ability to cope 3.0 mysql nosql? 2.0 1.0 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 104
  • 108. We Prefer Sophistication » the challenge for us was to scale ... without dropping features IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 105
  • 109. The typical CMS ‘architecture’ database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 106
  • 110. The typical CMS ‘architecture’ application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 107
  • 111. The typical CMS ‘architecture’ more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 108
  • 112. The typical CMS ‘architecture’ even more cache more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 109
  • 113. The typical CMS ‘architecture’ client even more cache more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 110
  • 114. The typical CMS ‘architecture’ client (+cache) even more cache more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 111
  • 115. What we found hard to scale » access control » facet browsing » all the nifty stuff people were using our software for » ... anything that required random access to in-memory-cache data for computations IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 112
  • 116. Beyond the ‘scaling’ problem » three-prong data layer fs » result set merging (between MySQL & Lucene) » happened in appcode/memory » ‘transactions’, set operations = hard IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 113
  • 117. Requirements, phase I » automatic scaling to large data sets » fault-tolerance: replication, automatic handling of failing nodes » a flexible data model supporting sparse data » runs on commodity hardware » efficient random access to data » open source, ability to participate in the development thus drive the direction of the project » some preference for a Java-based solution IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 114
  • 118. Requirements, phase II » After careful consideration, we realized the important choices were also: » consistency: no chance of having two conflicting versions of a row » atomic updates of a single row, single-row transactions » bonus points for MapReduce integration » e.g. full-text index rebuilding IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 115
  • 119. That brought us to HBase, which bought us: » a datamodel where you can have column families which keep all versions and others which do not, which fits very well on our CMS document model » ordered tables with the ability to do range scans on them, which allows to build scalable indexes on top of it » HDFS, a convenient place to store large blobs » Apache license and community, a familiar environment for us IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 116
  • 120. » OK, so now we had a data store ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 117
  • 121. » However, content repository = store + search ! u ch o IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 118
  • 122. a s w at ! h sy T a .. .) e ver we h o ( IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 119
  • 123. Search ponderings » CMS = two types of search » structured search » numbers, strings » based on logic (SQL, anyone?) » information retrieval (or: full-text search) » text » based on statistics IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 120
  • 124. Search ponderings » All of that, at scale IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 121
  • 125. Structured Search » HBase Indexing Library » idea from Google App Engine datastore indexes » http://code.google.com/appengine/articles/ index_building.html rowkey col col rowkey col order A val3 foo6 val2-B B val2 foo7 val3-A content table index table A IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 122
  • 126. Full-text / IR search » Lucene? » no sharding (for scale) » no replication (for availability) » batched index updates (not real-time) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 123
  • 127. Beyond Lucene » Katta » scalable architecture, however only search, no indexing » Elastic Search » very young (sorry) » hbasene et al. » stores inverted index in HBase, might not scale all features » SOLR » widely used, schema, facets, query syntax, cloud branch More info: http://lilycms.org/lily/prerelease/technology.html IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 124
  • 128. ? + = r ? ! O a sy E IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 125
  • 129. Remember distribution ? Remember secondary indexes ? ➙ Need for reliable queuing IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 126
  • 130. Connecting things » we needed a reliable bridge between our main storage (HBase) and our index/search server(s) (SOLR) » indexing, reindexing, mass reindexing (M/R) » we need a reliable method of updating HBase secondary indexes » all of that eventually to run distributed » distribution means coping with failure IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 127
  • 131. Solution » ACMEMessageQueue ? Bzzzzzt. We wanted fault-safe HBase persistence for the queues. Also for ease of administration. » ➙ WAL & Queue implemented on top of HBase tables IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 128
  • 132. WAL / Queue » WAL » Queue » guaranteed execution » triggering of async of synchronous actions actions » call doesn’t return before » e.g. (re)index (updated) secondary action finishes record with SOLR back-end » e.g. update secondary actions » size depends on speed of » if all goes well, back-end process size = #concurrent ops » will be useful/made available outside of Lily context as well! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 129
  • 133. The Sum » Lily model (records & fields) » mapped onto HBase (=storage) » indexed and searchable through SOLR » using a WAL/Queue mechanism implemented in HBase » runtime based on Kauri » with client/server comms via Avro IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 130
  • 134. Architecture IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 131
  • 135. Architecture IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 132
  • 136. ere! th ea rly Roadmap N » June 7-8: release of learning material (architecture, model, API, Javadoc) ➥ www.lilycms.org ➥ bit.ly/lilyprerelease » Tomorrow: WAL/queue » Mid July = ‘proof of architecture’ release » from there on, ca. 3-monthly releases leading up to Lily 1.0 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 133
  • 137. bit.ly/lilyprerelease IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 134
  • 138. License » Apache IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 135
  • 139. Business model » Consulting, mentoring, turn-key projects » audience: developers » Strong focus on partner relations » targeting vertical markets » geographic coverage » SaaS offerings » Markets: media, finance, insurance, govt, heritage ... LOTS of semi-structured data » Not: OLAP IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 136
  • 140. Reading material » Amazon Dynamo, Google BigTable, CAP » http://nosql.mypopescu.com/ » http://nosql-database.org/ » http://twitter.com/nosqlupdate » http://highscalability.com/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 137
  • 141. Questions? http://www.flickr.com/photos/leehaywood/4237636853/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 138
  • 142. Thanks for your attention ! » stevenn@outerthought.org » @stevenn IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 139