0
NoSQL
with HBase and Hadoop
BeJUG - 17/6/2010




                                                                        ...
Who am I


» Steven Noels - stevenn@outerthought.org

» Outerthought : scalable content applications

» makers of Daisy, L...
1.        Intro
2.        Theory
3.        Technology
4.        Experiences
 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (...
An evolution
driven by pain.

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
History

                                                                                     2. simplification


         ...
History


                                     4. rethinking
                                     the problem


     RDBMS...
Four Trends


» Trend 1 : Data Size

» Trend 2 : Connectedness

» Trend 3 : Semi-structure

» Trend 4 : Architecture




 ...
Trend 1: Data size
               ExaBytes (10!") of data stored per year
                                                ...
Trend 2: Connectedness
                                                                                                   ...
Trend 3: Semi-structure
! Individualization of content
   • In the salary lists of the 1970s, all elements had exactly one...
Trend 4: Architecture

                      1980s: Mainframe applications


                                     Applicat...
Trend 4: Architecture

                  1990s: Database as integration hub


             Application             Applica...
Trend 4: Architecture

           2000s: (moving towards) Decoupled services
                               with their own...
http://bigdatamatters.com/bigdatamatters/2010/04/high-availability-with-oracle.html



             IIC » TECHNOLOGIEPARK ...
Enter NoSQL

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
It’s a Cambrian Explosion

                                       Cassandra




                                  NoSQL
  ...
Buzz-oriented
                                                            development




                                ...
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   18
Common themes

» SCALE SCALE SCALE

» new datamodels

» devops

» N-O-SQL

» The Cloud :
 technology is of no interest any...
New Data

» sparse structures

» weak schemas

» graphs

» semi-structured

» document-oriented



       IIC » TECHNOLOGI...
NoSQL

» Not a movement.

» Not ANSI NoSQL-2010.

» Not one-size-fits-all.

» Not (necessarily) anti-RDBMS.

» No silver bu...
NoSQL = pro Choice




    IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   22
NoSQL = toolbox




    IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   23
NOSQL, if you need ...


» horizontal scaling (out rather than up)

» unusually common data (aka free-structured)

» speed...
SQL/RDBMS, if you need ...


» SQL

» ACID

» normalisation

» a defined liability




        IIC » TECHNOLOGIEPARK 3 » B-...
Some RDBMS bashing
» sparse and dynamic tables




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerth...
Some RDBMS bashing
» solution

                    mysql> desc thefields;
                    +---------------------+-----...
More RDBMS bashing



» replication and failure recovery
  » (when working on a budget)

» application-level partitioning ...
1.        Intro
2.        Theory
3.        Technology
4.        Experiences
 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (...
Academic background



» Amazon Dynamo

» Google BigTable

» Eric Brewer CAP theorem




      IIC » TECHNOLOGIEPARK 3 » B...
Shameless plug




            nosqlsummer.org
    IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.o...
Shameless plug




            nosqlsummer.org
    IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.o...
Amazon Dynamo
» coined the term ‘eventual consistency’

» consistent hashing




       IIC » TECHNOLOGIEPARK 3 » B-9052 Z...
Eventual Consistency Gone Wild




    IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   33
1.
                                                replica
                  server 1         2.          server 2




1. ...
Consistent hashing
» a solution for naive mod n distributions
 » specifically in the case of adding or deleting nodes




 ...
Consistent hashing
                                               N value
                   2160                0

      ...
Google BigTable
» multi-dimensional column-oriented database

» on top of GoogleFileSystem

» object versioning




      ...
CAP theorem


             strong                                high
           consistency                          avai...
CAP
» Strong Consistency: all clients see the
 same view, even in the presence of updates
» High Availability: all clients...
Culture Clash
» ACID                                              » BASE
 » highest priority: strong                      ...
Availability ≠
total async !

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
✘
The Enterprise Service Bus




                                bus =

                          congestion



    IIC » ...
Bus systems

» objects don’t fit in a pipe

» object ➙ message

» serialization / de-serialization cost

» message size

» ...
Use a mixture of both



»async + sync



                                        stuff which matters !




    IIC » TECH...
2.1 Interlude

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
2.1 Interlude

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
2.1 Interlude

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Processing large datasets :
Hadoop + Map/Reduce


       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outertho...
Hadoop: HDFS + MapReduce
» single filesystem + single execution-space




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARD...
M/R Execution




                                                                                (c) Google


    IIC » T...
MapReduce example: WordCount




    IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   49
Hadoop ecosystem
» Hadoop Common
» Subprojects
  » Chukwa: A data collection system for managing large distributed systems...
Processing large datasets with MR

» Benefit from parallellisation

» Less modelling upfront (ad-hoc processing)

» Compart...
1.        Intro
2.        Theory
3.        Technology
4.        Experiences
 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (...
We welcome
the Polyglot
Persistence
overlords.
  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
The NOSQL footprint
                            free-structured or sparse data



                                        ...
Categories


» key-value stores

» column stores

» document stores

» graph databases




       IIC » TECHNOLOGIEPARK 3 ...
Key-value stores


» Focus on scaling to huge amounts of data

» Designed to handle big loads

» Often: cfr. Amazon Dynamo...
Key-value stores



» Redis

» Voldemort

» Tokyo Cabinet




          IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT)...
Redis



» REmote DIctionary Server

» http://code.google.com/p/redis/

» vmware




       IIC » TECHNOLOGIEPARK 3 » B-90...
Redis Features
» persisted memcache, ‘awesome’

» RAM-based + persistable

» key ➙ values: string, list, set

» higher-lev...
Voldemort




» http://project-voldemort.com/

» LinkedIn




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » ...
Voldemort


» persistent

» distributed

» fault-tolerant

» hash table




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNA...
Voldemort


                                                        API: GET, PUT,
                                       ...
Voldemort




                    routing logic moving up the stack,
                    smaller latency

    IIC » TECHNO...
Column stores


» BigTable clones

» Sparseness!

» Data model: columns ➙ column families ➙ cells
 » Datums keyed by: row,...
Column stores



» BigTable

» HBase

» Cassandra




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.oute...
BigTable



» http://labs.google.com/papers/bigtable.html

» Google

» layered on top of GFS




       IIC » TECHNOLOGIEP...
HBase




» http://hadoop.apache.org/hbase/

» StumbleUpon / Adobe / Cloudera




       IIC » TECHNOLOGIEPARK 3 » B-9052 ...
HBase
» sorted                                            » persisted
» distributed                                       ...
HBase data model
» Distributed multi-dimensional sparse map

» Multi-dimensional keys:
 (table, row, family:column, timest...
Sample schema
           !"#$%&'(%)*+#,-%
   )#&*+,-./%&
   0122345322!12




   )#&*+,-./%&
   0122345322!14



         ...
Storage architecture




                                                                                © lars george

  ...
Cassandra




» http://cassandra.apache.org/

» Rackspace / Facebook




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARD...
Cassandra
» Key-value store (with added structure)

» Reliability (identical nodes)

» Eventual consistent

» Distributed
...
Cassandra applicability
            FIT                                                      NO FIT

» Scalable reliabilit...
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   75
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   76
Document databases


» ≈ K/V stores, but DB knows what the Value is

» Lotus Notes heritage

» Data model: collections of ...
Document stores



» CouchDB

» MongoDB

» Riak




         IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.oute...
CouchDB




» http://couchdb.apache.org/

» couch.io




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.o...
CouchDB


» fault-tolerant

» schema-free

» document-oriented

» accessible via a RESTful HTTP/JSON API




       IIC » ...
CouchDB documents

{
    “_id”: ”BCCD12CBB”,
    “_rev”: ”AB764C”,
    “type”: ”person”,
    “name”: ”Darth Vader”,
    “a...
CouchDB REST API


» HTTP
 » PUT /db/docid
 » GET /db/docid
 » POST /db/docid
 » DELETE /db/docid




       IIC » TECHNOL...
CouchDB Views
» MapReduce-based

» Filter, Collate, Aggregate

» Javascript

         map                                 ...
CouchDB


» be careful on semantics
 » replication ≠ partioning/sharding !
 » distributed database = distributable databas...
MongoDB




» http://www.mongodb.org/

» 10gen




      IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outertho...
MongoDB
» cfr. CouchDB, really

» except for:
 » C++
 » performance focus
 » runtime queries (mapreduce still available)
 ...
Graph databases


» Focus on modeling structure of data -
 interconnectivity
» Scale, but only to the complexity of data

...
Graph databases




» Neo4j

» AllegroGraph (RDF)




      IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outer...
Neo4j




» http://neo4j.org/

» Neo Technology




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outert...
Neo4j
» data = nodes + relationships + key/value properties




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) ...
Neo4j
» many language bindings, little remoting

» ‘whiteboard’ friendly

» scaling to complexity (rather than volume?)

»...
Bandwagonjumpers



» JCR / Jackrabbit

» GT/M

» RDF stores




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT)...
Market
maturization

  IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Rise of integrators


» Cloudera (H-stack)

» Riptano (Cassandra)

» Cloudant (hosted CouchDB)

» (Outerthought: HBase)


...
VC capital

» Cloudera

» couch.io

» Neo

» 10gen

» many others



        IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (...
1.        Intro
2.        Theory
3.        Technology
4.        Experiences
 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (...
the fireside conversations




http://www.flickr.com/photos/52641994@N00/516394238/



                        IIC » TECHNOL...
NOSQL applicability

» Horizontal scaling

» Multi-Master

» Data representation
 » search of simplicity
 » data that does...
Tool selection
» be careful with the marketeese:
 smoke and mirrors beware!
» monitor dev list, IRC, Twitter, blogs

» mon...
Our Context: Lily



» cloud-scalable content store and search
 repository
» successor (in many ways) of Daisy




       ...
Complexity
complexity




                            software architecture
                                              ...
Complexity
 complexity
user interest




                                                                         3.0




...
Business Development 101
user interest




                                                                               ...
Solution
sophistication
ability to cope




                                                                       3.0
   ...
We Prefer Sophistication




» the challenge for us was to scale ...
 without dropping features




       IIC » TECHNOLOG...
The typical CMS ‘architecture’




  database (+opt. filesystem) (+ opt. full-text indexes)


         IIC » TECHNOLOGIEPAR...
The typical CMS ‘architecture’




  application                                  cache

  database (+opt. filesystem) (+ o...
The typical CMS ‘architecture’




  more cache

  application                                  cache

  database (+opt. fi...
The typical CMS ‘architecture’


  even more cache

  more cache

  application                                  cache

  ...
The typical CMS ‘architecture’

  client

  even more cache

  more cache

  application                                  ...
The typical CMS ‘architecture’

  client (+cache)

  even more cache

  more cache

  application                         ...
What we found hard to scale
» access control

» facet browsing

» all the nifty stuff people were using our
 software for
...
Beyond the ‘scaling’ problem
» three-prong data layer



                                                                 ...
Requirements, phase I
» automatic scaling to large data sets

» fault-tolerance: replication, automatic handling of failin...
Requirements, phase II

» After careful consideration, we realized the
 important choices were also:
 » consistency: no ch...
That brought us to HBase, which bought us:
» a datamodel where you can have column
 families which keep all versions and o...
» OK, so now we had a data store !




        IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org  ...
» However, content repository =
 store + search                          !
                                    u ch
      ...
a s
                                                        w
                                                      at !
 ...
Search ponderings

» CMS = two types of search
 » structured search
  » numbers, strings
  » based on logic         (SQL, ...
Search ponderings




» All of that, at scale




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outertho...
Structured Search
» HBase Indexing Library
 » idea from Google App Engine datastore indexes
 » http://code.google.com/appe...
Full-text / IR search


» Lucene?
 » no sharding (for scale)
 » no replication (for availability)
 » batched index updates...
Beyond Lucene
» Katta
  » scalable architecture, however only search, no indexing

» Elastic Search
  » very young (sorry)...
?
                             +
                         =
                                                r ?
          ...
Remember distribution ?
Remember secondary indexes ?




 ➙ Need for reliable queuing

    IIC » TECHNOLOGIEPARK 3 » B-905...
Connecting things
» we needed a reliable bridge between our
 main storage (HBase) and our index/search
 server(s) (SOLR)
 ...
Solution


» ACMEMessageQueue ? Bzzzzzt.
 We wanted fault-safe HBase persistence for
 the queues.
 Also for ease of admini...
WAL / Queue
» WAL                                                 » Queue
 » guaranteed execution                         ...
The Sum
» Lily model (records & fields)

» mapped onto HBase (=storage)

» indexed and searchable through
 SOLR
» using a W...
Architecture
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   131
Architecture
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   132
ere!
                                      th
                           ea rly
Roadmap                   N


» June 7-8: ...
bit.ly/lilyprerelease




       IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   134
License




» Apache




      IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org   135
Business model
» Consulting, mentoring, turn-key projects
 » audience: developers

» Strong focus on partner relations
 » ...
Reading material

» Amazon Dynamo, Google BigTable, CAP

» http://nosql.mypopescu.com/

» http://nosql-database.org/

» ht...
Questions?




                                                                  http://www.flickr.com/photos/leehaywood/42...
Thanks for your
                                    attention !




                                » stevenn@outerthought...
Upcoming SlideShare
Loading in...5
×

NoSQL with Hadoop and HBase

9,382

Published on

NoSQL overview presentation for BeJUG - 17/6/2010

Published in: Technology
0 Comments
21 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,382
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
611
Comments
0
Likes
21
Embeds 0
No embeds

No notes for slide

Transcript of "NoSQL with Hadoop and HBase"

  1. 1. NoSQL with HBase and Hadoop BeJUG - 17/6/2010 http://www.flickr.com/photos/wolfgangstaudt/2215246206/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  2. 2. Who am I » Steven Noels - stevenn@outerthought.org » Outerthought : scalable content applications » makers of Daisy, Lily and Kauri : open source internet/Java/REST/content apps IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 2
  3. 3. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  4. 4. An evolution driven by pain. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  5. 5. History 2. simplification 1. standardization hierarchical databases IMS XMLDB RDBMS OODBMS IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 5
  6. 6. History 4. rethinking the problem RDBMS NOSQL caching denormalisation sharding replication ... 3. pain IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 6
  7. 7. Four Trends » Trend 1 : Data Size » Trend 2 : Connectedness » Trend 3 : Semi-structure » Trend 4 : Architecture IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 7
  8. 8. Trend 1: Data size ExaBytes (10!") of data stored per year 988 1000 Each year more and more digital data is created. Over t wo 750 years we create more digital data than all 623 the data created in history before that. 500 397 253 250 161 0 2006 2007 2008 2009 2010 Data source: IDC 2007 3 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 8
  9. 9. Trend 2: Connectedness Giant Global Graph (GGG) Over time data has evolved to Ontologies be more and more interlinked and connected. RDF Hypertext has links, Blogs have pingback, Tagging groups all related data Folksonomies Information connectivity Tagging Wikis User-generated content Blogs RSS Hypertext Text documents web 1.0 web 2.0 “web 3.0” 1990 2000 2010 2020 4 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 9
  10. 10. Trend 3: Semi-structure ! Individualization of content • In the salary lists of the 1970s, all elements had exactly one job • In Or 15? lists of the 2000s, we need 5 job columns! Or 8? the salary ! All encompassing “entire world views” • Store more data about each entity ! Trend accelerated by the decentralization of content generation that is the hallmark of the age of participation (“web 2.0”) 5 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 10
  11. 11. Trend 4: Architecture 1980s: Mainframe applications Application DB 6 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 11
  12. 12. Trend 4: Architecture 1990s: Database as integration hub Application Application Application DB 7 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 12
  13. 13. Trend 4: Architecture 2000s: (moving towards) Decoupled services with their own backend Application Application Application DB DB DB 8 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 13
  14. 14. http://bigdatamatters.com/bigdatamatters/2010/04/high-availability-with-oracle.html IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 14
  15. 15. Enter NoSQL IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  16. 16. It’s a Cambrian Explosion Cassandra NoSQL neo4j IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 16
  17. 17. Buzz-oriented development ? IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 17
  18. 18. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 18
  19. 19. Common themes » SCALE SCALE SCALE » new datamodels » devops » N-O-SQL » The Cloud : technology is of no interest anymore IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 19
  20. 20. New Data » sparse structures » weak schemas » graphs » semi-structured » document-oriented IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 20
  21. 21. NoSQL » Not a movement. » Not ANSI NoSQL-2010. » Not one-size-fits-all. » Not (necessarily) anti-RDBMS. » No silver bullet. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 21
  22. 22. NoSQL = pro Choice IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 22
  23. 23. NoSQL = toolbox IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 23
  24. 24. NOSQL, if you need ... » horizontal scaling (out rather than up) » unusually common data (aka free-structured) » speed (especially for writes) » the bleeding edge IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 24
  25. 25. SQL/RDBMS, if you need ... » SQL » ACID » normalisation » a defined liability IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25
  26. 26. Some RDBMS bashing » sparse and dynamic tables IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 26
  27. 27. Some RDBMS bashing » solution mysql> desc thefields; +---------------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------------+---------------+------+-----+---------+-------+ | doc_id | bigint(20) | NO | PRI | NULL | | ... | fieldtype_id | bigint(20) | NO | PRI | NULL | | ... | stringvalue | varchar(255) | YES | MUL | NULL | | | datevalue | datetime | YES | MUL | NULL | | | datetimevalue | datetime | YES | MUL | NULL | | | integervalue | bigint(20) | YES | MUL | NULL | | | floatvalue | double | YES | MUL | NULL | | | decimalvalue | decimal(10,5) | YES | MUL | NULL | | | booleanvalue | char(1) | YES | MUL | NULL | | ... +---------------------+---------------+------+-----+---------+-------+ 25 rows in set (0.00 sec) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 27
  28. 28. More RDBMS bashing » replication and failure recovery » (when working on a budget) » application-level partitioning logic IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 28
  29. 29. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  30. 30. Academic background » Amazon Dynamo » Google BigTable » Eric Brewer CAP theorem IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 30
  31. 31. Shameless plug nosqlsummer.org IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 31
  32. 32. Shameless plug nosqlsummer.org IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 31
  33. 33. Amazon Dynamo » coined the term ‘eventual consistency’ » consistent hashing IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 32
  34. 34. Eventual Consistency Gone Wild IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 33
  35. 35. 1. replica server 1 2. server 2 1. update ACL: disallow mother from folder ‘spring break’ 2. upload spring break pictures how is my boy doing on his spring break? IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 34
  36. 36. Consistent hashing » a solution for naive mod n distributions » specifically in the case of adding or deleting nodes IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 35
  37. 37. Consistent hashing N value 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2 (c) Basho/Riak 5 Tuesday, November 17, 2009 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 36
  38. 38. Google BigTable » multi-dimensional column-oriented database » on top of GoogleFileSystem » object versioning IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 37
  39. 39. CAP theorem strong high consistency availability partition- tolerance IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 38
  40. 40. CAP » Strong Consistency: all clients see the same view, even in the presence of updates » High Availability: all clients can find some replica of the data, even in the presence of failures » Partition-tolerance: the system properties hold even when the system is partitioned IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 39
  41. 41. Culture Clash » ACID » BASE » highest priority: strong » availability and scaling consistency for highest priorities transactions » weak consistency » availability less important » optimistic » pessimistic » best effort » rigorous analysis » simple and fast » complex mechanisms spectrum IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 40
  42. 42. Availability ≠ total async ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  43. 43. ✘ The Enterprise Service Bus bus = congestion IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 42
  44. 44. Bus systems » objects don’t fit in a pipe » object ➙ message » serialization / de-serialization cost » message size » queuing = cost IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 43
  45. 45. Use a mixture of both »async + sync stuff which matters ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 44
  46. 46. 2.1 Interlude IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  47. 47. 2.1 Interlude IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  48. 48. 2.1 Interlude IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  49. 49. Processing large datasets : Hadoop + Map/Reduce IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  50. 50. Hadoop: HDFS + MapReduce » single filesystem + single execution-space IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 47
  51. 51. M/R Execution (c) Google IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 48
  52. 52. MapReduce example: WordCount IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 49
  53. 53. Hadoop ecosystem » Hadoop Common » Subprojects » Chukwa: A data collection system for managing large distributed systems. » HBase: A scalable, distributed database that supports structured data storage for large tables. » HDFS: A distributed file system that provides high throughput access to application data. » Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying. » MapReduce: A software framework for distributed processing of large data sets on compute clusters. » Pig: A high-level data-flow language and execution framework for parallel computation. » ZooKeeper: A high-performance coordination service for distributed applications. » Mahout: machine learning libraries IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 50
  54. 54. Processing large datasets with MR » Benefit from parallellisation » Less modelling upfront (ad-hoc processing) » Compartmentalized approach reduces operational risks » AsterData et al. have SQL/MR hybrids for huge-scale BI IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 51
  55. 55. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  56. 56. We welcome the Polyglot Persistence overlords. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  57. 57. The NOSQL footprint free-structured or sparse data NOSQL MongoDB CouchDB neo4j Cassandra available (complexity) simple operational HBase highly scalable and constraints ACID, SQL referential integrity, typed data (c) me! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 54
  58. 58. Categories » key-value stores » column stores » document stores » graph databases IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 55
  59. 59. Key-value stores » Focus on scaling to huge amounts of data » Designed to handle big loads » Often: cfr. Amazon Dynamo » ring partitioning and replication » Data model: key/value pairs IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 56
  60. 60. Key-value stores » Redis » Voldemort » Tokyo Cabinet IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 57
  61. 61. Redis » REmote DIctionary Server » http://code.google.com/p/redis/ » vmware IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 58
  62. 62. Redis Features » persisted memcache, ‘awesome’ » RAM-based + persistable » key ➙ values: string, list, set » higher-level ops » i.e. push/pop and sort for lists » fast (very) » configurable durability » client-managed sharding IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 59
  63. 63. Voldemort » http://project-voldemort.com/ » LinkedIn IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 60
  64. 64. Voldemort » persistent » distributed » fault-tolerant » hash table IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 61
  65. 65. Voldemort API: GET, PUT, DELETE IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 62
  66. 66. Voldemort routing logic moving up the stack, smaller latency IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 63
  67. 67. Column stores » BigTable clones » Sparseness! » Data model: columns ➙ column families ➙ cells » Datums keyed by: row, column, time, index » Row-range ➙ tablet ➙ distribution IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 64
  68. 68. Column stores » BigTable » HBase » Cassandra IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 65
  69. 69. BigTable » http://labs.google.com/papers/bigtable.html » Google » layered on top of GFS IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 66
  70. 70. HBase » http://hadoop.apache.org/hbase/ » StumbleUpon / Adobe / Cloudera IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 67
  71. 71. HBase » sorted » persisted » distributed » storage system » column-oriented » multi-dimensional » highly-available » adds random access » high-performance reads and writes atop HDFS IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 68
  72. 72. HBase data model » Distributed multi-dimensional sparse map » Multi-dimensional keys: (table, row, family:column, timestamp) → value » Keys are arbitrary strings » Access to row data is atomic IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 69
  73. 73. Sample schema !"#$%&'(%)*+#,-% )#&*+,-./%& 0122345322!12 )#&*+,-./%& 0122345322!14 !"#$$%&'( (c) eCircle IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 70
  74. 74. Storage architecture © lars george IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 71
  75. 75. Cassandra » http://cassandra.apache.org/ » Rackspace / Facebook IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 72
  76. 76. Cassandra » Key-value store (with added structure) » Reliability (identical nodes) » Eventual consistent » Distributed A C » Tunable » Partitioning P » Replication IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 73
  77. 77. Cassandra applicability FIT NO FIT » Scalable reliability » Flexible indexing (through identical » Only PK-based nodes) querying » Linear scaling » Big Binary Data » Write throughput » 1 Row must fit in » Large Data Sets RAM entirely IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 74
  78. 78. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 75
  79. 79. IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 76
  80. 80. Document databases » ≈ K/V stores, but DB knows what the Value is » Lotus Notes heritage » Data model: collections of K/V collections » Documents often versioned IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 77
  81. 81. Document stores » CouchDB » MongoDB » Riak IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 78
  82. 82. CouchDB » http://couchdb.apache.org/ » couch.io IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 79
  83. 83. CouchDB » fault-tolerant » schema-free » document-oriented » accessible via a RESTful HTTP/JSON API IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 80
  84. 84. CouchDB documents { “_id”: ”BCCD12CBB”, “_rev”: ”AB764C”, “type”: ”person”, “name”: ”Darth Vader”, “age”: 63, “headware”: [“Helmet”, “Sombrero”], “dark_side”: true } IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 81
  85. 85. CouchDB REST API » HTTP » PUT /db/docid » GET /db/docid » POST /db/docid » DELETE /db/docid IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 82
  86. 86. CouchDB Views » MapReduce-based » Filter, Collate, Aggregate » Javascript map reduce function (doc) { function (Key, Values) { for(var i in doc.tags) var sum = 0; emit(doc.tags[i], 1); for(var i in Values) } sum += Values[i]; return sum; } IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 83
  87. 87. CouchDB » be careful on semantics » replication ≠ partioning/sharding ! » distributed database = distributable database » sharded / distributed deployment requires proxy layer IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 84
  88. 88. MongoDB » http://www.mongodb.org/ » 10gen IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 85
  89. 89. MongoDB » cfr. CouchDB, really » except for: » C++ » performance focus » runtime queries (mapreduce still available) » native drivers (no REST/HTTP layering) » no MVCC: update-in-place » auto sharding (alpha) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 86
  90. 90. Graph databases » Focus on modeling structure of data - interconnectivity » Scale, but only to the complexity of data » Data model: property graphs IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 87
  91. 91. Graph databases » Neo4j » AllegroGraph (RDF) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 88
  92. 92. Neo4j » http://neo4j.org/ » Neo Technology IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 89
  93. 93. Neo4j » data = nodes + relationships + key/value properties IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 90
  94. 94. Neo4j » many language bindings, little remoting » ‘whiteboard’ friendly » scaling to complexity (rather than volume?) » lots of focus on domain modelling » SPARQL/SAIL impl for triple geeks » mostly RAM centric (with disk swapping & persistence) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 91
  95. 95. Bandwagonjumpers » JCR / Jackrabbit » GT/M » RDF stores IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 92
  96. 96. Market maturization IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  97. 97. Rise of integrators » Cloudera (H-stack) » Riptano (Cassandra) » Cloudant (hosted CouchDB) » (Outerthought: HBase) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 94
  98. 98. VC capital » Cloudera » couch.io » Neo » 10gen » many others IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 95
  99. 99. 1. Intro 2. Theory 3. Technology 4. Experiences IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
  100. 100. the fireside conversations http://www.flickr.com/photos/52641994@N00/516394238/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 97
  101. 101. NOSQL applicability » Horizontal scaling » Multi-Master » Data representation » search of simplicity » data that doesn’t fit the E-R model (graphs, trees, versions) » Speed IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 98
  102. 102. Tool selection » be careful with the marketeese: smoke and mirrors beware! » monitor dev list, IRC, Twitter, blogs » monitor project ‘sponsors’ » mix-and-match: polyglot persistency » DON’T NOSQL WITHOUT INTERNAL SYS ARCHS & DEV(OP)S ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 99
  103. 103. Our Context: Lily » cloud-scalable content store and search repository » successor (in many ways) of Daisy IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 100
  104. 104. Complexity complexity software architecture 3.0 2.0 1.0 age IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 101
  105. 105. Complexity complexity user interest 3.0 2.0 1.0 age IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 102
  106. 106. Business Development 101 user interest budget IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 103
  107. 107. Solution sophistication ability to cope 3.0 mysql nosql? 2.0 1.0 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 104
  108. 108. We Prefer Sophistication » the challenge for us was to scale ... without dropping features IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 105
  109. 109. The typical CMS ‘architecture’ database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 106
  110. 110. The typical CMS ‘architecture’ application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 107
  111. 111. The typical CMS ‘architecture’ more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 108
  112. 112. The typical CMS ‘architecture’ even more cache more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 109
  113. 113. The typical CMS ‘architecture’ client even more cache more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 110
  114. 114. The typical CMS ‘architecture’ client (+cache) even more cache more cache application cache database (+opt. filesystem) (+ opt. full-text indexes) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 111
  115. 115. What we found hard to scale » access control » facet browsing » all the nifty stuff people were using our software for » ... anything that required random access to in-memory-cache data for computations IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 112
  116. 116. Beyond the ‘scaling’ problem » three-prong data layer fs » result set merging (between MySQL & Lucene) » happened in appcode/memory » ‘transactions’, set operations = hard IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 113
  117. 117. Requirements, phase I » automatic scaling to large data sets » fault-tolerance: replication, automatic handling of failing nodes » a flexible data model supporting sparse data » runs on commodity hardware » efficient random access to data » open source, ability to participate in the development thus drive the direction of the project » some preference for a Java-based solution IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 114
  118. 118. Requirements, phase II » After careful consideration, we realized the important choices were also: » consistency: no chance of having two conflicting versions of a row » atomic updates of a single row, single-row transactions » bonus points for MapReduce integration » e.g. full-text index rebuilding IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 115
  119. 119. That brought us to HBase, which bought us: » a datamodel where you can have column families which keep all versions and others which do not, which fits very well on our CMS document model » ordered tables with the ability to do range scans on them, which allows to build scalable indexes on top of it » HDFS, a convenient place to store large blobs » Apache license and community, a familiar environment for us IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 116
  120. 120. » OK, so now we had a data store ! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 117
  121. 121. » However, content repository = store + search ! u ch o IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 118
  122. 122. a s w at ! h sy T a .. .) e ver we h o ( IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 119
  123. 123. Search ponderings » CMS = two types of search » structured search » numbers, strings » based on logic (SQL, anyone?) » information retrieval (or: full-text search) » text » based on statistics IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 120
  124. 124. Search ponderings » All of that, at scale IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 121
  125. 125. Structured Search » HBase Indexing Library » idea from Google App Engine datastore indexes » http://code.google.com/appengine/articles/ index_building.html rowkey col col rowkey col order A val3 foo6 val2-B B val2 foo7 val3-A content table index table A IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 122
  126. 126. Full-text / IR search » Lucene? » no sharding (for scale) » no replication (for availability) » batched index updates (not real-time) IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 123
  127. 127. Beyond Lucene » Katta » scalable architecture, however only search, no indexing » Elastic Search » very young (sorry) » hbasene et al. » stores inverted index in HBase, might not scale all features » SOLR » widely used, schema, facets, query syntax, cloud branch More info: http://lilycms.org/lily/prerelease/technology.html IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 124
  128. 128. ? + = r ? ! O a sy E IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 125
  129. 129. Remember distribution ? Remember secondary indexes ? ➙ Need for reliable queuing IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 126
  130. 130. Connecting things » we needed a reliable bridge between our main storage (HBase) and our index/search server(s) (SOLR) » indexing, reindexing, mass reindexing (M/R) » we need a reliable method of updating HBase secondary indexes » all of that eventually to run distributed » distribution means coping with failure IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 127
  131. 131. Solution » ACMEMessageQueue ? Bzzzzzt. We wanted fault-safe HBase persistence for the queues. Also for ease of administration. » ➙ WAL & Queue implemented on top of HBase tables IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 128
  132. 132. WAL / Queue » WAL » Queue » guaranteed execution » triggering of async of synchronous actions actions » call doesn’t return before » e.g. (re)index (updated) secondary action finishes record with SOLR back-end » e.g. update secondary actions » size depends on speed of » if all goes well, back-end process size = #concurrent ops » will be useful/made available outside of Lily context as well! IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 129
  133. 133. The Sum » Lily model (records & fields) » mapped onto HBase (=storage) » indexed and searchable through SOLR » using a WAL/Queue mechanism implemented in HBase » runtime based on Kauri » with client/server comms via Avro IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 130
  134. 134. Architecture IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 131
  135. 135. Architecture IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 132
  136. 136. ere! th ea rly Roadmap N » June 7-8: release of learning material (architecture, model, API, Javadoc) ➥ www.lilycms.org ➥ bit.ly/lilyprerelease » Tomorrow: WAL/queue » Mid July = ‘proof of architecture’ release » from there on, ca. 3-monthly releases leading up to Lily 1.0 IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 133
  137. 137. bit.ly/lilyprerelease IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 134
  138. 138. License » Apache IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 135
  139. 139. Business model » Consulting, mentoring, turn-key projects » audience: developers » Strong focus on partner relations » targeting vertical markets » geographic coverage » SaaS offerings » Markets: media, finance, insurance, govt, heritage ... LOTS of semi-structured data » Not: OLAP IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 136
  140. 140. Reading material » Amazon Dynamo, Google BigTable, CAP » http://nosql.mypopescu.com/ » http://nosql-database.org/ » http://twitter.com/nosqlupdate » http://highscalability.com/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 137
  141. 141. Questions? http://www.flickr.com/photos/leehaywood/4237636853/ IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 138
  142. 142. Thanks for your attention ! » stevenn@outerthought.org » @stevenn IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 139
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×