Lily for the Bay Area HBase UG - NYC edition

Presenting Lily
Bay Area HBase UG - NYC - 10/11/2010

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

Devoxx: Nov. 15-19, Antwerp, Belgium
NoSQL/Cloud track

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 2

Outerthought

» software product company

» scalable content applications

» open source product portfolio

» Java, REST, internet
THIS NOTEBOOK BELONGS TO:

Noteblock_03.indd 1 23/05/10 14:42


Technology

»Lily : NoSQL-based content
repository (HBaseN OTESOLR) N GS TO:
THIS
+ B OOK B ELO
» Kauri : REST centric webapp dev framework
» Daisy : techdoc / QDoc / publishing CMS


Needs for Scalable Content
» wire-speed capturing ➡ NoSQL & write-
optimized storage
» batch-oriented post-
processing ➡ map/reduce

» semantic lifting : ➡ Natural Language
extracting knowledge Processing
out of noise
» data and inferred data ➡ smart content
become one repositories


customers

The Lily Project

REST-centric content
cloud-scale content applications

batch
} partners

}
alternative processing and
content app UI augmentation ins and outs
indexes process
framework (enrichment)
coordination

us
content repository: store + search


Lily essentials

» www.lilyproject.org

» Apache license for maximal ﬂexibility

» (lots of) documentation at
docs.outerthought.org


Lily content repository

» Scalable store (HBase) and
search (SOLR)
content
» ﬂexible content model application
» index maintenance
repository

» high-level API

» base foundation


HBase
» a datamodel where you can have column
families which keep all versions and others
which do not, which ﬁts very well on our
CMS document model
» ordered tables with the ability to do range
scans on them, which allows to build
scalable indexes on top of it
» HDFS, a convenient place to store large blobs

» Apache license and community, a familiar
environment for us


1. Store, 2. Search...? Ouch.

» CMS = two types of search
» structured, ‘logic’ search
» numbers, strings
» based on logic (SQL, anyone?)

» information retrieval (or: full-text search)
» text
» based on statistics


Search ponderings

» All of that, at scale


Structured Search
» HBase Indexing Library
» idea from Google App Engine datastore indexes
» http://code.google.com/appengine/articles/
index_building.html

rowkey col col rowkey col

order
A val3 foo6 val2-B

B val2 foo7 val3-A

content table index table A


Full-text / IR search

» Lucene?
» no sharding (for scale)
» no replication (for availability)
» batched index updates (not real-time)


Beyond Lucene
» Katta
» scalable architecture, however only search, no indexing

» Elastic Search
» very young (sorry)

» hbasene et al.
» stores inverted index in HBase, does not scale all features

» SOLR
» widely used, schema, facets, query syntax, cloud branch


?
+
=
r ?
! O
asy
E

➙ Need for reliable queuing


Connecting things
» we needed a reliable bridge between our
main storage (HBase) and our index/search
server(s) (SOLR)
» indexing, reindexing, mass reindexing (M/R)

» we need a reliable method of updating
HBase secondary indexes
» all of that eventually to run distributed

» distribution means coping with failure


Solution

» ... a QUEUE ! (Meh)

» ACMEMessageQueue ? Bzzzzzt.
We wanted fault-safe HBase persistence for
the queues.
Also for ease of administration.
» ➙ WAL & Queue implemented on top of
HBase tables


WAL & Queue = RowLog Library
» WAL » Queue
» guaranteed execution » triggering of async
of synchronous actions actions
» call doesn’t return before » e.g. (re)index (updated)
secondary action ﬁnishes record with SOLR back-end
» e.g. update secondary indexes » size depends on speed of
» if all goes well, back-end process
size = #concurrent ops
» useful outside of Lily context
as well!


The Sum
» Lily model (records & ﬁelds)

» mapped onto HBase (=storage)

» indexed and searchable through
SOLR
» using a WAL/Queue mechanism
implemented in HBase
» runtime based on Kauri

» with client/server comms via Avro
(and a REST interface with JSON)


Architecture

Lily roadmap
» development started Sept. 2009
» development trunk opened Jul. 2010

» end of Oct. 2010: milestone/beta release
» fully distributable
» spec-complete
» Onwards:
» ‘business-level’ 1.0 release (packaging, testing, performance)
» user/auth management & access control
» UI framework (Kauri)
» ins and outs, semantic lifting


Thanks for your
hospitality and
attention !

THIS NOTEBOOK BELONGS TO:
» stevenn@outerthought.org

Noteblock_03.indd 1 23/05/10 14:42
» @stevenn


Lily for the Bay Area HBase UG - NYC edition

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Lily for the Bay Area HBase UG - NYC edition

Similar to Lily for the Bay Area HBase UG - NYC edition (20)

More from NGDATA

More from NGDATA (8)

Recently uploaded

Recently uploaded (20)

Lily for the Bay Area HBase UG - NYC edition