Lily at HUG UK

Lily
Smart data at scale

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

big data,
big problems


MOORE vs data

data

» coping with volume + need for
timeliness = parallel processing moore
» data becomes business-critical =
resilience through distributed
architectures
» Hadoop, MapReduce, HBase:
the future data platform

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 3

the CHALLENGES

» process ALL data
» process data in REAL-TIME
» derive INSIGHTS
» provide INSTANT FEEDBACK


current thinking

ETL

data
data STORE analytics
warehouse

batched, off-line, overnight

1. store and manage all YOUR data

DATA


2. store user behaviour, nearby

DATA

USER
Behavior


3. analyze usage patterns

DATA data processing

USER
Behavior


4. add domain knowledge


USER
Behavior

domain
knowledge
patterns
rules
keywords
lists
...


5. process, in real-time


recommendations
semantic augmentation
Analytics

USER
Behavior

domain
knowledge
patterns
rules
keywords
lists
...


6. augment data


recommendations
Analytics

USER
Behavior

domain
knowledge
patterns
rules
keywords
lists
...


data insights
SMARTER DATA data processing
s
relation
recommendations
Analytics

domain
knowledge
patterns
rules
keywords
lists
...


data insights
SMARTER DATA data processing
s
relation
recommendations
Analytics

domain
knowledge
patterns
rules
keywords
lists

SMART DATA, at SCALE
...

... and in real time

stories


HYPER-PERSONAL
recommendations
NEWS

TOGETHERNESS
interestingness

organisations
names
locations
brands

news aggregator
scale

up-selling
CROSS-SELLING
product
CATALOG
recommendedness
relatedness

product
families
related
activities
social graph

e-retail
real-time

competitive
innovation
patents

(dis)SIMILARITY

companies
people
materials
processes

IP research
insights

Outerthought
» software product company
» scalable content applications
» open source product portfolio
» Java, REST, internet

THIS NOTEBOOK BELONGS TO:

“The world is moving
Noteblock_03.indd 1 23/05/10 14:42

from content as a cost to
data as an opportunity.”


Lily 1.0 (CR)

data
data STORE + warehouse + analytics

real time
}
Lily 2.0

Lily (now)

» Large-scale content storage, indexing and search
» Current pilots

e-retail mobile media isp e-gov ip research

» up-to now: 4 man-years investment (since Sept/2009)


roadmap

» now: Lily 0.3
» Along the road:
» april 2011 : Lily 1.0
Lily SaaS edition
» Q3 2011
» real-time statistics + analytics
» Q2 2012 : Lily 2.0
» real-time data processing engine
» Data Insights


open source

» www.lilyproject.org
» docs.outerthought.org/lily-docs-current/


Lily Core Concepts
» storage
» HBase
» repository model
» versioning, varianting, mixins
» indexing
» mapping
» search
» SOLR


falling in love with Hbase : phase 1

» automatic scaling to large data sets
» fault-tolerance
» ﬂexible datamodel with sparse data
» commodity hardware
» efﬁcient random access
» community-based open source
» Java if possible



» need for consistency
» atomic single-row updates
» M/R for index regeneration



HBase
» datamodel with column families and cell versioning
» ordered tables with range scans
» HDFS for blob storage
» Apache


Lily Repository Model


Lily Datatypes


Mixins


Sample Lily Schema (excerpt)
  {
namespaces: {
    name: "b$name",
    /* Declaration of namespace prefixes. */
    valueType: { primitive: "STRING" },
    "org.lilyproject.bookssample": "b",
    scope: "versioned"
    "org.lilyproject.vtag": "vtag"
  },
  },
  {
fieldTypes: [
    name: "b$bio",
  {
    name: "b$title",
  },
  {
  },
    name: "vtag$last",
  {
    valueType: { primitive: "LONG" },
    name: "b$pages",
    scope: "non_versioned"
    valueType: { primitive: "INTEGER" },
  }
  ],
  },
recordTypes: [
  {
  {
    name: "b$language",
    name: "b$Book",
    fields: [
      {name: "b$title", mandatory: true },
  },
      {name: "b$pages", mandatory: false },
  {
      {name: "b$language", mandatory: false },
    name: "b$authors",
      {name: "b$authors", mandatory: false },
    valueType: { primitive: "LINK", multiValue: true },
      {name: "vtag$last", mandatory: false }
    ]
  },
  },

...


Lily Versioning


Flexible content model
» generic enough to accomodate many popular content
schemas
» HTML5, CMIS, RDF, NewsML, Dublin Core, ...
» academically veriﬁed
» not limited to ‘content applications’ only
» developer convenience
» higher level constructs
» schema reuse
» versioning, linking, ...


Lily Architecture
(deployment)


Lily Architecture
(components)


HBase RowLog Library

» need for sync/async operations
» updating of secondary indexes (i.e. tables)
» feeding of Indexer (= bridge to SOLR index maintenance)
» not: transactions
» need for distribution and durability


HBase RowLog Library
» WAL » Queue
» guaranteed execution of synchronous » triggering of async actions
actions
» e.g. (re)index (updated) record with
» call doesn’t return before secondary
SOLR back-end
action ﬁnishes
» size depends on speed of back-end
» e.g. update secondary index tables
process
» if all goes well,
size = #concurrent ops
» useful outside of Lily context as well!


The Lily Indexer

sharding towards
indexing of multiple incremental index blob content
denormalization batch index building multiple SOLR
versions of a record updating extraction
instances


Indexing configuration (SOLR)
<schema name="example" version="1.2">

<types>
[snipped: see SOLR example schema]
</types>

<fields>

<field name="@@key" type="string" indexed="true" stored="true" required="true"/>
<field name="@@id" type="string" indexed="true" stored="true" required="true"/>
<field name="@@vtag" type="string" indexed="true" stored="true" required="true"/>
<field name="@@versionless" type="string" indexed="true" stored="true" required="false"/>


<field name="title" type="text" indexed="true" stored="true" required="false"/>
<field name="authors" type="text" indexed="true" stored="true" required="false"
multiValued="true"/>
</fields>

<uniqueKey>@@key</uniqueKey>

<defaultSearchField>title</defaultSearchField>

<solrQueryParser defaultOperator="OR"/>

</schema>

Indexer configuration (Lily)
<?xml version="1.0"?>
<indexer xmlns:b="org.lilyproject.bookssample">
<cases>
<case recordType="b:Book" variant="*" vtags="last" indexVersionless="true"/>
</cases>

<indexFields>
<indexField name="title">
<value>
<field name="b:title"/>
</value>
</indexField>

<indexField name="authors">
<value>
<deref>
<follow field="b:authors"/>
<field name="b:name"/>
</deref>
</value>
</indexField>
</indexFields>

</indexer>


(opt.) Sharding configuration
{
shardingKey: {
    value: {
      source: "variantProperty",
      property: "language"
    },
    type: "string"
},

mapping: {
    type: "list",
    entries: [
      { shard: "shard1", values: ["en", "it"] },
      { shard: "shard2", values: ["nl", "de", "es"] }
    ]
}
}


Lily API

» Java (using Avro)
» http://docs.outerthought.org/lily-docs-current/g3/g1/390-lily.html

» REST (HTTP + JSON)
» http://docs.outerthought.org/lily-docs-current/g3/g2/427-lily.html

» All docs
» http://docs.outerthought.org/lily-docs-current/ext/toc/


Demo
» http://outerthought.blip.tv/ﬁle/4245615/


Lily and HBase

» adds high-level content model
» data types
» versioning
» blob storage on HDFS
» focus on sparse (efﬁcient) storage
» RowLog for synchronous cross-table updates and async
message queues


Lily and SOLR

» provides flexible mapping between HBase content
model and SOLR index fields
» interactive and batch (M/R) index maintenance
» sharding
» use(s) SOLR as-is: loose, flexible, extensible coupling
» search access via SOLR (HTTP) API


Lily and CDH

» we intend to rely on CDH-‘blessed’ versions of HBase/
HDFS/ZK
» 700 patches and testing
» next: adopting similar distribution lay-out
» since we contribute patches to ASF HBase trunk, we would
expect CDH to track closely (until HBase 1.0)
» some Lily users could be interested in ‘CDH-level’ services


goodbye

» It’s open source !
» Content Repository: available now
(Lily model + HBase + SOLR + RowLog)
» Lily 1.0 soon, will mainly focus on differentiating open
source and enterprise edition
» “HBase is wa de max maat.”


Thank you !
for your attention
for your questions

» stevenn@outerthought.org

» @stevenn


Lily at HUG UK

Recommended

Recommended

More Related Content

Similar to Lily at HUG UK

Similar to Lily at HUG UK (20)

More from NGDATA

More from NGDATA (11)

Recently uploaded

Recently uploaded (20)

Lily at HUG UK