Hugaccumulo 121018192044-phpapp02

sqrrl data, INC.
Secure.
Scale.
Adapt.

info@sqrrl.com | @sqrrl_inc | 617.520.4375 sqrrl data, INC., All Rights Reserved
Adam
Fuchs,
Chief
Technology
Oﬃcer

Who We are
2

Secure.

Scale.

Adapt.

is
the
commercial

provider
of

!   Mature
Database
Technology
-‐
Apache
Accumulo

!   Fine-‐Grained
Access
Controls
-‐
Data
Integra7on
and
Sharing

!   Proven
Performance
-‐
Petabytes
and
Beyond

!   Advanced
AnalyBcs
-‐
Search,
Sta7s7cs,
and
Graphs

Contents
Core
Philosophy

Technology

Techniques

ApplicaBon
APIs

3

Secure.

Scale.

Adapt.


Integra7on
across:

!   Mul7ple
business
lines

!   Mul7ple
data
sets

!   Mul7ple
applica7ons

!   Mul7ple
security,
privacy,
legal,

policy,
regulatory,
and

compliance
constraints

!   New
demands

Apache Accumulo Perspective
Applica7on

Data
Data
Data

Applica7on
Applica7on

4

Secure.

Scale.

Adapt.


Accumulo Design Drivers
Scalability

„  Near
linear
performance
improvements
at
thousands
of
nodes

„  Durable
and
reliable
under
increased
failures
that
come
with
scale

2
Diverse,
InteracBve
AnalyBcs

„  Sorted
key/value
core
performs
well
in
a
diverse
set
of
domains

„  Informa7on
retrieval,
sta7s7cs,
graph
analysis,
geo
indexing,
and
more

3
Cell-‐Level
Security

„  Express
common
security
requirements
in
the
infrastructure,
not
just
in
the
applica7on

„  Data-‐centric
approach
encourages
secure
sharing

1
5

Secure.

Scale.

Adapt.

Flexible,
AdapBve
Schema

„  Start
with
universal
structures
and
indexing

„  Reﬁne
the
schema
over
7me

4

Contents
Core
Philosophy

Technology

Techniques

ApplicaBon
APIs

6

Secure.

Scale.

Adapt.


Accumulo Key Structure
An
Accumulo
key
is
a
5-‐tuple,
consisBng
of:

!   Row:
Controls
Atomicity

!   Column
Family:
Controls
Locality

!   Column
Qualiﬁer:

Controls
Uniqueness

!   Visibility
Label:

Controls
Access

!   Timestamp:

Controls
Versioning

Row
Col.
Fam.
Col.
Qual.
Visibility
Timestamp
Value

John
Doe
Notes
PCP
PCP_JD
20120912

Pa7ent
suﬀers

from
an
acute
…

John
Doe
Test
Results
Cholesterol
JD|PCP_JD
20120912
183

John
Doe
Test
Results
Mental
Health
JD|PSYCH_JD
20120801
Pass

John
Doe
Test
Results
X-‐Ray
JD|PHYS_JD
20120513
1010110110100…

Accumulo
Key/Value
Example

7

Secure.

Scale.

Adapt.


Visibility Syntax & Semantics
8

Secure.

Scale.

Adapt.


Tablets
9

!   Collec7ons
of
KV
pairs
form
Tables

!   Tables
are
par77oned
into
Tablets

!   Metadata
tablets
hold
info
about

other
tablets,
forming
a
3-‐level

hierarchy

!   A
Tablet
is
a
unit
of
work
for
a
Tablet

Server

Root
Tablet

-‐∞
to
∞

Metadata
Tablet
1

-‐∞
to
“Encyclopedia:Ocelot”

Data
Tablet

-‐∞
:
thing

Data
Tablet

thing
:
∞

Data
Tablet

-‐∞
:
Ocelot

Data
Tablet

Ocelot
:
Yak

Data
Tablet

Yak
:
∞

Data
Tablet

-‐∞
to
∞

Metadata
Tablet
2

“Encyclopedia:Ocelot”
to
∞

Well-‐Known

Loca7on

(zookeeper)

Table:

Adam’s
Table
Table:

Encyclopedia
Table:

Foo

Secure.

Scale.

Adapt.


Accumulo Architecture
Tablet
Server

Tablet

Tablet
Server

Tablet

Tablet
Server

Tablet

Applica7on

Zookeeper

Zookeeper

Zookeeper

Master

Hadoop

Read/Write

Store/Replicate

Assign/Balance

Delegate

Authority

Delegate

Authority

Applica7on

Applica7on

10

Secure.

Scale.

Adapt.


Tablet Data Flow
In-‐Memory

Map

Write
Ahead

Log

(For
Recovery)

Sorted,

Indexed

File

Sorted,

Indexed

File

Sorted,

Indexed

File

Tablet

Reads

Iterator

Tree

Minor

Compac<on

Merging
/
Major

Compac<on

Iterator

Tree

Writes

11

Secure.

Scale.

Adapt.

Iterator

Tree

Scan


Iterator Framework
12

Secure.

Scale.

Adapt.

Iterator
OperaBons:

!   File
Reads

!   Block
Caching

!   Merging

!   Dele7on

!   Isola7on

!   Locality
Groups

!   Range
Selec7on

!   Column
Selec7on

!   Cell-‐level
Security

!   Versioning

!   Filtering

!   Aggrega7on

!   Par77oned
Joins


13

Testing Procedures
Frameworks

!   Unit:
Verify
correct
func7oning
of

each
module
separately

!   System:
Perform
correctness
and

performance
tests
on
a
small
running

instance

!   Load/Scale:
Generate
high
loads
at

scale,
measure
performance
and

correctness

!   Random
Walk:
Randomly,

repeatedly,
and
concurrently
execute

a
variety
of
test
modules

representa7ve
of
user
ac7vity

!   SimulaBon:
Evaluate
a
model
for

correctness
and
performance

TesBng
Complexity

!   Scope
of
test
may
include/exclude

server-‐side,
client-‐side,
dependent

processes,
etc.

!   Simula7ng
failures
of
distributed

components

!   Strange
failure
modes
(hardware/
physics
related)

!   Sta7c
vs.
dynamic
analysis

Secure.

Scale.

Adapt.


FATE: Fault-Tolerant Executor
14

!   Logically
Atomic
distributed
opera7ons

!   No
single
point
of
failure

!   Builds
on
quorum
consistency
model
of

Zookeeper

!   All-‐or-‐nothing
transac7onal
seman7cs
to
handle

failures
(opera7ons
must
be
idempotent
and

reversible)

!   Used
by
administra7ve
opera7ons
and
bulk

table
opera7ons
to
survive
our
toughest
tests

Secure.

Scale.

Adapt.


Verified State Models
15

Secure.

Scale.

Adapt.

Many
of
Accumulo’s
subsystems
use
models

that
are
mathema7cally
proven
correct.

Fate
States


Contents
Core
Philosophy

Technology

Techniques

ApplicaBon
APIs

16

Secure.

Scale.

Adapt.


Hierarchical Decomposition
17

Row:

Column
Family:

Column
Qualiﬁer:

Value:

<person>

aYribute
purchases
returns

age

<age>

discount

<cost>

hat

<cost>

sneakers

<40%>

Secure.

Scale.

Adapt.


Materialized Table
18

Row:

Column

Family:

Column

Qualiﬁer:

Value:

george

aYribute
purchases
returns

age

27
$83

hat

$42

sneakers

bill

aYribute
purchases

40%

sneakers

$100

discount

49

age

Secure.

Scale.

Adapt.

Key/Value
Pair


Forward and Inverted Index
19

Table:

Row:

Column
Family:

Column
Qualiﬁer:

Value:

Forward
Index

<UUID>

<Type>

<Field>

<Term>

Inverted
Index

<Term>

<Type>
+
<Field>

<UUID>

<Digest
of
Event>

Secure.

Scale.

Adapt.


Forward and Inverted Index
20

Secure.

Scale.

Adapt.


Graph Analysis
21

Table:

Row:

Column
Family:

Column
Qualiﬁer:

(Tuples):

Value:

Graph
Table

<Node
ID>

“Node
Info”
“Out
Edges”
“In
Edges”

<Field>

<Value>

<Node
ID>

<Edge
ID>

<Edge
Info>

<Node
ID>

<Edge
ID>

<Edge
Info>

Secure.

Scale.

Adapt.


Geospatial Queries
22

Table:

Row:

Column
Family:

Column
Qualiﬁer:

Value:

Geo
Index

<GeoHash>

<Event
Type>

<UUID>

<Digest
of
Event>

Secure.

Scale.

Adapt.

La7tude

10110101001

Longitude

00111010010

101001110111010101011100001011100

Depth

11010110110


Document Partitioning
23

Table:

Row:

Column
Family:

Column
Qualiﬁer

(Tuples):

Value:

Shard
Table

<ParBBon
ID>

“Docs”
“Inv.
Index”
“Field
Index”

<UUID>

<Value>

<Term>

<UUID>

<Field:Term>

<UUID>

Secure.

Scale.

Adapt.

<Field>

“Geo”

<Hash>

<UUID>


Document Partitioning
24

Secure.

Scale.

Adapt.


Intersecting Iterator
25

Secure.

Scale.

Adapt.


Intersecting Iterator
26

Secure.

Scale.

Adapt.

‘foo’
and
(‘bar’
or
‘baz’)

<ParBBon
ID>

“Docs”
“Inv.
Index”

<UUID>

<Value>

<Term>

<UUID>
<Field>


Contents
Core
Philosophy

Technology

Techniques

ApplicaBon
APIs

27

Secure.

Scale.

Adapt.


acorn
28

Key/Value
pairs
are
great!

How
do
I
construct
a
document

par<<oning
key
again?

!   Techniques
should
be
built
into
an
API

!   Let
the
people
have
polyglot

!   Lucene,
SQL,
SPARQL,
JAQL,
Matlab

(not
just
Key,
Value,
Range)

Secure.

Scale.

Adapt.

=

+

+

Combined IR + Graph Search
29

Secure.

Scale.

Adapt.


Schema-less Stats
30

Secure.

Scale.

Adapt.


Get Involved
hYp://accumulo.apache.org

Help
us
make
Accumulo
even
beYer!

31

Secure.

Scale.

Adapt.


Contact
32

Adam
Fuchs,
CTO

sqrrl
data,
Inc.

617-‐520-‐4375

www.sqrrl.com

@sqrrl_inc

info@sqrrl.com

Secure.

Scale.

Adapt.


Hugaccumulo 121018192044-phpapp02

More Related Content

Similar to Hugaccumulo 121018192044-phpapp02

More from Sqrrl

Recently uploaded

Hugaccumulo 121018192044-phpapp02