1. Couchbase
Server
for
Speed
and
Scale
with
Interac8ve
Applica8ons
Perry
Krug
Presenter
Title
1

2. Agenda
(just
an
outline)
• Needs
for
Interac8ve
Apps
– Speed
• Speed
of
access
• Speed
of
app
development
– Scale
• Built
to
scale
– Unique
architectural
principals
• Clustered
system
– Simple
for
applica8ons
– Automa8c
sharding
of
data
across
cluster
• Single
index
view
– No
need
to
define
sharding
keys
– Automa8c,
online
transfer
of
data
across
cluster
2

3. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
EXAMPLE
OF
AN
INTERACTIVE
APP
3

4. Instagrowth:
Android
Launch
E xample
• Instagram
gained
nearly
1
million
users
overnight
when
they
expanded
to
Android
4

5. xample
E 1 Instagram
=
7.5M MAU*
5

6. Draw
Something
-­‐
Social
Game
• 50
million
downloads
in
50
days
• That’s
almost
5
Instagrams.
In
2
months.
6

7. Goes
Viral
3
Weeks
AZer
Launch
Draw
Something
by
OMGPOP
Daily
Ac8ve
Users
(millions)
16
14
12
10
8
6
4
2
2/6
8
10
12
14
16
18
20
22
24
26
28
3/1
3
5
7
9
11
13
15
17
19
21
7

8. Database
Opera8ons
Draw
Something
by
OMGPOP
Daily
Ac8ve
Users
(millions)
16
14
12
10
8
By
March
19,
there
were
over
30,000,000
downloads
of
the
app,
6
over
5,000
drawings
being
stored
per
second,
over
2,200,000,000
drawings
stored,
over
105,000
database
transac8ons
per
second,
4
and
over
3.3
terabytes
of
data
stored.
2
2/6
8
10
12
14
16
18
20
22
24
26
28
3/1
3
5
7
9
11
13
15
17
19
21
8

9. Social
Gaming
on
Couchbase
iki
9

10. Speed
• Modern,
interac8ve
applica8ons
require
speed
– Web
pages
are
dynamic,
interac8ve
and
may
incrementally
fetch
content
for
display
– Mobile
applica8ons
cannot
carry
all
of
the
content,
so
they
need
to
be
able
to
fetch
data
on
demand
• Couchbase
Speed
via…
– Managed,
built
in
cache
• memcached
compa8ble,
built
on
memcached
– Protocol
design
• Client
can
directly
access
data
objects
within
the
cluster
• No
quorum
reads
needed
to
determine
version,
always
consistent
10

11. Scale
• Modern
app
deployments
are
always
on,
available
24x7x365
– Some
applica8ons
may
have
a
lifecycle
which
requires
scaling
down,
in
addi8onal
to
scaling
up
• Couchbase
scales
via…
– Automa8cally
sharding
data
among
the
nodes
of
the
cluster
and
atomically
transferring
vbuckets
– Coordina8ng
these
changes
with
client
libraries
accessing
the
cluster
11

12. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
CLUSTERING
ARCHITECTURE
12

13. Basic
Opera8on
APP
SERVER
1
APP
SERVER
2
 Docs
distributed
evenly
across
COUCHBASE
CLIENT
LIBRARY
servers
in
the
cluster
COUCHBASE
CLIENT
LIBRARY
 Each
server
stores
both
ac/ve
CLUSTER
MAP
CLUSTER
MAP
&
replica
docs
 Only
one
server
ac8ve
at
a
8me
 Client
library
provides
app
with
Read/Write/Update
Read/Write/Update
simple
interface
to
database
 Cluster
map
provides
map
to
which
server
doc
is
on
 App
never
needs
to
know
SERVER
1
SERVER
2
SERVER
3
 App
reads,
writes,
updates
Ac8ve
Docs
Ac8ve
Docs
Ac8ve
Docs
docs
Doc
5
DOC
Doc
4
DOC
Doc
1
DOC
 Mul8ple
App
Servers
can
Doc
2
DOC
Doc
7
DOC
Doc
3
DOC
access
same
document
at
Doc
9
DOC
Doc
8
DOC
Doc
6
DOC
same
8me
Replica
Docs
Replica
Docs
Replica
Docs
Doc
4
DOC
Doc
6
DOC
Doc
7
DOC
Doc
1
DOC
Doc
3
DOC
Doc
9
DOC
Doc
8
DOC
Doc
2
DOC
Doc
5
DOC
COUCHBASE
SERVER
CLUSTER
User
Configured
Replica
Count
=
1
13

14. Add
Nodes
APP
SERVER
1
APP
SERVER
2
 Two
servers
added
to
COUCHBASE
CLIENT
LIBRARY
COUCHBASE
CLIENT
LIBRARY
cluster
 One-­‐click
opera8on
CLUSTER
MAP
CLUSTER
MAP
 Docs
automa8cally
rebalanced
across
cluster
 Even
distribu8on
of
docs
Read/Write/Update
Read/Write/Update
 Minimum
doc
movement
 Cluster
map
updated
 App
database
calls
now
distributed
over
larger
#
SERVER
1
SERVER
2
SERVER
3
SERVER
4
SERVER
5
of
servers
Ac8ve
Docs
Ac8ve
Docs
Ac8ve
Docs
ocs
Ac8ve
Docs
Ac8ve
Docs
Ac8ve
D
Doc
5
DOC
Doc
4
DOC
Doc
1
DOC
Doc
3
Doc
2
DOC
Doc
7
DOC
Doc
3
DOC
Doc
6
Doc
9
DOC
Doc
8
DOC
Doc
6
DOC
Replica
Docs
Replica
Docs
Replica
Docs
Replica
Docs
Replica
Docs
Replica
Docs
Doc
4
DOC
Doc
6
DOC
Doc
7
7
DOC
Doc
Doc
1
DOC
Doc
3
DOC
Doc
9
9
DOC
Doc
Doc
8
DOC
Doc
2
DOC
Doc
5
DOC
COUCHBASE
SERVER
CLUSTER
User
Configured
Replica
Count
=
1
14

15. Fail
Over
Node
APP
SERVER
1
APP
SERVER
2
 App
servers
happily
accessing
docs
on
Server
3
COUCHBASE
CLIENT
LIBRARY
 Server
fails
COUCHBASE
CLIENT
LIBRARY
 App
server
requests
to
server
3
fail
CLUSTER
MAP
CLUSTER
MAP
 Cluster
detects
server
has
failed
 Promotes
replicas
of
docs
to
ac/ve
 Updates
cluster
map
 App
server
requests
for
docs
now
go
to
appropriate
server
 Typically
rebalance
would
follow
SERVER
1
SERVER
2
SERVER
3
SERVER
4
SERVER
5
Ac8ve
Docs
Ac8ve
Docs
Ac8ve
Docs
ocs
Ac8ve
Docs
Ac8ve
Docs
Ac8ve
D
Doc
5
DOC
Doc
4
DOC
Doc
1
DOC
Doc
9
DOC
Doc
6
DOC
Doc
3
Doc
2
DOC
Doc
7
DOC
Doc
3
Doc
8
DOC
Doc
6
DOC
Replica
Docs
Replica
Docs
Replica
Docs
Replica
Docs
Replica
Docs
Replica
Docs
Doc
4
DOC
Doc
6
DOC
Doc
7
7
DOC
Doc
Doc
5
DOC
Doc
8
DOC
Doc
1
DOC
Doc
3
DOC
Doc
9
9
DOC
Doc
Doc
2
DOC
COUCHBASE
SERVER
CLUSTER
User
Configured
Replica
Count
=
1
15

16. The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
SINGLE
INDEX
VIEW
16

17. Couchbase
Server
Views
• Couchbase
has
a
method
of
building
and
map:
function(doc)
maintaining
views
over
the
{
set
of
documents
within
a
emit(doc.name,
[doc.city, doc.salary]);
bucket
}
• Can
be
used
for:
reduce:
– Secondary
indexes
function (key, values, rereduce) {
var sum = 0;
– Aggrega8ons
and
stats
for(i=0; i < values.length; i++)
{
– Analyzing
datasets
sum = sum + values[i];
}
• Built
for
speed
}
return(sum);
– Allow
relaxed
consistency
for
speed,
or
consistent
if
required
17

18. Views
are
B+Trees
Over
the
Data
Key: "f" Key: "f"
Sum: 17 Sum: 17
bits 0 0 0 0 bits 0 0 0 0
Key: "f" Key: "f" Key: "f" Key: "f"
Sum: 17 Sum: 17 Sum: 17 Sum: 17
bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0
Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Key: "f" Key: "f"
Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17 Sum: 17
bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0
Key: "a" Key: "b" Key: "c" Key: "d" Key: "e" Key: "f" Key: "j" Key: "k" Key: "l" Key: "m" Key: "r" Key: "s" Key: "t" Key: "u" Key: "v" Key: "w" Key: "x" Key: "y" Key: "z"
Value: 4 Value: 2 Value: 0 Value: 6 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4
vBucket: 1 vBucket: 2 vBucket: 3 vBucket: 4 vBucket: 5 vBucket: 6 vBucket: 10 vBucket: 11 vBucket: 12 vBucket: 13 vBucket: 18 vBucket: 19 vBucket: 20 vBucket: 21 vBucket: 22 vBucket: 23 vBucket: 24 vBucket: 25 vBucket: 26
Key: "n" Key: "o" Key: "p" Key: "q"
Key: "g" Key: "h" Key: "i" Value: 4 Value: 4 Value: 4 Value: 4
Value: 4 Value: 4 Value: 4 vBucket: 14 vBucket: 15 vBucket: 16 vBucket: 17
vBucket: 7 vBucket: 8 vBucket: 9
Views
are
Built
Atop
Incremental
Map
Reduce
• Distributed
throughout
the
cluster
• Query
API
abstracts
the
cluster
topology,
and
automa8cally
gathers
the
results
• Rebuilds
subsets
of
the
B+Tree
based
on
updates
18

19. Key: "m" Key: "z"
Sum: 92 Sum: 17
bits 0 0 0 0 bits 0
Key: "f" Key: "m"
Sum: 21 Sum: 71
bits 0 0 0 0 bits 0 0 0 0
Key: "d" Key: "f" Key: "i" Key: "m"
Sum: 10 Sum: 11 Sum: 24 Sum: 47
bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0 bits 0 0 0 0
Key: "a" Key: "b" Key: "c" Key: "d" Key: "e" Key: "f" Key: "j" Key: "k" Key: "l" Ke
Value: 4 Value: 2 Value: 0 Value: 6 Value: 4 Value: 4 Value: 4 Value: 4 Value: 4 Va
vBucket: 1 vBucket: 2 vBucket: 3 vBucket: 4 vBucket: 5 vBucket: 6 vBucket: 10 vBucket: 11 vBucket: 12 vBuc
Key: "g" Key: "h" Key: "i"
Value: 4 Value: 4 Value: 4
19
vBucket: 7 vBucket: 8 vBucket: 9

20. Data
is
Distributed
Using
vbuckets,
but…
• Couchbase
Server
has
1024
vbuckets
stored
throughout
the
cluster
20

21. B-­‐Superstar:
a
new
B+Tree
ready
for
vbuckets
Only
one
B+Tree
per
node
with
masking
of
ac8ve
vbucket
Data.
S8ll
allows
concurrent
Read
more
on
Damien’s
blog:
access
and
even
hmp://damienkatz.net/2012/05/stabilizing_couchbase_server_2.html
materializing
tree
during
rebalance
21

22. Thanks
Q&A
Perry
Krug
22