1.
CouchDB at Its Core
Global Data Storage and Rich Incremental Indexing at Cloudant
Adam Kocoloski
StampedeCon 2013

2.
What is Cloudant?
• Founded by “big data” scientists
• Particle physicists @ MIT analyzing
petabytes of collider data
• Frustrated by inadequate tools,
founders became experts in
scaling CouchDB (“BigCouch”)
2
• Started Cloudant in 2008 as a managed data layer
• Premise: Apps should grow into their data layer, not out of it
• Built: Scalable, global, fault-tolerant data layer managed service
• Funded by Avalon, Devonshire (Fidelity), IQT, Rackspace, Samsung Ventures, Toba
Capital, Y Combinator

3.
Cloudant Overview
• Operational JSON document store
• Web service
• Advanced APIs
• Replication & Sync
• Full-text Search
• Geospatial
• Incremental MapReduce
• Scalable, Highly Available Performance
• Cross-data center data distribution & fail over
• Geo load balancing
• Multi-tenant and single-tenant clusters
• Monitoring, admin & dev dashboards
• Managed 24x7 by experts
4

4.
5
Cloudant: 34 locations on 5 hosting providers

5.
Anatomy of the Cloudant Data Network
US-EAST “Node”
Single-
tenant
cluster
Multi-tenant
cluster
HTTP POST, GET,…{JSON doc}
Edge Database Cluster
Mobile Devices
AP-JP
Filtered
Replication
& Sync
Secondary Data Centers
(for DR & distributed access)
EU-NL
6

6.
Horizontal Clustering Framework
How CouchDB Fits In
Visualization
Lucene
Search
Chainable
MapReduce
Management
Monitoring
IOQ
Fabric Mem3 Rexi
Apache CouchDB
Docs: JSON,
Attachments
Developer APIs
Prioritizing IO types; prevents
“noisy neighbors” in multi-tenancy
Clustering API, Sharding,
Intra-cluster messaging
GET/PUT docs, Views,
Replication…
Horizontal Clustering Framework
Geospatial
Indexing
Geo-Load Balancing Connects users to closest copy of
data
Dashboards-Monitoring, Admin,
Development
7

7.
Why CouchDB?
8
• Durable append-only storage engine
• Sequence tree enabling incremental processing of updates
• Data structures supporting eventual consistency
• Sophisticated replication & synchronization
The right primitives for a global data network

8.
9
Append-only Storage

9.
Append-only Storage
10
• Rewrite path to root in each index on
document update
• Large sequential writes, smaller random reads
• Wasted space must be periodically vacuumed
• Disk is cheap
• SSD-friendly access pattern
• We build what we run ➜ we make things
that are easy to run
• (We automated the heck out of the compactor)
This used to be controversial, now everyone does it

10.
11
Sequence Index

11.
Sequence Index
12
1
foo
2
bar
3
baz
4
bif
GET /db/_changes
{“seq”:1, “id”: “foo”, “rev”:”1-...”}
{“seq”:2, “id”: “bar”, “rev”:”1-...”}
{“seq”:3, “id”: “baz”, “rev”:”1-...”}
{“seq”:4, “id”: “bif”, “rev”:”1-...”}

12.
Sequence Index
13
1
foo
2
bar
3
baz
4
bif
GET /db/_changes
{“seq”:1, “id”: “foo”, “rev”:”1-...”}
{“seq”:3, “id”: “baz”, “rev”:”1-...”}
{“seq”:4, “id”: “bif”, “rev”:”1-...”}
{“seq”:5, “id”: “bar”, “rev”:”2-...”}
5
bar
OR
GET /db/_changes?since=4
{“seq”:5, “id”: “bar”, “rev”:”2-...”}

13.
Sequence Index
14
• Index each document in order of most recent update
• Allows incremental, resumable processing in the background
• Originally, MapReduce views
• First class API endpoint ➜ DIY integrations (c.f. ElasticSearch)
• Lucene-based text search
• Geospatial indexes and querying
• First class internal service ➜ add additional consumers as need arises

14.
15
Eventual Consistency

15.
Eventual Consistency
16
• CAP theorem (Brewer)
• O"en over-simplified
• I’ll offer my own oversimplification: “You must choose P”
• When faced with a network partition, you optimize for consistency
or availability
• Cloudant is an ODS
• Availability is paramount
• Strong consistency across geographies introduces unacceptable latency*
✱ Unless you’re Google and you install atomic clocks in your data centers

16.
Eventual Consistency: Hash Histories
17
• Multiple concurrent versions of data will happen
• Default strategy cannot be to discard user data
• Hash histories track versions of a document
• Baked into every document
• Think git
• Document versions derived from contents + edit history
• Same series of edits, applied in same order, yield same
version ID
• History comparison detects divergences and how the
versions fit into the “family tree”
1-5a4...
2-ab6...
3-085...3-f57...
4-7ba... 4-8bf...
5-d4e...

17.
18
Replication & Synchronization

18.
Replication & Sync
19
1-5a4...
2-ab6...
3-085...
4-7ba... 4-8bf...
5-d4e...
1-5a4...
2-ab6...
3-085...3-f57...
/db1/foo /db2/foo

19.
Replication & Sync
20
1-5a4...
2-ab6...
3-085...3-f57...
4-7ba... 4-8bf...
5-d4e...
/db1/foo /db2/foo
1-5a4...
2-ab6...
3-085...3-f57...

20.
Replication & Sync
21
1-5a4...
2-ab6...
3-085...3-f57...
4-7ba... 4-8bf...
5-d4e...
1-5a4...
2-ab6...
3-085...3-f57...
4-7ba... 4-8bf...
5-d4e...
/db1/foo /db2/foo

21.
Replication & Sync
22
• Not your RDBMS’ notion of replication
• Transfers updates from any source DB to any target DB
• Builds on earlier primitives
• Leverages sequence index to determine what’s changed
• Leverages hash histories to determine what’s missing on the target
• Critical “anti-entropy” element in clusters
• DBs are divided into partitions, copies of each partition are stored on
multiple distinct nodes
• Partition copies replicate with each other to ensure that documents are
durably stored and that consistency is achieved ... eventually

22.
Why CouchDB Recap
23
• Durable append-only storage engine
• Sequence tree enabling incremental processing of updates
• Data structures supporting eventual consistency
• Sophisticated replication & synchronization

23.
What’s Next?
24
• BigCouch ➜ CouchDB
• Cloudant will continue development under ASF umbrella
• Fewer code forks ➜ better velocity
• New CouchDB web UI “Fauxton”
• Better developer tooling for server-side code
• Plugins for Cloudant-specific functionality
• Cloudant is betting on data “at the edge”

24.
Thank You
adam@cloudant.com
@kocolosk