![A
New
Technology?
Bigtable
November
2006
Dynamo
October
2007
Cassandra
August
2008
Voldemort
February
2009
Building
new
database
to
answer
the
following
requirements
• No
schema
required
before
inser)ng
data
• No
schema
change
required
to
change
data
format
• Auto-‐sharding
without
applica)on
par)cipa)on
• Distributed
queries
• Integrated
main
memory
caching
• Data
synchroniza)on
(
mul)-‐datacenter)
Very
few
organiza)ons
want
to
(fewer
can)
build
and
maintain
database
so]ware
technology.
But
every
organiza)on
building
interac)ve
web
applica)ons
needs
this
technology.
Monday, October 28, 13](https://image.slidesharecdn.com/softshake-2013-intro-to-nosq-131028043629-phpapp02/85/Softshake-2013-Introduction-to-NoSQL-with-Couchbase-9-320.jpg)
![Document
Database
o::1001
{
uid: “ji22jd”,
customer: “Ann”,
line_items: [
{ sku: 0321293533, qty: 2, price: 48.0 },
{ sku: 0321601912, qty: 1, price: 39.0 },
{ sku: 0131495054, qty: 1, price: 51.0 }
],
payment: {
type: “Amex”,
expiry: “04/2001”,
last5: 12345
}
}
Aggregate
Oriented
Database
hEp://marAnfowler.com/bliki/AggregateOrientedDatabase.html
Monday, October 28, 13](https://image.slidesharecdn.com/softshake-2013-intro-to-nosq-131028043629-phpapp02/85/Softshake-2013-Introduction-to-NoSQL-with-Couchbase-16-320.jpg)
![Schema
Update
:
Document
user:rbin
{ user:rbin
{ user:rbin
type: “user”,
{
type: “user”,
name: “Robin Johnson”,
type: “user”,
name: “Robin Johnson”,
}
} name: “Robin Johnson”,
}
event:1
{ event:1
1. Change
The
ApplicaPon
{ event:1
type: “event”,
{ event:1
type: “event”, the
• Update
{ event:1
event:1
title: “Paris JUG”, Data
when
“you”
want
type: “Paris JUG”,
{
title: “event”, the
index
to
query
the
new
format
“event”,
{type: “Paris
•
speaker: “rbin” JUG”,
title: Change
type:“rbin” JUG”,
title: “event”,
} speaker: “Paris JUG”,
type: “Paris
title: “event”,
} speaker: “rbin” JUG”,
title: “rbin”
} speaker: “Paris
speaker: “rbin”
} speaker: [“rbin”,”tgrall”]
}
}
Monday, October 28, 13](https://image.slidesharecdn.com/softshake-2013-intro-to-nosq-131028043629-phpapp02/85/Softshake-2013-Introduction-to-NoSQL-with-Couchbase-18-320.jpg)
Uploaded byTugdual Grall
Softshake 2013: Introduction to NoSQL with Couchbase
The document provides an introduction to NoSQL databases with Couchbase, highlighting the challenges of scaling traditional RDBMS and the advantages of NoSQL technologies. It discusses the scalability, flexibility, and performance benefits of NoSQL solutions, particularly Couchbase, compared to traditional relational databases. Furthermore, it outlines Couchbase's architecture, its core principles, and its features that allow for smooth scaling, data management, and real-time querying.
More Related Content
Similar to Softshake 2013: Introduction to NoSQL with Couchbase
Softshake 2013: Introduction to NoSQL with Couchbase
- 1. Introduc)on to NoSQL with Couchbase Tugdual “Tug” Grall Technical Evangelist tug@couchbase.com Monday, October 28, 13 TwiBer: tgrall
- 2. {“about” : “me”} •Tugdual “Tug” Grall -‐ Couchbase -‐ -‐ eXo -‐ -‐ Technical Evangelist CTO Oracle • Web -‐ -‐ -‐ @tgrall hEp://blog.grallandco.com tgrall • NantesJUG co-‐founder • Pet Project : • hEp://www.resultri.com -‐ • tug@couchbase.com -‐ -‐ Developer/Product Manager Mainly Java/SOA • tugdual@gmail.com Developer in consulAng firms Monday, October 28, 13
- 3. RDBMS are not Enough? Monday, October 28, 13
- 4. Growth is the New Reality • Instagram gained nearly 1 million users overnight when then expanded to Android Monday, October 28, 13
- 5. 50 Million Users in 50 Days Draw Something by OMGPOP Daily Ac)ve Users (millions) 16 14 12 10 8 6 4 2 2/6 8 10 Monday, October 28, 13 12 14 16 18 20 22 24 26 28 3/1 3 5 7 9 11 13 15 17 19 21
- 6. How do you take this growth? Applica;on Scales Out Just add more commodity web servers System Cost Applica;on Performance Web/App Server Tier Users RDBMS Scales Up Get a bigger, more complex server System Cost Applica;on Performance Won’t scale beyond this point Rela2onal Database Users RDBMS is good for many thing, but hard to scale Monday, October 28, 13
- 7. Scaling out RDBMS •Run Many SQL Servers Web/App Server Tier • Data could be sharded Memcached Tier Done by the applicaAon code • Caching for faster response )me MySQL Tier Monday, October 28, 13
- 8. NoSQL Technology Scales Out Applica;on Scales Out Just add more commodity web servers System Cost Applica;on Performance Web/App Server Tier Users NoSQL Database Scales Out Cost and performance mirrors app ;er System Cost Applica;on Performance NoSQL Distributed Data Store Users Scaling out fla?ens the cost and performance curves Monday, October 28, 13
- 9. A New Technology? Bigtable November 2006 Dynamo October 2007 Cassandra August 2008 Voldemort February 2009 Building new database to answer the following requirements • No schema required before inser)ng data • No schema change required to change data format • Auto-‐sharding without applica)on par)cipa)on • Distributed queries • Integrated main memory caching • Data synchroniza)on ( mul)-‐datacenter) Very few organiza)ons want to (fewer can) build and maintain database so]ware technology. But every organiza)on building interac)ve web applica)ons needs this technology. Monday, October 28, 13
- 10. What Is Biggest Data Management Problem Driving Use of NoSQL in Coming Year? 49% 35% 29% 16% Lack of flexibility/ rigid schemas Inability to scale Performance challenges out data Source: Couchbase Survey, December 2011, n = 1351. Monday, October 28, 13 Cost 12% 11% All of these Other
- 11. NoSQL Catalog Cache (memory only) Key-‐Value Memcached Data Structure Document Column Graph Redis Database (memory/disk) Coherence Membase Cassandra Neo4j MongoDB Monday, October 28, 13 Couchbase HBase InfiniteGraph
- 12. Opera)onal vs. Analy)c Databases Real-‐Pme, InteracPve Databases AnalyPc Databases NoSQL Fast access to data Couchbase MongoDB Monday, October 28, 13 Get insights from data Cloudera Cassandra Hortonworks Hbase Mapr
- 13.
- 14. Rela;onal vs Document Data Model C1 C2 C3 C4 { JSON JSON } JSON RelaPonal data model Document data model Highly-‐structured table organiza;on with rigidly-‐defined data formats and record structure. Collec;on of complex documents with arbitrary, nested data formats and varying “record” format. Monday, October 28, 13
- 15.
- 16. Document Database o::1001 { uid: “ji22jd”, customer:“Ann”, line_items: [ { sku: 0321293533, qty: 2, price: 48.0 }, { sku: 0321601912, qty: 1, price: 39.0 }, { sku: 0131495054, qty: 1, price: 51.0 } ], payment: { type: “Amex”, expiry: “04/2001”, last5: 12345 } } Aggregate Oriented Database hEp://marAnfowler.com/bliki/AggregateOrientedDatabase.html Monday, October 28, 13
- 17. Schema Update : RDBMS Event Info ID 1 Title Speaker Speaker_id Paris JUG ID Name rbin rbin Robin Johnson 1. Create a new table 2 tgrall QCON UK 2.“Prepare” the Data (move to new tgrall drop column) table, Tug Grall A talk could be done by one or more speakers! 3. Change the constraints 3 4 NoSQL CLN johnz johnz John Zablo fweigel fweigel Frank Weigel 4. Change the app Devoxx Monday, October 28, 13
- 18. Schema Update : Document user:rbin { user:rbin { user:rbin type: “user”, { type: “user”, name: “Robin Johnson”, type: “user”, name: “Robin Johnson”, } } name: “Robin Johnson”, } event:1 { event:1 1. Change The ApplicaPon { event:1 type: “event”, { event:1 type: “event”, the • Update { event:1 event:1 title: “Paris JUG”, Data when “you” want type: “Paris JUG”, { title: “event”, the index to query the new format “event”, {type: “Paris • speaker: “rbin” JUG”, title: Change type:“rbin” JUG”, title: “event”, } speaker: “Paris JUG”, type: “Paris title: “event”, } speaker: “rbin” JUG”, title: “rbin” } speaker: “Paris speaker: “rbin” } speaker: [“rbin”,”tgrall”] } } Monday, October 28, 13
- 19.
- 20. Couchbase Server Core Principles Easy Scalability PE RF O R M A N C E Grow cluster without applicaAon changes, without downAme with a single click Always On 24x365 No downAme for sodware upgrades, hardware maintenance, etc. Monday, October 28, 13 Consistent High Performance Consistent sub-‐millisecond read and write response Ames with consistent high throughput JSON JSON JSO JSON N JSON Flexible Data Model JSON document model with no fixed schema.
- 21.
- 22. Couchbase Architecture 8092 11211 Query API Memcapable 1.0 11210 Memcapable 2.0 New Persistence Layer vBucket state and replica;on manager Node health monitor Rebalance orchestrator Global singleton supervisor storage interface Configura;on manager Data Manager Process monitor Couchbase EP Engine Heartbeat Memcached REST management API/Web UI Query Engine Moxi Cluster Manager hRp on each node one per cluster Erlang/OTP HTTP 8091 Monday, October 28, 13 Erlang port mapper 4369 Distributed Erlang 21100 -‐ 21199
- 23. Couchbase Architecture 8092 11211 Query API Memcapable 1.0 11210 Memcapable 2.0 New Persistence Layer Disk Persistence vBucket state and replica;on manager Node health monitor Rebalance orchestrator Global singleton supervisor Configura;on manager storage interface Server/Cluster Management & Communica2on (Erlang) hRp Process monitor RAM Cache, Indexing Couchbase EP Engine & Persistence Management (C & V8) Heartbeat Object-‐level Cache REST management API/Web UI Query Engine Moxi on each node one per cluster Erlang/OTP The Unreasonable Effectiveness of C by Damien Katz HTTP 8091 Monday, October 28, 13 Erlang port mapper 4369 Distributed Erlang 21100 -‐ 21199
- 24. Open Source Project Apache 2.0 hEps://github.com/couchbase/ hEps://github.com/couchbaselabs/ Monday, October 28, 13 Gerrit: hEp://review.couchbase.org/
- 25. Cluster-‐wide Basic Opera)on App Server 1 App Server 2 COUCHBASE Client Library COUCHBASE Client Library Cluster Map Cluster Map READ/WRITE/UPDATE Server 1 Server 2 Server 3 Ac2ve Ac2ve Ac2ve Doc 5 Doc Doc 4 Doc Doc 1 Doc Doc 2 Doc Doc 7 Doc Doc 2 Doc Doc 9 Doc Doc 8 Doc Doc 6 Doc REPLICA REPLICA REPLICA • Docs distributed evenly across servers • Each server stores both ac)ve and replica docs Only one server acAve at a Ame • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on Doc 4 Doc Doc 6 Doc Doc 7 Doc Doc 1 Doc Doc 3 Doc Doc 9 Doc • App reads, writes, updates docs Doc 8 Doc Doc 2 Doc Doc 5 Doc • Mul)ple app servers can access same document at same )me Couchbase Server Cluster User Configured Replica Count = 1 Monday, October 28, 13 App never needs to know
- 26. Add Nodes to Cluster App Server 1 App Server 2 COUCHBASE Client Library COUCHBASE Client Library Cluster Map Cluster Map READ/WRITE/UPDATE READ/WRITE/UPDATE Server 1 Server 2 Server 3 Server 4 Server 5 Ac2ve Ac2ve Ac2ve Ac2ve Ac2ve Doc 5 Doc Doc 4 Doc Doc 1 Doc Doc 2 Doc Doc 7 Doc Doc 2 Doc Doc 9 Doc Doc 8 Doc Doc 6 Doc REPLICA REPLICA REPLICA Doc 4 Doc Doc 6 Doc Doc 7 Doc Doc 1 Doc Doc 3 Doc Doc 9 Doc Doc 8 Doc Doc 2 Doc Doc 5 Doc Couchbase Server Cluster User Configured Replica Count = 1 Monday, October 28, 13 • Two servers added One-‐click opera)on • Docs automa)cally rebalanced across cluster Even distribuAon of docs Minimum doc movement REPLICA REPLICA • Cluster map updated • App database calls now distributed over larger number of servers
- 27. Fail Over Node App Server 1 App Server 2 COUCHBASE Client Library COUCHBASE Client Library Cluster Map Cluster Map Server 1 Server 2 Server 3 Server 4 Server 5 Ac2ve Ac2ve Ac2ve Ac2ve Ac2ve Doc 5 Doc Doc 4 Doc Doc 1 Doc Doc 9 Doc Doc 2 Doc Doc 7 Doc Doc 2 Doc Doc 8 Doc Doc 1 Doc 6 Doc Doc Doc 3 REPLICA REPLICA REPLICA REPLICA Doc 4 Doc Doc 6 Doc Doc 7 Doc Doc 5 Doc 1 Doc Doc 3 Doc Doc 9 Doc Doc 2 Couchbase Server Cluster User Configured Replica Count = 1 Monday, October 28, 13 Doc REPLICA Doc 8 Doc Doc • App servers accessing docs • Requests to Server 3 fail • Cluster detects server failed Promotes replicas of docs to acAve Updates cluster map • Requests for docs now go to appropriate server • Typically rebalance would follow
- 28. Indexing and Querying APP SERVER 1 APP SERVER 2 COUCHBASE Client Library COUCHBASE Client Library CLUSTER MAP CLUSTER MAP Query SERVER 1 SERVER 2 ACTIVE ACTIVE SERVER 3 ACTIVE • Indexing work is distributed amongst nodes Doc 5 Doc Doc 5 Doc Doc 5 Doc Doc 2 Doc Doc 2 Doc Doc 2 Doc • Parallelize the effort Doc 9 Doc Doc 9 Doc Doc 9 Doc • Each node has index for data stored on it REPLICA REPLICA REPLICA Doc 4 Doc Doc 4 Doc Doc 4 Doc Doc 1 Doc Doc 1 Doc Doc 1 Doc Doc 8 Doc Doc 8 Doc Doc 8 Doc COUCHBASE SERVER CLUSTER Monday, October 28, 13 • Large data set possible • Queries combine the results from required nodes
- 29. Couchbase SDK Official SDKs Ruby Python Community SDKs Go Clojure www.couchbase.com/develop Monday, October 28, 13
- 30. Full Text Search •Elastic Search is good for ad-hoc queries and faceted browsing • Our adapter is aware of changing Couchbase topology • Indexed by Elastic Search after stored to disk in Couchbase ElasPcSearch Monday, October 28, 13
- 31. The Complete Mobile Solu;on Available now in Beta : hfp://mobile.couchbase.com Monday, October 28, 13
- 32. Couchbase Docs www.couchbase.com/docs/ Couchbase Forums www.couchbase.com/communi;es Meetup hbp://meetup.com/Couchbase-‐France/ Mobile (beta) hbp://mobile.couchbase.com/ N1QL -‐ New Query Engine (DP) hbp://query.couchbase.com Monday, October 28, 13 Contact me on Twi?er @tgrall Contact me by Email tug@couchbase.com IRC #couchbase #libcouchbase
- 33. Introduc)on to NoSQL with Couchbase Tugdual “Tug” Grall Technical Evangelist tug@couchbase.com Monday, October 28, 13 TwiBer: tgrall