Your SlideShare is downloading. ×
0
Riak Tutorial         Sean Cribbs & Ian Plosker           Basho Technologiesbasho
What is Riak?
• Key-Value store (plus extras)
• Key-Value store (plus extras)• Distributed, horizontally scalable
• Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant
• Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant• Highly-available
• Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant• Highly-available• Built for the Web
• Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant• Highly-available• Built for the Web• ...
Key-Value
Key-Value• Simple operations - get, put, delete
Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)
Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras
Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce
Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce • Links
Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce • Links • Full...
Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce • Links • Full...
Distributed &Horizontally
Distributed &     Horizontally• Default configuration is in a cluster
Distributed &     Horizontally• Default configuration is in a cluster• Load and data are spread evenly
Distributed &     Horizontally• Default configuration is in a cluster• Load and data are spread evenly• Add more nodes to g...
Fault-Tolerant
Fault-Tolerant• All nodes participate equally - no  SPOF
Fault-Tolerant• All nodes participate equally - no  SPOF• All data is replicated
Fault-Tolerant• All nodes participate equally - no  SPOF• All data is replicated• Cluster transparently survives...
Fault-Tolerant• All nodes participate equally - no  SPOF• All data is replicated• Cluster transparently survives... • node...
Fault-Tolerant• All nodes participate equally - no  SPOF• All data is replicated• Cluster transparently survives... • node...
Fault-Tolerant• All nodes participate equally - no  SPOF• All data is replicated• Cluster transparently survives... • node...
Highly-Available
Highly-Available• Any node can serve client requests
Highly-Available• Any node can serve client requests• Fallbacks are used when nodes are  down
Highly-Available• Any node can serve client requests• Fallbacks are used when nodes are  down• Always accepts read and wri...
Highly-Available• Any node can serve client requests• Fallbacks are used when nodes are  down• Always accepts read and wri...
Built for the Web
Built for the Web• HTTP is default (but not only) interface
Built for the Web• HTTP is default (but not only) interface• Does HTTP/REST well (see  Webmachine)
Built for the Web• HTTP is default (but not only) interface• Does HTTP/REST well (see  Webmachine)• Plays well with HTTP i...
Built for the Web• HTTP is default (but not only) interface• Does HTTP/REST well (see  Webmachine)• Plays well with HTTP i...
Inspired byAmazon’s Dynamo
Inspired byAmazon’s Dynamo• Masterless, peer-coordinated  replication
Inspired byAmazon’s Dynamo• Masterless, peer-coordinated  replication• Consistent hashing
Inspired byAmazon’s Dynamo• Masterless, peer-coordinated  replication• Consistent hashing• Eventually consistent
Inspired byAmazon’s Dynamo• Masterless, peer-coordinated  replication• Consistent hashing• Eventually consistent• Quorum r...
Inspired byAmazon’s Dynamo• Masterless, peer-coordinated  replication• Consistent hashing• Eventually consistent• Quorum r...
Lab: Riak Basics
Installing Riak
Installing Riak• Download from our local server
Installing Riak• Download from our local server• Download a package:  http://downloads.basho.com/riak/  CURRENT
Installing Riak• Download from our local server• Download a package:  http://downloads.basho.com/riak/  CURRENT• Clone & b...
You need `curl`    apt-get install curl     yum install curl
Start the Cluster
Start the Cluster$ surge-start.sh
Start the Cluster$ surge-start.shdev1/bin/riak startdev2/bin/riak startdev3/bin/riak startdev4/bin/riak start
Start the Cluster$ surge-start.shdev1/bin/riak startdev2/bin/riak startdev3/bin/riak startdev4/bin/riak startdev2/bin/riak...
Check its status
Check its status$ dev1/bin/riak-admin member_status$ dev1/bin/riak-admin ring_status$ dev1/bin/riak-admin status
PUT
PUT$ surge-put.sh
PUT$ surge-put.shEnter a key: your-key
PUT$ surge-put.shEnter a key: your-keyEnter a value: your-value
PUT$ surge-put.shEnter a key: your-keyEnter a value: your-valueEnter a write quorum value:
PUT$ surge-put.shEnter a key: your-keyEnter a value: your-valueEnter a write quorum value:* About to connect() to 127.0.0....
PUT$ surge-put.shEnter a key: your-keyEnter a value: your-valueEnter a write quorum value:* About to connect() to 127.0.0....
GET
GET$ surge-get.sh
GET$ surge-get.shEnter a key: your-key
GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):
GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8...
GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8...
GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8...
GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8...
GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8...
DELETE
DELETE$ surge-delete.sh
DELETE$ surge-delete.shType a key: your-key
DELETE$ surge-delete.shType a key: your-key* About to connect() to 127.0.0.1 port 8091 (#0)*   Trying 127.0.0.1... connect...
DELETE$ surge-delete.shType a key: your-key* About to connect() to 127.0.0.1 port 8091 (#0)*   Trying 127.0.0.1... connect...
Kill two nodes
Kill two nodes$ dev4/bin/riak stop
Kill two nodes$ dev4/bin/riak stopok
Kill two nodes$ dev4/bin/riak stopok$ dev3/bin/riak stop
Kill two nodes$ dev4/bin/riak stopok$ dev3/bin/riak stopok
Try GETting your key
Try GETting your key$ . ~/Desktop/surge/surge-get.sh
Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-key
Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):
Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* Abo...
Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* Abo...
Riak Architecture
Key-Value Data    Model
Key-Value Data       Model• All data (objects) are  referenced by keys                              mark                  ...
Key-Value Data       Model                                 n_val                                         allow_mult       ...
Key-Value Data       Model                                 n_val                                         allow_mult       ...
Key-Value Data       Model• All data (objects) are  referenced by keys                           people/beth• Keys are gro...
Consistent Hashing   & The Ring
Consistent Hashing   & The Ring• 160-bit integer  keyspace                      0                              2 160/4    ...
Consistent Hashing   & The Ring• 160-bit integer  keyspace                                    0• Divided into fixed  number...
Consistent Hashing   & The Ring                                            node 0                                         ...
Consistent Hashing   & The Ring                 node 0                              node 1• 160-bit integer             no...
Consistent Hashing   & The Ring                                node 0                                             node 1• ...
Request Quorums
Request Quorums• Every request contacts all replicas of  key
Request Quorums• Every request contacts all replicas of  key• N - number of replicas (default 3)
Request Quorums• Every request contacts all replicas of  key• N - number of replicas (default 3)• R - read quorum
Request Quorums• Every request contacts all replicas of  key• N - number of replicas (default 3)• R - read quorum• W - wri...
Vector Clocks
Vector Clocks• Every node has an ID (changed in 1.0)
Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every  “put” or “delete” request
Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every  “put” or “delete” request• Ria...
Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every  “put” or “delete” request• Ria...
Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every  “put” or “delete” request• Ria...
Vector Clockshttp://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/• Every node has an ID (changed in 1.0)• Send las...
Hinted Handoff
Hinted Handoff• Node fails               X                   X                               X               X            ...
Hinted Handoff• Node fails                              X                                X• Requests go to fallback       ...
Hinted Handoff• Node fails• Requests go to fallback• Node comes back                hash(“conferences/surge”)
Hinted Handoff• Node fails• Requests go to fallback• Node comes back• “Handoff” - data  returns to recovered  node        ...
Hinted Handoff• Node fails• Requests go to fallback• Node comes back• “Handoff” - data  returns to recovered  node• Normal...
Anatomy of a  Request get(“conferences/surge”)
Anatomy of a           Request          get(“conferences/surge”)client Riak
Anatomy of a           Request          get(“conferences/surge”)client Riak               Get Handler (FSM)
Anatomy of a           Request          get(“conferences/surge”)client Riak                                     hash(“conf...
Anatomy of a                          Request                         get(“conferences/surge”)          client           R...
Anatomy of a                           Request                             get(“conferences/surge”)          client       ...
Anatomy of a                          Request                         get(“conferences/surge”)          client           R...
Anatomy of a                          Request                         get(“conferences/surge”)          client           R...
Anatomy of a                 Request                    get(“conferences/surge”)      client       RiakR=2      v1    v2  ...
Anatomy of a                 Request                    get(“conferences/surge”)      client                          v2  ...
Anatomy of a  Request get(“conferences/surge”)                 v2
Read Repair                             get(“conferences/surge”)          client                               v2         ...
Read Repair                             get(“conferences/surge”)          client                               v2         ...
Read Repair                             get(“conferences/surge”)          client                               v2         ...
Read Repair                             get(“conferences/surge”)          client                                     v2   ...
Riak ArchitectureErlang/OTP Runtime
Riak ArchitectureErlang/OTP Runtime                     Riak KV
Riak ArchitectureErlang/OTP Runtime      Client APIs                     Riak KV
Riak ArchitectureErlang/OTP Runtime      Client APIs    HTTP                            Riak KV
Riak ArchitectureErlang/OTP Runtime      Client APIs    HTTP   Protocol Buffers                                       Riak...
Riak ArchitectureErlang/OTP Runtime      Client APIs    HTTP               Protocol Buffers                            Erl...
Riak ArchitectureErlang/OTP Runtime      Client APIs      HTTP               Protocol Buffers                             ...
Riak ArchitectureErlang/OTP Runtime      Client APIs      HTTP               Protocol Buffers                             ...
Riak ArchitectureErlang/OTP Runtime      Client APIs      HTTP               Protocol Buffers                             ...
Riak ArchitectureErlang/OTP Runtime      Client APIs        HTTP                 Protocol Buffers                         ...
Riak ArchitectureErlang/OTP Runtime      Client APIs        HTTP                 Protocol Buffers                         ...
Riak ArchitectureErlang/OTP Runtime      Client APIs        HTTP                  Protocol Buffers                        ...
Riak ArchitectureErlang/OTP Runtime      Client APIs       HTTP               Protocol Buffers                            ...
Riak ArchitectureErlang/OTP Runtime      Client APIs       HTTP               Protocol Buffers                            ...
Riak ArchitectureErlang/OTP Runtime      Client APIs      HTTP               Protocol Buffers                             ...
Riak ArchitectureErlang/OTP Runtime      Client APIs          HTTP                Protocol Buffers                        ...
Riak ArchitectureErlang/OTP Runtime      Client APIs          HTTP                Protocol Buffers                        ...
Riak ArchitectureErlang/OTP Runtime      Client APIs          HTTP                Protocol Buffers                        ...
Riak ArchitectureErlang/OTP Runtime      Client APIs          HTTP                Protocol Buffers                        ...
Modeling & Querying
Application  Design
Application        Design• No intrinsic schema
Application        Design• No intrinsic schema• Your application defines the  structure and semantics
Application        Design• No intrinsic schema• Your application defines the  structure and semantics• Your application res...
Modeling Tools
Modeling Tools• Key-Value
Modeling Tools• Key-Value• Links
Modeling Tools• Key-Value• Links• Full-text Search
Modeling Tools• Key-Value• Links• Full-text Search• Secondary Indexes (2I)
Modeling Tools• Key-Value• Links• Full-text Search• Secondary Indexes (2I)• MapReduce
Key-Value
Key-Value• Content-Types
Key-Value• Content-Types• Denormalize
Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys
Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys • Composites
Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys • Composites • Time-boxing
Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys • Composites • Time-boxing• References (value is a k...
Links
Links• Lightweight relationships, like <a>
Links• Lightweight relationships, like <a>• Includes a “tag”
Links• Lightweight relationships, like <a>• Includes a “tag”• Built-in traversal op (“walking”)  GET /riak/b/k/[bucket],[t...
Links• Lightweight relationships, like <a>• Includes a “tag”• Built-in traversal op (“walking”)  GET /riak/b/k/[bucket],[t...
Full-text Search
Full-text Search• Designed for searching prose
Full-text Search• Designed for searching prose• Lucene/Solr-like query interface
Full-text Search• Designed for searching prose• Lucene/Solr-like query interface• Automatically index K/V values
Full-text Search• Designed for searching prose• Lucene/Solr-like query interface• Automatically index K/V values• Input to...
Full-text Search• Designed for searching prose• Lucene/Solr-like query interface• Automatically index K/V values• Input to...
Secondary Indexes
Secondary Indexes• Defined as metadata
Secondary Indexes• Defined as metadata• Two index types: _int and _bin
Secondary Indexes• Defined as metadata• Two index types: _int and _bin• Two query types: equal and range
Secondary Indexes• Defined as metadata• Two index types: _int and _bin• Two query types: equal and range• Input to MapReduce
MapReduce
MapReduce• For more involved queries
MapReduce• For more involved queries • Specify input keys
MapReduce• For more involved queries • Specify input keys • Process data in “map” and   “reduce” functions
MapReduce• For more involved queries • Specify input keys • Process data in “map” and   “reduce” functions   • JavaScript ...
MapReduce• For more involved queries • Specify input keys • Process data in “map” and   “reduce” functions   • JavaScript ...
Lab: Querying
Lab: Walk Links
Lab: Walk Links• Store an object with a Link
Lab: Walk Links• Store an object with a Link• Store the target of the Link
Lab: Walk Links• Store an object with a Link• Store the target of the Link• Walk from one to the other
Lab: Store Links
Lab: Store Links$ ./store-linked.sh
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Sto...
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Sto...
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Sto...
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Sto...
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Sto...
Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Sto...
Lab: Walk Links (1/3)
Lab: Walk Links (1/3)$ ./walk-linked.sh
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge...
Lab: Walk Links (2/3)
Lab: Walk Links (2/3)Enter the origin key: seanEnter the link spec (default: _,_,_): surge,_,_
Lab: Walk Links (2/3)Enter the origin key: seanEnter the link spec (default: _,_,_): surge,_,_> GET /riak/surge/sean/surge...
Lab: Walk Links (2/3)Enter the origin key: seanEnter the link spec (default: _,_,_): surge,_,_> GET /riak/surge/sean/surge...
Lab: Walk Links (3/3)
Lab: Walk Links (3/3)Enter the origin key: seanEnter the link spec (default: _,_,_): foo,_,_
Lab: Walk Links (3/3)Enter the origin key: seanEnter the link spec (default: _,_,_): foo,_,_...> GET /riak/surge/sean/foo,...
Lab: Secondary   Indexes
Lab: Secondary      Indexes• Create some objects with indexes
Lab: Secondary      Indexes• Create some objects with indexes• Query the index for their keys
Lab: Secondary      Indexes• Create some objects with indexes• Query the index for their keys• Query input to MapReduce
Lab: Create 2I Objects
Lab: Create 2I Objects$ ./store-indexed.sh
Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40
Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40* About to connect() to localhost port ...
Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40       key* About to connect() to local...
Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40* About to connect() to localhost port ...
Lab: Query 2I (1/2)
Lab: Query 2I (1/2)$ ./query-indexed.sh
Lab: Query 2I (1/2)$ ./query-indexed.shQuery on range or equality? (r/e) eIndex value: 10
Lab: Query 2I (1/2)$ ./query-indexed.shQuery on range or equality? (r/e) eIndex value: 10* About to connect() to localhost...
Lab: Query 2I (2/2)
Lab: Query 2I (2/2)Query on range or equality? (r/e) rRange start: 5Range end: 15
Lab: Query 2I (2/2)Query on range or equality? (r/e) rRange start: 5Range end: 15* About to connect() to localhost port 80...
Lab: 2I MapReduce
Lab: 2I MapReduce$ ./mapred-indexed.sh
Lab: 2I MapReduce$ ./mapred-indexed.shQuery on range or equality? (r/e) rRange start: 35Range end: 40
Lab: 2I MapReduce$ ./mapred-indexed.shQuery on range or equality? (r/e) rRange start: 35Range end: 40Executing MapReduce:{...
Lab: 2I MapReduce$ ./mapred-indexed.shQuery on range or equality? (r/e) rRange start: 35Range end: 40Executing MapReduce:{...
Riak Operations
Configuration
File Locations
File Locations• Configuration
File Locations• Configuration  •   /etc/riak
File Locations• Configuration  •   /etc/riak• Binaries
File Locations• Configuration  •   /etc/riak• Binaries  •   /usr/sbin/riak
File Locations• Configuration  •   /etc/riak• Binaries  •   /usr/sbin/riak  •   /usr/sbin/riak-admin
File Locations• Configuration  •   /etc/riak• Binaries  •   /usr/sbin/riak  •   /usr/sbin/riak-admin  •   /usr/[lib,lib64]/...
File Locations• Configuration                • Logs  •   /etc/riak• Binaries  •   /usr/sbin/riak  •   /usr/sbin/riak-admin ...
File Locations• Configuration                • Logs  •   /etc/riak                 •   /var/log/riak• Binaries  •   /usr/sb...
File Locations• Configuration                • Logs  •   /etc/riak                 •   /var/log/riak• Binaries             ...
File Locations• Configuration                • Logs  •   /etc/riak                 •   /var/log/riak• Binaries             ...
File Locations• Configuration                • Logs  •   /etc/riak                 •   /var/log/riak• Binaries             ...
File Locations• Configuration                • Logs  •   /etc/riak                 •   /var/log/riak• Binaries             ...
/etc/riak/vm.args
/etc/riak/vm.args• Overview
/etc/riak/vm.args• Overview • Erlang VM configuration settings.
/etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings
/etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name.
/etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name. • Security cookie.
/etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name. • Security cookie.• Docume...
/etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name. • Security cookie.• Docume...
/etc/riak/vm.args
/etc/riak/vm.args• Node Name & Security Cookie  ## Name of the riak node
/etc/riak/vm.args• Node Name & Security Cookie  ## Name of the riak node  -name riak@127.0.0.1
/etc/riak/vm.args• Node Name & Security Cookie  ## Name of the riak node  -name riak@127.0.0.1  ## Cookie for distributed ...
/etc/riak/vm.args• Node Name & Security Cookie  ## Name of the riak node  -name riak@127.0.0.1  ## Cookie for distributed ...
/etc/riak/app.config
/etc/riak/               app.config•   Overview
/etc/riak/               app.config•   Overview    •   Riak (and dependency) configuration settings.
/etc/riak/               app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/                app.config•   Overview    •   Riak (and dependency) configuration settings.•   Important Settings ...
/etc/riak/app.config
/etc/riak/             app.config• Storage Engine   %% Storage_backend specifies the Erlang module defining the storage
/etc/riak/             app.config• Storage Engine   %% Storage_backend specifies the Erlang module defining the storage   %% ...
/etc/riak/             app.config• Storage Engine    %% Storage_backend specifies the Erlang module defining the storage   %%...
/etc/riak/             app.config• Storage Engine    %% Storage_backend specifies the Erlang module defining the storage   %%...
/etc/riak/             app.config• Storage Engine    %% Storage_backend specifies the Erlang module defining the storage   %%...
/etc/riak/             app.config• Storage Engine    %% Storage_backend specifies the Erlang module defining the storage   %%...
/etc/riak/             app.config• Storage Engine    %% Storage_backend specifies the Erlang module defining the storage   %%...
/etc/riak/             app.config• Storage Engine    %% Storage_backend specifies the Erlang module defining the storage   %%...
/etc/riak/          app.config%% Bitcask Config{bitcask, [            {data_root, "data/bitcask"}          ]},{eleveldb, [ ...
/etc/riak/          app.config• File Locations%% Bitcask Config{bitcask, [            {data_root, "data/bitcask"}          ...
/etc/riak/          app.config• File Locations%% Bitcask Config{bitcask, [            {data_root, "data/bitcask"}          ...
/etc/riak/          app.config• File Locations%% Bitcask Config{bitcask, [            {data_root, "data/bitcask"}          ...
/etc/riak/          app.config• File Locations%% Bitcask Config{bitcask, [            {data_root, "data/bitcask"}          ...
/etc/riak/          app.config• File Locations%% Bitcask Config{bitcask, [            {data_root, "data/bitcask"}          ...
/etc/riak/             app.config%% http is a list of IP addresses and TCP ports that theRiak%% HTTP interface will bind.{h...
/etc/riak/             app.config• HTTP & Protocol Buffers%% http is a list of IP addresses and TCP ports that theRiak%% HT...
Securing Riak
Securing Riak•   Overview
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.    •   Ri...
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.    •   Ri...
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.    •   Ri...
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.    •   Ri...
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.    •   Ri...
Securing Riak•   Overview    •   Erlang has a “cookie” for “security.”        •   Do not rely on it. Plain text.    •   Ri...
Load Balancing
Load Balancing•   Overview
Load Balancing•   Overview    •   Any node can handle any request.
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.•   Load Balancing HTTP
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.•   Load Balancing HTT...
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.•   Load Balancing HTT...
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.•   Load Balancing HTT...
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.•   Load Balancing HTT...
Load Balancing•   Overview    •   Any node can handle any request.    •   Makes load balancing easy.•   Load Balancing HTT...
Storage Backends
There are 5 to choose from
There are 5 to      choose from• Bitcask
There are 5 to      choose from• Bitcask• Innostore
There are 5 to      choose from• Bitcask• Innostore• LevelDB
There are 5 to      choose from• Bitcask• Innostore• LevelDB• Memory
There are 5 to      choose from• Bitcask• Innostore• LevelDB• Memory• Multi
Bitcask
Bitcask• A fast, append-only key-value store
Bitcask• A fast, append-only key-value store• In memory key lookup table  (key_dir)
Bitcask• A fast, append-only key-value store• In memory key lookup table  (key_dir)• Closed files are immutable
Bitcask• A fast, append-only key-value store• In memory key lookup table  (key_dir)• Closed files are immutable• Merging cl...
Bitcask• A fast, append-only key-value store• In memory key lookup table  (key_dir)• Closed files are immutable• Merging cl...
Innostore
Innostore• Based on Embedded InnoDB
Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage
Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage• Similar characterics to the MySQL  InnoDB plugin
Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage• Similar characterics to the MySQL  InnoDB plugin...
Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage• Similar characterics to the MySQL  InnoDB plugin...
LevelDB
LevelDB• A Google developed key-value store
LevelDB• A Google developed key-value store• Append-only
LevelDB• A Google developed key-value store• Append-only• Multiple levels of SSTable-like  structures
LevelDB• A Google developed key-value store• Append-only• Multiple levels of SSTable-like  structures• BSD license
Cluster Capacity   Planning
Cluster Capacity   Planning
Cluster Capacity       Planning•   Overview
Cluster Capacity       Planning•   Overview    •   There are MANY variables that affect performance.
Cluster Capacity       Planning•   Overview    •   There are MANY variables that affect performance.    •   Safest route i...
Cluster Capacity       Planning•   Overview    •   There are MANY variables that affect performance.    •   Safest route i...
Cluster Capacity       Planning•   Overview    •   There are MANY variables that affect performance.    •   Safest route i...
Cluster Capacity       Planning•   Overview    •   There are MANY variables that affect performance.    •   Safest route i...
Cluster Capacity       Planning•   Overview    •   There are MANY variables that affect performance.    •   Safest route i...
Things to Consider      (1/3)
Things to Consider      (1/3)•   Virtualization
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (1/3)•   Virtualization    •   Riak’s workload is I/O bound    •   Run on bare metal for best perf...
Things to Consider      (2/3)
Things to Consider      (2/3)•   “Randomness” of Reads /    Writes
Things to Consider      (2/3)•   “Randomness” of Reads /    Writes    •   Recently accessed        objects are cached in  ...
Things to Consider      (2/3)•   “Randomness” of Reads /    Writes    •   Recently accessed        objects are cached in  ...
Things to Consider      (2/3)•   “Randomness” of Reads /    Writes    •   Recently accessed        objects are cached in  ...
Things to Consider      (2/3)•   “Randomness” of Reads /    Writes    •   Recently accessed        objects are cached in  ...
Things to Consider      (2/3)•   “Randomness” of Reads /      •   Disk Speed    Writes    •   Recently accessed        obj...
Things to Consider      (2/3)•   “Randomness” of Reads /      •   Disk Speed    Writes                                    ...
Things to Consider      (2/3)•   “Randomness” of Reads /      •   Disk Speed    Writes                                    ...
Things to Consider      (2/3)•   “Randomness” of Reads /      •   Disk Speed    Writes                                    ...
Things to Consider      (2/3)•   “Randomness” of Reads /      •   Disk Speed    Writes                                    ...
Things to Consider      (3/3)
Things to Consider      (3/3)•   Features
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Things to Consider      (3/3)•   Features    •   RiakKV vs. Riak 2I vs. Riak Search•   Backends    •   Bitcask vs. Innosto...
Performance  Tuning
Performance          Tuning•   General Tips
Performance            Tuning•   General Tips    •   Start with a “DB” Like Machine Profile
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance              Tuning•   General Tips    •   Start with a “DB” Like Machine Profile        •   Multi-core 64-bit ...
Performance  Tuning
Performance          Tuning•   Bitcask Backend
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance            Tuning•   Bitcask Backend    •   KeyDir is in Memory    •   Filesystem Cache (Access Profile Conside...
Performance  Tuning
Performance          Tuning•   Innostore Backend
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size    •   o_direct
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size    •   o_direc...
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size    •   o_direc...
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size    •   o_direc...
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size    •   o_direc...
Performance            Tuning•   Innostore Backend    •   Similar to MySQL + InnoDB    •   buffer pool size    •   o_direc...
basho_bench
basho_bench•   What Does It Do?
basho_bench•   What Does It Do?    •   Generates load against a Riak cluster. Logs and graphs results.
basho_bench•   What Does It Do?    •   Generates load against a Riak cluster. Logs and graphs results.    •   Intended for...
basho_bench•   What Does It Do?    •   Generates load against a Riak cluster. Logs and graphs results.    •   Intended for...
basho_bench•   What Does It Do?    •   Generates load against a Riak cluster. Logs and graphs results.    •   Intended for...
basho_bench•   What Does It Do?    •   Generates load against a Riak cluster. Logs and graphs results.    •   Intended for...
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
Upcoming SlideShare
Loading in...5
×

Riak Training Session — Surge 2011

14,068

Published on

ion

Published in: Technology, Business
1 Comment
52 Likes
Statistics
Notes
No Downloads
Views
Total Views
14,068
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
911
Comments
1
Likes
52
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Think of it like a big hash-table\n
  • Think of it like a big hash-table\n
  • Think of it like a big hash-table\n
  • Think of it like a big hash-table\n
  • Think of it like a big hash-table\n
  • Think of it like a big hash-table\n
  • Think of it like a big hash-table\n
  • X = throughput, compute power for MapReduce, storage, lower latency\n
  • X = throughput, compute power for MapReduce, storage, lower latency\n
  • X = throughput, compute power for MapReduce, storage, lower latency\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Consistent hashing means:\n1) large, fixed-size key-space\n2) no rehashing of keys - always hash the same way\n
  • Consistent hashing means:\n1) large, fixed-size key-space\n2) no rehashing of keys - always hash the same way\n
  • Consistent hashing means:\n1) large, fixed-size key-space\n2) no rehashing of keys - always hash the same way\n
  • Consistent hashing means:\n1) large, fixed-size key-space\n2) no rehashing of keys - always hash the same way\n
  • Consistent hashing means:\n1) large, fixed-size key-space\n2) no rehashing of keys - always hash the same way\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • 1) Client requests a key\n2) Get handler starts up to service the request\n3) Hashes key to its owner partitions (N=3)\n4) Sends similar &amp;#x201C;get&amp;#x201D; request to those partitions\n5) Waits for R replies that concur (R=2)\n6) Resolves the object, replies to client\n7) Third reply may come back at any time, but FSM replies as soon as quorum is satisfied/violated\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript of "Riak Training Session — Surge 2011"

    1. 1. Riak Tutorial Sean Cribbs & Ian Plosker Basho Technologiesbasho
    2. 2. What is Riak?
    3. 3. • Key-Value store (plus extras)
    4. 4. • Key-Value store (plus extras)• Distributed, horizontally scalable
    5. 5. • Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant
    6. 6. • Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant• Highly-available
    7. 7. • Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant• Highly-available• Built for the Web
    8. 8. • Key-Value store (plus extras)• Distributed, horizontally scalable• Fault-tolerant• Highly-available• Built for the Web• Inspired by Amazon’s Dynamo
    9. 9. Key-Value
    10. 10. Key-Value• Simple operations - get, put, delete
    11. 11. Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)
    12. 12. Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras
    13. 13. Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce
    14. 14. Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce • Links
    15. 15. Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce • Links • Full-text search (new, optional)
    16. 16. Key-Value• Simple operations - get, put, delete• Value is mostly opaque (some metadata)• Extras • MapReduce • Links • Full-text search (new, optional) • Secondary Indexes
    17. 17. Distributed &Horizontally
    18. 18. Distributed & Horizontally• Default configuration is in a cluster
    19. 19. Distributed & Horizontally• Default configuration is in a cluster• Load and data are spread evenly
    20. 20. Distributed & Horizontally• Default configuration is in a cluster• Load and data are spread evenly• Add more nodes to get more X
    21. 21. Fault-Tolerant
    22. 22. Fault-Tolerant• All nodes participate equally - no SPOF
    23. 23. Fault-Tolerant• All nodes participate equally - no SPOF• All data is replicated
    24. 24. Fault-Tolerant• All nodes participate equally - no SPOF• All data is replicated• Cluster transparently survives...
    25. 25. Fault-Tolerant• All nodes participate equally - no SPOF• All data is replicated• Cluster transparently survives... • node failure
    26. 26. Fault-Tolerant• All nodes participate equally - no SPOF• All data is replicated• Cluster transparently survives... • node failure • network partitions
    27. 27. Fault-Tolerant• All nodes participate equally - no SPOF• All data is replicated• Cluster transparently survives... • node failure • network partitions• Built on Erlang/OTP (designed for FT)
    28. 28. Highly-Available
    29. 29. Highly-Available• Any node can serve client requests
    30. 30. Highly-Available• Any node can serve client requests• Fallbacks are used when nodes are down
    31. 31. Highly-Available• Any node can serve client requests• Fallbacks are used when nodes are down• Always accepts read and write requests
    32. 32. Highly-Available• Any node can serve client requests• Fallbacks are used when nodes are down• Always accepts read and write requests• Per-request quorums
    33. 33. Built for the Web
    34. 34. Built for the Web• HTTP is default (but not only) interface
    35. 35. Built for the Web• HTTP is default (but not only) interface• Does HTTP/REST well (see Webmachine)
    36. 36. Built for the Web• HTTP is default (but not only) interface• Does HTTP/REST well (see Webmachine)• Plays well with HTTP infrastructure - reverse proxy caches, load balancers, web servers
    37. 37. Built for the Web• HTTP is default (but not only) interface• Does HTTP/REST well (see Webmachine)• Plays well with HTTP infrastructure - reverse proxy caches, load balancers, web servers• Suitable to many web applications
    38. 38. Inspired byAmazon’s Dynamo
    39. 39. Inspired byAmazon’s Dynamo• Masterless, peer-coordinated replication
    40. 40. Inspired byAmazon’s Dynamo• Masterless, peer-coordinated replication• Consistent hashing
    41. 41. Inspired byAmazon’s Dynamo• Masterless, peer-coordinated replication• Consistent hashing• Eventually consistent
    42. 42. Inspired byAmazon’s Dynamo• Masterless, peer-coordinated replication• Consistent hashing• Eventually consistent• Quorum reads and writes
    43. 43. Inspired byAmazon’s Dynamo• Masterless, peer-coordinated replication• Consistent hashing• Eventually consistent• Quorum reads and writes• Anti-entropy: read repair, hinted handoff
    44. 44. Lab: Riak Basics
    45. 45. Installing Riak
    46. 46. Installing Riak• Download from our local server
    47. 47. Installing Riak• Download from our local server• Download a package: http://downloads.basho.com/riak/ CURRENT
    48. 48. Installing Riak• Download from our local server• Download a package: http://downloads.basho.com/riak/ CURRENT• Clone & build (need git and Erlang) : $ git clone git://github.com/basho/riak.git $ make all devrel
    49. 49. You need `curl` apt-get install curl yum install curl
    50. 50. Start the Cluster
    51. 51. Start the Cluster$ surge-start.sh
    52. 52. Start the Cluster$ surge-start.shdev1/bin/riak startdev2/bin/riak startdev3/bin/riak startdev4/bin/riak start
    53. 53. Start the Cluster$ surge-start.shdev1/bin/riak startdev2/bin/riak startdev3/bin/riak startdev4/bin/riak startdev2/bin/riak-admin join dev1Sent join request to dev1dev3/bin/riak-admin join dev1Sent join request to dev1dev4/bin/riak-admin join dev1Sent join request to dev1
    54. 54. Check its status
    55. 55. Check its status$ dev1/bin/riak-admin member_status$ dev1/bin/riak-admin ring_status$ dev1/bin/riak-admin status
    56. 56. PUT
    57. 57. PUT$ surge-put.sh
    58. 58. PUT$ surge-put.shEnter a key: your-key
    59. 59. PUT$ surge-put.shEnter a key: your-keyEnter a value: your-value
    60. 60. PUT$ surge-put.shEnter a key: your-keyEnter a value: your-valueEnter a write quorum value:
    61. 61. PUT$ surge-put.shEnter a key: your-keyEnter a value: your-valueEnter a write quorum value:* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> PUT /riak/surge/your-key?returnbody=true&w=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*> Content-Type: text/plain> Content-Length: 10>
    62. 62. PUT$ surge-put.shEnter a key: your-keyEnter a value: your-valueEnter a write quorum value:* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> PUT /riak/surge/your-key?returnbody=true&w=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*> Content-Type: text/plain> Content-Length: 10>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1qb3LYEpkzGNleL/k23G+LAA=< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Date: Tue, 27 Sep 2011 19:21:08 GMT< Content-Type: text/plain< Content-Length: 10<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0your-value
    63. 63. GET
    64. 64. GET$ surge-get.sh
    65. 65. GET$ surge-get.shEnter a key: your-key
    66. 66. GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):
    67. 67. GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>
    68. 68. GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1qb3LYEpkzGNleL/k23G+LAA=< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Last-Modified: Tue, 27 Sep 2011 19:01:45 GMT< ETag: "51h3q7RjTNaHWYpO4P0MJj"< Date: Tue, 27 Sep 2011 19:31:01 GMT< Content-Type: text/plain< Content-Length: 10<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0your-value
    69. 69. GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1qb3LYEpkzGNleL/k23G+LAA=< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Last-Modified: Tue, 27 Sep 2011 19:01:45 GMT< ETag: "51h3q7RjTNaHWYpO4P0MJj"< Date: Tue, 27 Sep 2011 19:31:01 GMT< Content-Type: text/plain vector clock< Content-Length: 10<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0your-value
    70. 70. GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1qb3LYEpkzGNleL/k23G+LAA=< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Last-Modified: Tue, 27 Sep 2011 19:01:45 GMT< ETag: "51h3q7RjTNaHWYpO4P0MJj"< Date: Tue, 27 Sep 2011 19:31:01 GMT< Content-Type: text/plain< Content-Length: 10<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0your-value caching headers
    71. 71. GET$ surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1qb3LYEpkzGNleL/k23G+LAA=< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Last-Modified: Tue, 27 Sep 2011 19:01:45 GMT< ETag: "51h3q7RjTNaHWYpO4P0MJj"< Date: Tue, 27 Sep 2011 19:31:01 GMT< Content-Type: text/plain< Content-Length: 10<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0your-value content-type
    72. 72. DELETE
    73. 73. DELETE$ surge-delete.sh
    74. 74. DELETE$ surge-delete.shType a key: your-key
    75. 75. DELETE$ surge-delete.shType a key: your-key* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> DELETE /riak/surge/your-key HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>
    76. 76. DELETE$ surge-delete.shType a key: your-key* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> DELETE /riak/surge/your-key HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>< HTTP/1.1 204 No Content< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Date: Tue, 27 Sep 2011 19:33:59 GMT< Content-Type: text/plain< Content-Length: 0<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0
    77. 77. Kill two nodes
    78. 78. Kill two nodes$ dev4/bin/riak stop
    79. 79. Kill two nodes$ dev4/bin/riak stopok
    80. 80. Kill two nodes$ dev4/bin/riak stopok$ dev3/bin/riak stop
    81. 81. Kill two nodes$ dev4/bin/riak stopok$ dev3/bin/riak stopok
    82. 82. Try GETting your key
    83. 83. Try GETting your key$ . ~/Desktop/surge/surge-get.sh
    84. 84. Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-key
    85. 85. Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):
    86. 86. Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>
    87. 87. Try GETting your key$ . ~/Desktop/surge/surge-get.shEnter a key: your-keyEnter a read quorum value (default: quorum):* About to connect() to 127.0.0.1 port 8091 (#0)* Trying 127.0.0.1... connected* Connected to 127.0.0.1 (127.0.0.1) port 8091 (#0)> GET /riak/surge/your-key?r=quorum HTTP/1.1> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8rzlib/1.2.5> Host: 127.0.0.1:8091> Accept: */*>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1qb3LYEpkzGNleL/k23G+LAA=< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Last-Modified: Tue, 27 Sep 2011 19:50:07 GMT< ETag: "51h3q7RjTNaHWYpO4P0MJj"< Date: Tue, 27 Sep 2011 19:51:10 GMT< Content-Type: text/plain< Content-Length: 10<* Connection #0 to host 127.0.0.1 left intact* Closing connection #0your-value
    88. 88. Riak Architecture
    89. 89. Key-Value Data Model
    90. 90. Key-Value Data Model• All data (objects) are referenced by keys mark beth sean
    91. 91. Key-Value Data Model n_val allow_mult quora• All data (objects) are people referenced by keys mark• Keys are grouped into buckets beth sean
    92. 92. Key-Value Data Model n_val allow_mult quora• All data (objects) are people referenced by keys mark• Keys are grouped into buckets beth sean• Simple operations: get, put, delete
    93. 93. Key-Value Data Model• All data (objects) are referenced by keys people/beth• Keys are grouped into vclock: ... content-type: text/html buckets last-mod: 20101108T... links: ...• Simple operations: get, put, delete <html><head>....• Object is composed of metadata and value
    94. 94. Consistent Hashing & The Ring
    95. 95. Consistent Hashing & The Ring• 160-bit integer keyspace 0 2 160/4 2 160/2
    96. 96. Consistent Hashing & The Ring• 160-bit integer keyspace 0• Divided into fixed number of evenly-sized partitions 32 partitions 2 160/4 2 160/2
    97. 97. Consistent Hashing & The Ring node 0 node 1• 160-bit integer node 2 keyspace 0 node 3• Divided into fixed number of evenly-sized partitions 32 partitions 2 160/4• Partitions are claimed by nodes in the cluster 2 160/2
    98. 98. Consistent Hashing & The Ring node 0 node 1• 160-bit integer node 2 keyspace node 3• Divided into fixed number of evenly-sized partitions• Partitions are claimed by nodes in the cluster• Replicas go to the N partitions following the key
    99. 99. Consistent Hashing & The Ring node 0 node 1• 160-bit integer node 2 keyspace node 3• Divided into fixed number of evenly-sized partitions• Partitions are claimed by N=3 nodes in the cluster• Replicas go to the N partitions following the key hash(“conferences/surge”)
    100. 100. Request Quorums
    101. 101. Request Quorums• Every request contacts all replicas of key
    102. 102. Request Quorums• Every request contacts all replicas of key• N - number of replicas (default 3)
    103. 103. Request Quorums• Every request contacts all replicas of key• N - number of replicas (default 3)• R - read quorum
    104. 104. Request Quorums• Every request contacts all replicas of key• N - number of replicas (default 3)• R - read quorum• W - write quorum
    105. 105. Vector Clocks
    106. 106. Vector Clocks• Every node has an ID (changed in 1.0)
    107. 107. Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every “put” or “delete” request
    108. 108. Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every “put” or “delete” request• Riak tracks history of updates
    109. 109. Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every “put” or “delete” request• Riak tracks history of updates • Auto-resolves stale versions
    110. 110. Vector Clocks• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every “put” or “delete” request• Riak tracks history of updates • Auto-resolves stale versions • Lets you decide conflicts
    111. 111. Vector Clockshttp://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/• Every node has an ID (changed in 1.0)• Send last-seen vector clock in every “put” or “delete” request• Riak tracks history of updates • Auto-resolves stale versions • Lets you decide conflicts
    112. 112. Hinted Handoff
    113. 113. Hinted Handoff• Node fails X X X X X X X X
    114. 114. Hinted Handoff• Node fails X X• Requests go to fallback X X X X X X hash(“conferences/surge”)
    115. 115. Hinted Handoff• Node fails• Requests go to fallback• Node comes back hash(“conferences/surge”)
    116. 116. Hinted Handoff• Node fails• Requests go to fallback• Node comes back• “Handoff” - data returns to recovered node hash(“conferences/surge”)
    117. 117. Hinted Handoff• Node fails• Requests go to fallback• Node comes back• “Handoff” - data returns to recovered node• Normal operations resume hash(“conferences/surge”)
    118. 118. Anatomy of a Request get(“conferences/surge”)
    119. 119. Anatomy of a Request get(“conferences/surge”)client Riak
    120. 120. Anatomy of a Request get(“conferences/surge”)client Riak Get Handler (FSM)
    121. 121. Anatomy of a Request get(“conferences/surge”)client Riak hash(“conferences/oredev”) Get Handler (FSM) == 10, 11, 12
    122. 122. Anatomy of a Request get(“conferences/surge”) client Riak hash(“conferences/oredev”) Get Handler (FSM) == 10, 11, 12 Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 The Ring
    123. 123. Anatomy of a Request get(“conferences/surge”) client Riak Get Handler (FSM) get(“conferences/oredev”) Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 The Ring
    124. 124. Anatomy of a Request get(“conferences/surge”) client Riak R=2 Get Handler (FSM) Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 The Ring
    125. 125. Anatomy of a Request get(“conferences/surge”) client Riak R=2 v1 Get Handler (FSM) Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 The Ring
    126. 126. Anatomy of a Request get(“conferences/surge”) client RiakR=2 v1 v2 Get Handler (FSM)
    127. 127. Anatomy of a Request get(“conferences/surge”) client v2 RiakR=2 v2 Get Handler (FSM)
    128. 128. Anatomy of a Request get(“conferences/surge”) v2
    129. 129. Read Repair get(“conferences/surge”) client v2 Riak R=2 v1 v2 Get Handler (FSM) Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 v1 v2
    130. 130. Read Repair get(“conferences/surge”) client v2 Riak R=2 v2 Get Handler (FSM) Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 v1 v2
    131. 131. Read Repair get(“conferences/surge”) client v2 Riak R=2 v2 Get Handler (FSM) Coordinating node Cluster6 7 8 9 10 11 12 13 14 15 16 v1 v1 v2
    132. 132. Read Repair get(“conferences/surge”) client v2 Riak R=2 v2 Get Handler (FSM) Coordinating node Cluster v2 v26 7 8 9 10 11 12 13 14 15 16 v2 v2 v2
    133. 133. Riak ArchitectureErlang/OTP Runtime
    134. 134. Riak ArchitectureErlang/OTP Runtime Riak KV
    135. 135. Riak ArchitectureErlang/OTP Runtime Client APIs Riak KV
    136. 136. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Riak KV
    137. 137. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Riak KV
    138. 138. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Riak KV
    139. 139. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination Riak KV
    140. 140. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce Riak KV
    141. 141. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce Riak Core Riak KV
    142. 142. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing Riak Core Riak KV
    143. 143. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing membership Riak Core Riak KV
    144. 144. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff membership Riak Core Riak KV
    145. 145. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff membership node-liveness Riak Core Riak KV
    146. 146. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness Riak Core Riak KV
    147. 147. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core Riak KV
    148. 148. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master Riak KV
    149. 149. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes Riak KV
    150. 150. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes storage backend Riak KV
    151. 151. Riak ArchitectureErlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes Workers storage backend Riak KV
    152. 152. Modeling & Querying
    153. 153. Application Design
    154. 154. Application Design• No intrinsic schema
    155. 155. Application Design• No intrinsic schema• Your application defines the structure and semantics
    156. 156. Application Design• No intrinsic schema• Your application defines the structure and semantics• Your application resolves conflicts (if you care)
    157. 157. Modeling Tools
    158. 158. Modeling Tools• Key-Value
    159. 159. Modeling Tools• Key-Value• Links
    160. 160. Modeling Tools• Key-Value• Links• Full-text Search
    161. 161. Modeling Tools• Key-Value• Links• Full-text Search• Secondary Indexes (2I)
    162. 162. Modeling Tools• Key-Value• Links• Full-text Search• Secondary Indexes (2I)• MapReduce
    163. 163. Key-Value
    164. 164. Key-Value• Content-Types
    165. 165. Key-Value• Content-Types• Denormalize
    166. 166. Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys
    167. 167. Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys • Composites
    168. 168. Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys • Composites • Time-boxing
    169. 169. Key-Value• Content-Types• Denormalize• Meaningful or “guessable” keys • Composites • Time-boxing• References (value is a key or list)
    170. 170. Links
    171. 171. Links• Lightweight relationships, like <a>
    172. 172. Links• Lightweight relationships, like <a>• Includes a “tag”
    173. 173. Links• Lightweight relationships, like <a>• Includes a “tag”• Built-in traversal op (“walking”) GET /riak/b/k/[bucket],[tag],[keep]
    174. 174. Links• Lightweight relationships, like <a>• Includes a “tag”• Built-in traversal op (“walking”) GET /riak/b/k/[bucket],[tag],[keep]• Limited in number (part of meta)
    175. 175. Full-text Search
    176. 176. Full-text Search• Designed for searching prose
    177. 177. Full-text Search• Designed for searching prose• Lucene/Solr-like query interface
    178. 178. Full-text Search• Designed for searching prose• Lucene/Solr-like query interface• Automatically index K/V values
    179. 179. Full-text Search• Designed for searching prose• Lucene/Solr-like query interface• Automatically index K/V values• Input to MapReduce
    180. 180. Full-text Search• Designed for searching prose• Lucene/Solr-like query interface• Automatically index K/V values• Input to MapReduce• Customizable index schemas
    181. 181. Secondary Indexes
    182. 182. Secondary Indexes• Defined as metadata
    183. 183. Secondary Indexes• Defined as metadata• Two index types: _int and _bin
    184. 184. Secondary Indexes• Defined as metadata• Two index types: _int and _bin• Two query types: equal and range
    185. 185. Secondary Indexes• Defined as metadata• Two index types: _int and _bin• Two query types: equal and range• Input to MapReduce
    186. 186. MapReduce
    187. 187. MapReduce• For more involved queries
    188. 188. MapReduce• For more involved queries • Specify input keys
    189. 189. MapReduce• For more involved queries • Specify input keys • Process data in “map” and “reduce” functions
    190. 190. MapReduce• For more involved queries • Specify input keys • Process data in “map” and “reduce” functions • JavaScript or Erlang
    191. 191. MapReduce• For more involved queries • Specify input keys • Process data in “map” and “reduce” functions • JavaScript or Erlang• Not tuned for batch processing
    192. 192. Lab: Querying
    193. 193. Lab: Walk Links
    194. 194. Lab: Walk Links• Store an object with a Link
    195. 195. Lab: Walk Links• Store an object with a Link• Store the target of the Link
    196. 196. Lab: Walk Links• Store an object with a Link• Store the target of the Link• Walk from one to the other
    197. 197. Lab: Store Links
    198. 198. Lab: Store Links$ ./store-linked.sh
    199. 199. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker
    200. 200. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Storing the origin **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/sean HTTP/1.1> Link: </riak/surge/ian>;riaktag="coworker"...< HTTP/1.1 204 No Content...
    201. 201. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Storing the origin **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/sean HTTP/1.1> Link: </riak/surge/ian>;riaktag="coworker"...< HTTP/1.1 204 No Content...
    202. 202. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Storing the origin **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/sean HTTP/1.1> Link: </riak/surge/ian>;riaktag="coworker"...< HTTP/1.1 204 No Content...
    203. 203. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Storing the origin **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/sean HTTP/1.1> Link: </riak/surge/ian>;riaktag="coworker"...< HTTP/1.1 204 No Content...
    204. 204. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Storing the origin **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/sean HTTP/1.1> Link: </riak/surge/ian>;riaktag="coworker"...< HTTP/1.1 204 No Content...*** Storing the target **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/ian HTTP/1.1...< HTTP/1.1 204 No Content
    205. 205. Lab: Store Links$ ./store-linked.shEnter the origin key: seanEnter the target key: ianEnter the links tag: coworker*** Storing the origin **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/sean HTTP/1.1> Link: </riak/surge/ian>;riaktag="coworker"...< HTTP/1.1 204 No Content...*** Storing the target **** About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/ian HTTP/1.1...< HTTP/1.1 204 No Content
    206. 206. Lab: Walk Links (1/3)
    207. 207. Lab: Walk Links (1/3)$ ./walk-linked.sh
    208. 208. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):
    209. 209. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...
    210. 210. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...
    211. 211. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...
    212. 212. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Expires: Tue, 27 Sep 2011 20:51:37 GMT< Date: Tue, 27 Sep 2011 20:41:37 GMT< Content-Type: multipart/mixed; boundary=J6O2SZfLfSEdAv5dv3mttR7AVOF< Content-Length: 438<
    213. 213. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Expires: Tue, 27 Sep 2011 20:51:37 GMT< Date: Tue, 27 Sep 2011 20:41:37 GMT< Content-Type: multipart/mixed; boundary=J6O2SZfLfSEdAv5dv3mttR7AVOF< Content-Length: 438<
    214. 214. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Expires: Tue, 27 Sep 2011 20:51:37 GMT< Date: Tue, 27 Sep 2011 20:41:37 GMT< Content-Type: multipart/mixed; boundary=J6O2SZfLfSEdAv5dv3mttR7AVOF< Content-Length: 438<--J6O2SZfLfSEdAv5dv3mttR7AVOFContent-Type: multipart/mixed; boundary=C1NUMpwbmSvb7CbqEAy1KdYRiUH--C1NUMpwbmSvb7CbqEAy1KdYRiUHX-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/k1GU/KYEpkymNl0Nzw7ThfFgA=Location: /riak/surge/ianContent-Type: text/plainLink: </riak/surge>; rel="up"Etag: 51h3q7RjTNaHWYpO4P0MJjLast-Modified: Tue, 27 Sep 2011 20:38:01 GMTtarget--C1NUMpwbmSvb7CbqEAy1KdYRiUH----J6O2SZfLfSEdAv5dv3mttR7AVOF--
    215. 215. Lab: Walk Links (1/3)$ ./walk-linked.shEnter the origin key: seanEnter the link spec (default: _,_,_):...> GET /riak/surge/sean/_,_,_ HTTP/1.1...< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Expires: Tue, 27 Sep 2011 20:51:37 GMT< Date: Tue, 27 Sep 2011 20:41:37 GMT< Content-Type: multipart/mixed; boundary=J6O2SZfLfSEdAv5dv3mttR7AVOF< Content-Length: 438<--J6O2SZfLfSEdAv5dv3mttR7AVOFContent-Type: multipart/mixed; boundary=C1NUMpwbmSvb7CbqEAy1KdYRiUH--C1NUMpwbmSvb7CbqEAy1KdYRiUHX-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/k1GU/KYEpkymNl0Nzw7ThfFgA=Location: /riak/surge/ianContent-Type: text/plainLink: </riak/surge>; rel="up"Etag: 51h3q7RjTNaHWYpO4P0MJjLast-Modified: Tue, 27 Sep 2011 20:38:01 GMTtarget--C1NUMpwbmSvb7CbqEAy1KdYRiUH----J6O2SZfLfSEdAv5dv3mttR7AVOF--
    216. 216. Lab: Walk Links (2/3)
    217. 217. Lab: Walk Links (2/3)Enter the origin key: seanEnter the link spec (default: _,_,_): surge,_,_
    218. 218. Lab: Walk Links (2/3)Enter the origin key: seanEnter the link spec (default: _,_,_): surge,_,_> GET /riak/surge/sean/surge,_,_ HTTP/1.1...
    219. 219. Lab: Walk Links (2/3)Enter the origin key: seanEnter the link spec (default: _,_,_): surge,_,_> GET /riak/surge/sean/surge,_,_ HTTP/1.1...>< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Expires: Tue, 27 Sep 2011 20:52:53 GMT< Date: Tue, 27 Sep 2011 20:42:53 GMT< Content-Type: multipart/mixed; boundary=CsxMIsO9tfadrRJ7EQ3XL2ivQ4f< Content-Length: 438<--CsxMIsO9tfadrRJ7EQ3XL2ivQ4fContent-Type: multipart/mixed; boundary=I3U33LkzkqJi6HtbwsU0qRd4k4y--I3U33LkzkqJi6HtbwsU0qRd4k4yX-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/k1GU/KYEpkymNl0Nzw7ThfFgA=Location: /riak/surge/ianContent-Type: text/plainLink: </riak/surge>; rel="up"Etag: 51h3q7RjTNaHWYpO4P0MJjLast-Modified: Tue, 27 Sep 2011 20:38:01 GMTtarget--I3U33LkzkqJi6HtbwsU0qRd4k4y----CsxMIsO9tfadrRJ7EQ3XL2ivQ4f--
    220. 220. Lab: Walk Links (3/3)
    221. 221. Lab: Walk Links (3/3)Enter the origin key: seanEnter the link spec (default: _,_,_): foo,_,_
    222. 222. Lab: Walk Links (3/3)Enter the origin key: seanEnter the link spec (default: _,_,_): foo,_,_...> GET /riak/surge/sean/foo,_,_ HTTP/1.1< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Expires: Tue, 27 Sep 2011 20:53:08 GMT< Date: Tue, 27 Sep 2011 20:43:08 GMT< Content-Type: multipart/mixed; boundary=3AsxaHhVMlDEQCakLSmNqQUTS4Y< Content-Length: 172<--3AsxaHhVMlDEQCakLSmNqQUTS4YContent-Type: multipart/mixed; boundary=Ivtq9RgHpydECNEZOnHpiqHcFYl--Ivtq9RgHpydECNEZOnHpiqHcFYl----3AsxaHhVMlDEQCakLSmNqQUTS4Y--
    223. 223. Lab: Secondary Indexes
    224. 224. Lab: Secondary Indexes• Create some objects with indexes
    225. 225. Lab: Secondary Indexes• Create some objects with indexes• Query the index for their keys
    226. 226. Lab: Secondary Indexes• Create some objects with indexes• Query the index for their keys• Query input to MapReduce
    227. 227. Lab: Create 2I Objects
    228. 228. Lab: Create 2I Objects$ ./store-indexed.sh
    229. 229. Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40
    230. 230. Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40* About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/surgeobj1?returnbody=true HTTP/1.1> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8rzlib/1.2.3> Host: localhost:8091> Accept: */*> Content-Type: application/json> X-Riak-Index-surgeobj_int: 1> Content-Length: 14>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1uZVkMCUy5rEyHDn57ThfFgA=< x-riak-index-surgeobj_int: 1< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Date: Tue, 27 Sep 2011 22:27:16 GMT< Content-Type: application/json< Content-Length: 14<* Connection #0 to host localhost left intact* Closing connection #0{"surgeobj":1} ...
    231. 231. Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40 key* About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/surgeobj1?returnbody=true HTTP/1.1> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8rzlib/1.2.3> Host: localhost:8091> Accept: */*> Content-Type: application/json> X-Riak-Index-surgeobj_int: 1> Content-Length: 14>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1uZVkMCUy5rEyHDn57ThfFgA=< x-riak-index-surgeobj_int: 1< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Date: Tue, 27 Sep 2011 22:27:16 GMT< Content-Type: application/json< Content-Length: 14<* Connection #0 to host localhost left intact* Closing connection #0{"surgeobj":1} ...
    232. 232. Lab: Create 2I Objects$ ./store-indexed.shEnter the number of objects to create: 40* About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> PUT /riak/surge/surgeobj1?returnbody=true HTTP/1.1> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8rzlib/1.2.3> Host: localhost:8091> Accept: */* index header> Content-Type: application/json> X-Riak-Index-surgeobj_int: 1> Content-Length: 14>< HTTP/1.1 200 OK< X-Riak-Vclock: a85hYGBgzGDKBVIcR4M2cvs1uZVkMCUy5rEyHDn57ThfFgA=< x-riak-index-surgeobj_int: 1< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Link: </riak/surge>; rel="up"< Date: Tue, 27 Sep 2011 22:27:16 GMT< Content-Type: application/json< Content-Length: 14<* Connection #0 to host localhost left intact* Closing connection #0{"surgeobj":1} ...
    233. 233. Lab: Query 2I (1/2)
    234. 234. Lab: Query 2I (1/2)$ ./query-indexed.sh
    235. 235. Lab: Query 2I (1/2)$ ./query-indexed.shQuery on range or equality? (r/e) eIndex value: 10
    236. 236. Lab: Query 2I (1/2)$ ./query-indexed.shQuery on range or equality? (r/e) eIndex value: 10* About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> GET /buckets/surge/index/surgeobj_int/10 HTTP/1.1> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7OpenSSL/0.9.8r zlib/1.2.3> Host: localhost:8091> Accept: */*>< HTTP/1.1 200 OK< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Date: Tue, 27 Sep 2011 22:27:27 GMT< Content-Type: application/json< Content-Length: 23<* Connection #0 to host localhost left intact* Closing connection #0{"keys":["surgeobj10"]}
    237. 237. Lab: Query 2I (2/2)
    238. 238. Lab: Query 2I (2/2)Query on range or equality? (r/e) rRange start: 5Range end: 15
    239. 239. Lab: Query 2I (2/2)Query on range or equality? (r/e) rRange start: 5Range end: 15* About to connect() to localhost port 8091 (#0)* Trying 127.0.0.1... connected* Connected to localhost (127.0.0.1) port 8091 (#0)> GET /buckets/surge/index/surgeobj_int/5/15 HTTP/1.1> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7OpenSSL/0.9.8r zlib/1.2.3> Host: localhost:8091> Accept: */*>< HTTP/1.1 200 OK< Vary: Accept-Encoding< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Date: Tue, 27 Sep 2011 22:27:36 GMT< Content-Type: application/json< Content-Length: 148<* Connection #0 to host localhost left intact* Closing connection #0{"keys":["surgeobj12","surgeobj15","surgeobj6","surgeobj5","surgeobj13","surgeobj8","surgeobj11","surgeobj14","surgeobj9","surgeobj10","surgeobj7"]}
    240. 240. Lab: 2I MapReduce
    241. 241. Lab: 2I MapReduce$ ./mapred-indexed.sh
    242. 242. Lab: 2I MapReduce$ ./mapred-indexed.shQuery on range or equality? (r/e) rRange start: 35Range end: 40
    243. 243. Lab: 2I MapReduce$ ./mapred-indexed.shQuery on range or equality? (r/e) rRange start: 35Range end: 40Executing MapReduce:{ "inputs":{ "bucket":"surge", "index":"surgeobj_int" ,"start":35,"end":40},"query":[ {"map":{"name":"Riak.mapValuesJson","language":"javascript","keep":true}} ]}
    244. 244. Lab: 2I MapReduce$ ./mapred-indexed.shQuery on range or equality? (r/e) rRange start: 35Range end: 40Executing MapReduce:{ "inputs":{ "bucket":"surge", "index":"surgeobj_int" ,"start":35,"end":40},"query":[ {"map":{"name":"Riak.mapValuesJson","language":"javascript","keep":true}} ]}> POST /mapred HTTP/1.1> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3> Host: localhost:8091> Accept: */*> Content-Type: application/json> Content-Length: 164>< HTTP/1.1 200 OK< Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic)< Date: Wed, 28 Sep 2011 01:41:53 GMT< Content-Type: application/json< Content-Length: 97<* Connection #0 to host localhost left intact* Closing connection #0[{"surgeobj":35},{"surgeobj":40},{"surgeobj":36},{"surgeobj":37},{"surgeobj":39},{"surgeobj":38}]
    245. 245. Riak Operations
    246. 246. Configuration
    247. 247. File Locations
    248. 248. File Locations• Configuration
    249. 249. File Locations• Configuration • /etc/riak
    250. 250. File Locations• Configuration • /etc/riak• Binaries
    251. 251. File Locations• Configuration • /etc/riak• Binaries • /usr/sbin/riak
    252. 252. File Locations• Configuration • /etc/riak• Binaries • /usr/sbin/riak • /usr/sbin/riak-admin
    253. 253. File Locations• Configuration • /etc/riak• Binaries • /usr/sbin/riak • /usr/sbin/riak-admin • /usr/[lib,lib64]/riak
    254. 254. File Locations• Configuration • Logs • /etc/riak• Binaries • /usr/sbin/riak • /usr/sbin/riak-admin • /usr/[lib,lib64]/riak
    255. 255. File Locations• Configuration • Logs • /etc/riak • /var/log/riak• Binaries • /usr/sbin/riak • /usr/sbin/riak-admin • /usr/[lib,lib64]/riak
    256. 256. File Locations• Configuration • Logs • /etc/riak • /var/log/riak• Binaries • Data • /usr/sbin/riak • /usr/sbin/riak-admin • /usr/[lib,lib64]/riak
    257. 257. File Locations• Configuration • Logs • /etc/riak • /var/log/riak• Binaries • Data • /usr/sbin/riak • /var/lib/riak • /usr/sbin/riak-admin • /usr/[lib,lib64]/riak
    258. 258. File Locations• Configuration • Logs • /etc/riak • /var/log/riak• Binaries • Data • /usr/sbin/riak • /var/lib/riak • /usr/sbin/riak-admin • Handles / Temp Files • /usr/[lib,lib64]/riak
    259. 259. File Locations• Configuration • Logs • /etc/riak • /var/log/riak• Binaries • Data • /usr/sbin/riak • /var/lib/riak • /usr/sbin/riak-admin • Handles / Temp Files • /usr/[lib,lib64]/riak • /tmp/riak
    260. 260. /etc/riak/vm.args
    261. 261. /etc/riak/vm.args• Overview
    262. 262. /etc/riak/vm.args• Overview • Erlang VM configuration settings.
    263. 263. /etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings
    264. 264. /etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name.
    265. 265. /etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name. • Security cookie.
    266. 266. /etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name. • Security cookie.• Documentation
    267. 267. /etc/riak/vm.args• Overview • Erlang VM configuration settings.• Important Settings • Node name. • Security cookie.• Documentation • http://www.erlang.org/doc/man/erl.html
    268. 268. /etc/riak/vm.args
    269. 269. /etc/riak/vm.args• Node Name & Security Cookie ## Name of the riak node
    270. 270. /etc/riak/vm.args• Node Name & Security Cookie ## Name of the riak node -name riak@127.0.0.1
    271. 271. /etc/riak/vm.args• Node Name & Security Cookie ## Name of the riak node -name riak@127.0.0.1 ## Cookie for distributed erlang
    272. 272. /etc/riak/vm.args• Node Name & Security Cookie ## Name of the riak node -name riak@127.0.0.1 ## Cookie for distributed erlang -setcookie riak
    273. 273. /etc/riak/app.config
    274. 274. /etc/riak/ app.config• Overview
    275. 275. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.
    276. 276. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings
    277. 277. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports
    278. 278. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports • Directories
    279. 279. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports • Directories • Number of Partitions
    280. 280. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports • Directories • Number of Partitions • Storage Backend
    281. 281. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports • Directories • Number of Partitions • Storage Backend• Documentation
    282. 282. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports • Directories • Number of Partitions • Storage Backend• Documentation • http://wiki.basho.com/Configuration-Files.html
    283. 283. /etc/riak/ app.config• Overview • Riak (and dependency) configuration settings.• Important Settings • Ports • Directories • Number of Partitions • Storage Backend• Documentation • http://wiki.basho.com/Configuration-Files.html • Inline comments
    284. 284. /etc/riak/app.config
    285. 285. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage
    286. 286. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node.
    287. 287. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node. {storage_backend, riak_kv_bitcask_backend},
    288. 288. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node. {storage_backend, riak_kv_bitcask_backend},• About Storage Engines
    289. 289. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node. {storage_backend, riak_kv_bitcask_backend},• About Storage Engines • Riak has pluggable storage engines. (Bitcask, Innostore, LevelDB)
    290. 290. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node. {storage_backend, riak_kv_bitcask_backend},• About Storage Engines • Riak has pluggable storage engines. (Bitcask, Innostore, LevelDB) • Different engines have different tradeoffs.
    291. 291. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node. {storage_backend, riak_kv_bitcask_backend},• About Storage Engines • Riak has pluggable storage engines. (Bitcask, Innostore, LevelDB) • Different engines have different tradeoffs. • Engine selection depends on shape of data.
    292. 292. /etc/riak/ app.config• Storage Engine %% Storage_backend specifies the Erlang module defining the storage %% mechanism that will be used on this node. {storage_backend, riak_kv_bitcask_backend},• About Storage Engines • Riak has pluggable storage engines. (Bitcask, Innostore, LevelDB) • Different engines have different tradeoffs. • Engine selection depends on shape of data. • More on this later.
    293. 293. /etc/riak/ app.config%% Bitcask Config{bitcask, [ {data_root, "data/bitcask"} ]},{eleveldb, [ {data_root, “data/leveldb”} ]},
    294. 294. /etc/riak/ app.config• File Locations%% Bitcask Config{bitcask, [ {data_root, "data/bitcask"} ]},{eleveldb, [ {data_root, “data/leveldb”} ]},
    295. 295. /etc/riak/ app.config• File Locations%% Bitcask Config{bitcask, [ {data_root, "data/bitcask"} ]},{eleveldb, [ {data_root, “data/leveldb”} ]},
    296. 296. /etc/riak/ app.config• File Locations%% Bitcask Config{bitcask, [ {data_root, "data/bitcask"} ]},{eleveldb, [ {data_root, “data/leveldb”} ]},
    297. 297. /etc/riak/ app.config• File Locations%% Bitcask Config{bitcask, [ {data_root, "data/bitcask"} ]},{eleveldb, [ {data_root, “data/leveldb”} ]},
    298. 298. /etc/riak/ app.config• File Locations%% Bitcask Config{bitcask, [ {data_root, "data/bitcask"} ]},{eleveldb, [ {data_root, “data/leveldb”}• ]},
    299. 299. /etc/riak/ app.config%% http is a list of IP addresses and TCP ports that theRiak%% HTTP interface will bind.{http, [ {"127.0.0.1", 8091 } ]},%% pb_ip is the IP address that the Riak Protocol Buffersinterface%% will bind to. If this is undefined, the interface willnot run.{pb_ip, "127.0.0.1" },%% pb_port is the TCP port that the Riak Protocol Buffersinterface%% will bind to{pb_port, 8081 },
    300. 300. /etc/riak/ app.config• HTTP & Protocol Buffers%% http is a list of IP addresses and TCP ports that theRiak%% HTTP interface will bind.{http, [ {"127.0.0.1", 8091 } ]},%% pb_ip is the IP address that the Riak Protocol Buffersinterface%% will bind to. If this is undefined, the interface willnot run.{pb_ip, "127.0.0.1" },%% pb_port is the TCP port that the Riak Protocol Buffersinterface%% will bind to{pb_port, 8081 },
    301. 301. Securing Riak
    302. 302. Securing Riak• Overview
    303. 303. Securing Riak• Overview • Erlang has a “cookie” for “security.”
    304. 304. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text.
    305. 305. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text. • Riak assumes internal environment is trusted.
    306. 306. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text. • Riak assumes internal environment is trusted.• Securing the HTTP Interface
    307. 307. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text. • Riak assumes internal environment is trusted.• Securing the HTTP Interface • HTTP Auth via Proxy
    308. 308. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text. • Riak assumes internal environment is trusted.• Securing the HTTP Interface • HTTP Auth via Proxy • Does support SSL
    309. 309. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text. • Riak assumes internal environment is trusted.• Securing the HTTP Interface • HTTP Auth via Proxy • Does support SSL• Securing Protocol Buffers
    310. 310. Securing Riak• Overview • Erlang has a “cookie” for “security.” • Do not rely on it. Plain text. • Riak assumes internal environment is trusted.• Securing the HTTP Interface • HTTP Auth via Proxy • Does support SSL• Securing Protocol Buffers • No security.
    311. 311. Load Balancing
    312. 312. Load Balancing• Overview
    313. 313. Load Balancing• Overview • Any node can handle any request.
    314. 314. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.
    315. 315. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.• Load Balancing HTTP
    316. 316. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.• Load Balancing HTTP • HAProxy, nginx, etc...
    317. 317. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.• Load Balancing HTTP • HAProxy, nginx, etc...• Load Balancing Protocol Buffers
    318. 318. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.• Load Balancing HTTP • HAProxy, nginx, etc...• Load Balancing Protocol Buffers • Any TCP Load Balancer
    319. 319. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.• Load Balancing HTTP • HAProxy, nginx, etc...• Load Balancing Protocol Buffers • Any TCP Load Balancer• In General
    320. 320. Load Balancing• Overview • Any node can handle any request. • Makes load balancing easy.• Load Balancing HTTP • HAProxy, nginx, etc...• Load Balancing Protocol Buffers • Any TCP Load Balancer• In General • Use “least connected” strategy.
    321. 321. Storage Backends
    322. 322. There are 5 to choose from
    323. 323. There are 5 to choose from• Bitcask
    324. 324. There are 5 to choose from• Bitcask• Innostore
    325. 325. There are 5 to choose from• Bitcask• Innostore• LevelDB
    326. 326. There are 5 to choose from• Bitcask• Innostore• LevelDB• Memory
    327. 327. There are 5 to choose from• Bitcask• Innostore• LevelDB• Memory• Multi
    328. 328. Bitcask
    329. 329. Bitcask• A fast, append-only key-value store
    330. 330. Bitcask• A fast, append-only key-value store• In memory key lookup table (key_dir)
    331. 331. Bitcask• A fast, append-only key-value store• In memory key lookup table (key_dir)• Closed files are immutable
    332. 332. Bitcask• A fast, append-only key-value store• In memory key lookup table (key_dir)• Closed files are immutable• Merging cleans up old data
    333. 333. Bitcask• A fast, append-only key-value store• In memory key lookup table (key_dir)• Closed files are immutable• Merging cleans up old data• Apache 2 license
    334. 334. Innostore
    335. 335. Innostore• Based on Embedded InnoDB
    336. 336. Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage
    337. 337. Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage• Similar characterics to the MySQL InnoDB plugin
    338. 338. Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage• Similar characterics to the MySQL InnoDB plugin• Manages which pages of the B-trees are in memory
    339. 339. Innostore• Based on Embedded InnoDB• Write ahead log plus B-tree storage• Similar characterics to the MySQL InnoDB plugin• Manages which pages of the B-trees are in memory• GPL license
    340. 340. LevelDB
    341. 341. LevelDB• A Google developed key-value store
    342. 342. LevelDB• A Google developed key-value store• Append-only
    343. 343. LevelDB• A Google developed key-value store• Append-only• Multiple levels of SSTable-like structures
    344. 344. LevelDB• A Google developed key-value store• Append-only• Multiple levels of SSTable-like structures• BSD license
    345. 345. Cluster Capacity Planning
    346. 346. Cluster Capacity Planning
    347. 347. Cluster Capacity Planning• Overview
    348. 348. Cluster Capacity Planning• Overview • There are MANY variables that affect performance.
    349. 349. Cluster Capacity Planning• Overview • There are MANY variables that affect performance. • Safest route is to simulate load and see what happens.
    350. 350. Cluster Capacity Planning• Overview • There are MANY variables that affect performance. • Safest route is to simulate load and see what happens. • basho_bench can help
    351. 351. Cluster Capacity Planning• Overview • There are MANY variables that affect performance. • Safest route is to simulate load and see what happens. • basho_bench can help• Documentation
    352. 352. Cluster Capacity Planning• Overview • There are MANY variables that affect performance. • Safest route is to simulate load and see what happens. • basho_bench can help• Documentation • http://wiki.basho.com/Cluster-Capacity- Planning.html
    353. 353. Cluster Capacity Planning• Overview • There are MANY variables that affect performance. • Safest route is to simulate load and see what happens. • basho_bench can help• Documentation • http://wiki.basho.com/Cluster-Capacity- Planning.html • http://wiki.basho.com/System- Requirements.html
    354. 354. Things to Consider (1/3)
    355. 355. Things to Consider (1/3)• Virtualization
    356. 356. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound
    357. 357. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance.
    358. 358. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance. • Virtualize for other factors (agility or cost)
    359. 359. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance. • Virtualize for other factors (agility or cost)• Object size
    360. 360. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance. • Virtualize for other factors (agility or cost)• Object size • Too small means unnecessary disk and network overhead
    361. 361. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance. • Virtualize for other factors (agility or cost)• Object size • Too small means unnecessary disk and network overhead • Too large means chunky reads and writes.
    362. 362. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance. • Virtualize for other factors (agility or cost)• Object size • Too small means unnecessary disk and network overhead • Too large means chunky reads and writes.• Read/Write Ratio
    363. 363. Things to Consider (1/3)• Virtualization • Riak’s workload is I/O bound • Run on bare metal for best performance. • Virtualize for other factors (agility or cost)• Object size • Too small means unnecessary disk and network overhead • Too large means chunky reads and writes.• Read/Write Ratio • Riak is not meant for dumping many tiny datapoints.
    364. 364. Things to Consider (2/3)
    365. 365. Things to Consider (2/3)• “Randomness” of Reads / Writes
    366. 366. Things to Consider (2/3)• “Randomness” of Reads / Writes • Recently accessed objects are cached in memory.
    367. 367. Things to Consider (2/3)• “Randomness” of Reads / Writes • Recently accessed objects are cached in memory. • A small working set of objects will be fast.
    368. 368. Things to Consider (2/3)• “Randomness” of Reads / Writes • Recently accessed objects are cached in memory. • A small working set of objects will be fast.• Amount of RAM
    369. 369. Things to Consider (2/3)• “Randomness” of Reads / Writes • Recently accessed objects are cached in memory. • A small working set of objects will be fast.• Amount of RAM • Expands the size of cached working set, lowers latency.
    370. 370. Things to Consider (2/3)• “Randomness” of Reads / • Disk Speed Writes • Recently accessed objects are cached in memory. • A small working set of objects will be fast.• Amount of RAM • Expands the size of cached working set, lowers latency.
    371. 371. Things to Consider (2/3)• “Randomness” of Reads / • Disk Speed Writes • Faster disk means • Recently accessed lower read/write objects are cached in latency. memory. • A small working set of objects will be fast.• Amount of RAM • Expands the size of cached working set, lowers latency.
    372. 372. Things to Consider (2/3)• “Randomness” of Reads / • Disk Speed Writes • Faster disk means • Recently accessed lower read/write objects are cached in latency. memory. • Number of Processors • A small working set of objects will be fast.• Amount of RAM • Expands the size of cached working set, lowers latency.
    373. 373. Things to Consider (2/3)• “Randomness” of Reads / • Disk Speed Writes • Faster disk means • Recently accessed lower read/write objects are cached in latency. memory. • Number of Processors • A small working set of objects will be fast. • More processors == more Map/Reduce• Amount of RAM throughput. • Expands the size of cached working set, lowers latency.
    374. 374. Things to Consider (2/3)• “Randomness” of Reads / • Disk Speed Writes • Faster disk means • Recently accessed lower read/write objects are cached in latency. memory. • Number of Processors • A small working set of objects will be fast. • More processors == more Map/Reduce• Amount of RAM throughput. • Expands the size of • Faster processors == cached working set, lower Map/Reduce lowers latency. latency.
    375. 375. Things to Consider (3/3)
    376. 376. Things to Consider (3/3)• Features
    377. 377. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search
    378. 378. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends
    379. 379. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB
    380. 380. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB • Space/Time tradeoff
    381. 381. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB • Space/Time tradeoff• Partitions
    382. 382. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB • Space/Time tradeoff• Partitions • Depends on eventual number of boxes.
    383. 383. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB • Space/Time tradeoff• Partitions • Depends on eventual number of boxes.• Protocol
    384. 384. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB • Space/Time tradeoff• Partitions • Depends on eventual number of boxes.• Protocol • HTTP vs. Protocol Buffers
    385. 385. Things to Consider (3/3)• Features • RiakKV vs. Riak 2I vs. Riak Search• Backends • Bitcask vs. Innostore vs. eLevelDB • Space/Time tradeoff• Partitions • Depends on eventual number of boxes.• Protocol • HTTP vs. Protocol Buffers • All clients support HTTP, some support PB.
    386. 386. Performance Tuning
    387. 387. Performance Tuning• General Tips
    388. 388. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile
    389. 389. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs
    390. 390. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs • The More RAM the Better
    391. 391. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs • The More RAM the Better • Fast Disk
    392. 392. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs • The More RAM the Better • Fast Disk • Use noatime mounts
    393. 393. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs • The More RAM the Better • Fast Disk • Use noatime mounts • Limit List Keys and Full Bucket MapReduce
    394. 394. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs • The More RAM the Better • Fast Disk • Use noatime mounts • Limit List Keys and Full Bucket MapReduce • Benchmark in Advance
    395. 395. Performance Tuning• General Tips • Start with a “DB” Like Machine Profile • Multi-core 64-bit CPUs • The More RAM the Better • Fast Disk • Use noatime mounts • Limit List Keys and Full Bucket MapReduce • Benchmark in Advance • Graph Everything
    396. 396. Performance Tuning
    397. 397. Performance Tuning• Bitcask Backend
    398. 398. Performance Tuning• Bitcask Backend • KeyDir is in Memory
    399. 399. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations)
    400. 400. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations) • Merge Trigger Settings
    401. 401. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations) • Merge Trigger Settings • Scheduling Bitcask Merges
    402. 402. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations) • Merge Trigger Settings • Scheduling Bitcask Merges • Key Expiration
    403. 403. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations) • Merge Trigger Settings • Scheduling Bitcask Merges • Key Expiration• Documentation
    404. 404. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations) • Merge Trigger Settings • Scheduling Bitcask Merges • Key Expiration• Documentation • http://wiki.basho.com/Bitcask.html
    405. 405. Performance Tuning• Bitcask Backend • KeyDir is in Memory • Filesystem Cache (Access Profile Considerations) • Merge Trigger Settings • Scheduling Bitcask Merges • Key Expiration• Documentation • http://wiki.basho.com/Bitcask.html • http://wiki.basho.com/Bitcask-Capacity-Planning.html
    406. 406. Performance Tuning
    407. 407. Performance Tuning• Innostore Backend
    408. 408. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB
    409. 409. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size
    410. 410. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size • o_direct
    411. 411. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size • o_direct • log_files_in_groups
    412. 412. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size • o_direct • log_files_in_groups • Separate Spindles for Log and Data
    413. 413. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size • o_direct • log_files_in_groups • Separate Spindles for Log and Data• Documentation
    414. 414. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size • o_direct • log_files_in_groups • Separate Spindles for Log and Data• Documentation • http://wiki.basho.com/Setting-Up-Innostore.html
    415. 415. Performance Tuning• Innostore Backend • Similar to MySQL + InnoDB • buffer pool size • o_direct • log_files_in_groups • Separate Spindles for Log and Data• Documentation • http://wiki.basho.com/Setting-Up-Innostore.html • http://wiki.basho.com/Innostore-Configuration-and- Tuning.html
    416. 416. basho_bench
    417. 417. basho_bench• What Does It Do?
    418. 418. basho_bench• What Does It Do? • Generates load against a Riak cluster. Logs and graphs results.
    419. 419. basho_bench• What Does It Do? • Generates load against a Riak cluster. Logs and graphs results. • Intended for cluster capacity planning.
    420. 420. basho_bench• What Does It Do? • Generates load against a Riak cluster. Logs and graphs results. • Intended for cluster capacity planning.• Major Parameters
    421. 421. basho_bench• What Does It Do? • Generates load against a Riak cluster. Logs and graphs results. • Intended for cluster capacity planning.• Major Parameters • Test Harness / Key Generator / Value Generator
    422. 422. basho_bench• What Does It Do? • Generates load against a Riak cluster. Logs and graphs results. • Intended for cluster capacity planning.• Major Parameters • Test Harness / Key Generator / Value Generator • Read / write / update mix.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×