• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Introduction to Riak
 

Introduction to Riak

on

  • 3,587 views

 

Statistics

Views

Total Views
3,587
Views on SlideShare
2,939
Embed Views
648

Actions

Likes
8
Downloads
66
Comments
0

15 Embeds 648

http://yosuke-perspective.blogspot.jp 319
http://yosuke-perspective.blogspot.com 297
http://yosuke-perspective.blogspot.ru 13
http://a0.twimg.com 3
http://yosuke-perspective.blogspot.sg 3
http://webcache.googleusercontent.com 2
http://yosuke-perspective.blogspot.de 2
http://yosuke-perspective.blogspot.ca 2
http://yosuke-perspective.blogspot.fr 1
http://yosuke-perspective.blogspot.hk 1
http://yosuke-perspective.blogspot.ie 1
http://yosuke-perspective.blogspot.mx 1
http://yosuke-perspective.blogspot.com.es 1
http://paper.li 1
http://leapf.org 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Introduction to Riak Introduction to Riak Presentation Transcript

    • Introduction to Riak Python hack-a-thon 2011.10.15 Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, October 15, 11 1
    • Who am I ?Saturday, October 15, 11 2
    • twitter-id: yosukehara RIA Distributed StorageSaturday, October 15, 11 3
    • What’s ‘Riak’ ? bashoSaturday, October 15, 11 4
    • KVS + ExtraSaturday, October 15, 11 5
    • Basho Bitcask or LevelDB (Google, from 1.0)Saturday, October 15, 11 6
    • Architecture Erlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes Workers storage backend Riak KVSaturday, October 15, 11 7
    • The horizontal scaling bitsSaturday, October 15, 11 8
    • Inspired Amazon’s Dynamo (2007) KVS (Key-Value Storage) + Extra Masterless, P2P-Replication Consistent Hashing Failover - Quorums, Hinted handoff Eventual ConsistencySaturday, October 15, 11 9
    • Inspired Amazon’s Dynamo http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html “The reliability and scalability of a system is dependent on how its application state is managed.” - Werner VogelsSaturday, October 15, 11 10
    • KVS (Key-Value-Store) Simple operations - get, put, delete Value is mostly opaque (some metadata) Extras MapReduce (depends on Javascript) Links Full-text search (Optional - Riak Search) Secondary Indexes (from ver 1.0)Saturday, October 15, 11 11
    • P2P - MasterlessSaturday, October 15, 11 12
    • P2P - Masterless NOT SPOFSaturday, October 15, 11 12
    • Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
    • Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node Logical subdivision of the cluster “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
    • For Parallelism # of VNodes == Maximum Concurrent Reqs For Rebalancing the Cluster Smallest block that can be shifted to a new node For Resiliance The system restarts failed VNodesSaturday, October 15, 11 14
    • Quorum Every request contacts all replicas of key N : number of replicas (default 3) R : read quorum W : write quorum Able to adjust to “Consistency Level”Saturday, October 15, 11 15
    • Distributed Data (1) PUT File (3) if # of replicas needed for a successful Write > 2 # of replicas = 3 (4) return “ok” # of successful WRITEs = 2 Node-0 Node-1 Node-2 Node-3 (2) Replicate - Router on each nodeSaturday, October 15, 11 16
    • Hinted Handoff X (1) PUT File X (3) if # of replicas needed X for a successful Write > 2 # of replicas = 3 X (4) return “ok” X # of successful WRITEs = 2 X Node-0 Node-1 X X Node-2 (2) Replicate Node-3 X (5) RecoverSaturday, October 15, 11 17
    • Eventual ConsistencySaturday, October 15, 11 18
    • CAP Theorem Consistency Availability Partition ToleranceSaturday, October 15, 11 19
    • The doc-database bitsSaturday, October 15, 11 20
    • Store objects as JSON or any format Link between objects No SQL - Query with Javascript Map-ReduceSaturday, October 15, 11 21
    • The operations-friendly bitsSaturday, October 15, 11 22
    • Web-shaped storage Store data in its original format Using HTTP Load balancing Caches No special-nodes - grow horizontallySaturday, October 15, 11 23
    • DEMO bashoSaturday, October 15, 11 24
    • PUT curl -X PUT -H "x-riak-index-service_bin: news" -H "x-riak-index-range_int: 201110151300" -H "Content-Type: application/json" -d {"operation":"W", "key":"news/photo/20111015_small.jpg", "service":"news","range":"201110151300"} http://127.0.0.1:8098/riak/news/log_001 KEY VALUE For Secondary IndexSaturday, October 15, 11 25
    • GET curl http://127.0.0.1:8098/demo/news/log_001 | python -mjson.tool { "key": "news/photo/20111015_small.jpg", "operation": "W", "range": "201110151300", "service": "news" } KEY VALUE For Secondary IndexSaturday, October 15, 11 26
    • Seconday Index (2I) curl http://127.0.0.1:8098/buckets/demo/index/service_bin/news | python -mjson.tool { "keys": [ "log_001", "log_002" ] } KEY VALUE For Secondary IndexSaturday, October 15, 11 27
    • Map Reduce curl -H "Content-Type: application/json" http://127.0.0.1:8098/mapred --data @-<<EOF {"inputs":{"bucket":"demo","index":"range_int","key":"201110151300"}, "query":[ {"map": {"name":"Riak.mapValuesJson","language":"javascript","keep":false}}, {"reduce":{"language":"javascript","source":" function(v) { var json_object = eval(v); var result = {}; for (i = 0; i < json_object.length; i++) { var key = json_object[i].service + _ + json_object[i].operation; var cur = result[key] == undefined ? 0 : result[key]; result[key] = cur + 1; } return result; } "}}]} Result: {"news_W":2,"social_W":1} EOFSaturday, October 15, 11 28
    • basho_benchSaturday, October 15, 11 29
    • basho_bench Depend on Erlang Able to be ‘Easy Operation’ Able to create ‘Pretty Chart’ for KVS and OthersSaturday, October 15, 11 30
    • Benchmartk ResultSaturday, October 15, 11 31
    • basho_bench config {mode, max}. {duration, 15}. {concurrent, 5}. {driver, basho_bench_driver_riakclient}. {code_paths, ["deps/stats", "/Users/jmeredith/basho/riak/apps/riak_kv", "/Users/jmeredith/basho/riak/apps/riak_core"]}. {key_generator, {int_to_bin, {uniform_int, 35000}}}. {value_generator, {fixed_bin, 10000}}. {riakclient_nodes, [riak@127.0.0.1]}. {riakclient_mynode, [riak_bench@127.0.0.1, longnames]}. {riakclient_replies, 1}.Saturday, October 15, 11 32
    • Thank you for listeningSaturday, October 15, 11 33