Mio - a distributed Skip Graph based orderd KVS

Mio a distributed Skip Graph based orderd KVS Cybozu Labs, Inc. Taro Minowa (Higepon)

Introduce myself @higepon Mona OS  http://www.monaos.org Mosh  A fast Scheme interpreter Outputz  http://outputz.com Feb 26 2010 Mio - a Skip Graph based ordered KVS 2

Summary Mio is...  a distributed orderd KVS  memcached + range search  Skip Graph based  Written in Erlang  http://github.com/higepon/mio  In alpha quality Feb 26 2010 Mio - a Skip Graph based ordered KVS 3

Background Feb 26 2010 Mio - a Skip Graph based ordered KVS 4

RDBMS vs KVS Scalability KVS set/get volatile High functionality Transaction SQL RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 5

RDBMS vs KVS Scalability KVS set/get volatile High Complement each other functionality Transaction SQL RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 5

Mio Scalability KVS High functionality RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 6

Mio Scalability KVS Mio +Range search High functionality RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 6

Mio Scalability KVS Mio +Range search Makes RDBMS High lighter workload functionality RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 6

Range search? Queries  last 7 days  prev/next  Top 10 ranking SQL  SELECT * FROM photos WHERE date between xxx and xx order by date limit 10 RDBMS handles these queires Feb 26 2010 Mio - a Skip Graph based ordered KVS 7

Mio Feb 26 2010 Mio - a Skip Graph based ordered KVS 8

The Challenges and Design Decisions Range search  Ordered structure  Skip Graphs algorithm Scale-Out  distributed using Erlang functions memcached compatible I/F Volatile  keep it simple Feb 26 2010 Mio - a Skip Graph based ordered KVS 9

Skip Graphs James Aspnes (2003) Feb 26 2010 Mio - a Skip Graph based ordered KVS 10

Supported operations search by key insert (join) remove range search by key1 and key2 Feb 26 2010 Mio - a Skip Graph based ordered KVS 11

Set of sorted doubly linked lists Shibuya Shinjuku Tamachi Ueno Yoyogi Same as railway stations  All keys (stations) consist doubly linked list  Knows only his left and right station  Keep sorted by key Search Shibuya start from Ueno  Go to left. O(n) Feb 26 2010 Mio - a Skip Graph based ordered KVS 12

Make an express lane Skip Express Shinjuku Ueno Local Shibuya Shinjuku Tamachi Ueno Yoyogi Skip some stations Ueno -> Shinjuku -> Shibuya Tamachi is placed on another express lane Feb 26 2010 Mio - a Skip Graph based ordered KVS 13

Multiple lanes Level 2 Level 1 Level 0 Shibuya Shinjuku Tamachi Ueno Yoyogi Level 0 lane  all keys are in the list Level n (n > 0) lane  express lane  n + 1 lane is more express than n lane. Feb 26 2010 Mio - a Skip Graph based ordered KVS 14

Search Level 2 Level 1 Level 0 Shibuya Shinjuku Tamachi Ueno Yoyogi Start from highest to lower level Can search from any stations O(log n) Feb 26 2010 Mio - a Skip Graph based ordered KVS 15

Range Search Level 2 Level 1 Level 0 Shibuya Shinjuku Tamachi Ueno Yoyogi Search key1 Collect matched on Level 0 ex. Key1 = Ueno , Key2 = Shibuya Feb 26 2010 Mio - a Skip Graph based ordered KVS 16

Remove B A B C A C Remove on each Level  Update neighbor’s links  Highest to lower Feb 26 2010 Mio - a Skip Graph based ordered KVS 17

Insert B A C A B C Insert on each Level  Update neighbors’s links  Lowest to higher (in reverse order to remove)  In which express lane is a new station insereted? radomly located uniform Feb 26 2010 Mio - a Skip Graph based ordered KVS 18

Easy to implement? No  Really simple, but ...  We should support concurrent insert/remove If neighbor is removed when inserting? If someone inserts another to neighbor? Searching crash?  Fragile linked list  We can’t find any perfect concurrent join algorithm. Feb 26 2010 Mio - a Skip Graph based ordered KVS 19

Our concurrent algorithm Lock some nodes Please read the source code :) Defined three invariants A B C A C A C B B Feb 26 2010 Mio - a Skip Graph based ordered KVS 20

Implementation Feb 26 2010 Mio - a Skip Graph based ordered KVS 21

Written in Erlang A station(key, value) is a process  gen_server process  Hold left/right on each level  Follow left/right = gen_server:call/2  No distinction between local and remote process Erlang is great! Ditributed with -name option  erl -name name@FQDN Feb 26 2010 Mio - a Skip Graph based ordered KVS 22

Performance 5000 qps on single node really slow on multiple nodes  need less communication between nodes  need better algorithm Feb 26 2010 Mio - a Skip Graph based ordered KVS 23

Demo Feb 26 2010 Mio - a Skip Graph based ordered KVS 24

Tips for practical Erlang Max process option +P  Set proper value. Don’t use MAX. gerbage_collect()  Fast enough, reduce memory usage.  hibernate is slow... refactorerl fprof on gen_server shows nothing  Use dynomite profile Feb 26 2010 Mio - a Skip Graph based ordered KVS 25

Tips for practical Erlang Common test  Coverage  load test gen_server:call is slow  Use mnesia for property access.  Easy replication Easy to run  Should users run erl with many options?  Shell script borrowed from RabbitMQ Feb 26 2010 Mio - a Skip Graph based ordered KVS 26

Summary, Once more Mio is...  a distributed orderd KVS  memcached + range search  Skip Graph based  Written in Erlang  http://github.com/higepon/mio  In alpha quality Feb 26 2010 Mio - a Skip Graph based ordered KVS 27

http://higepon.blogspot.com	2547
http://developer.cybozu.co.jp	937
http://www.slideshare.net	10
http://translate.googleusercontent.com	5
http://higepon.blogspot.com.sixxs.org	5
http://www.redditmedia.com	3
http://higepon.tumblr.com	2
http://cc.bingj.com	2
http://www.techgig.com	1

Mio - a distributed Skip Graph based orderd KVS

by Taro Higepon on Feb 25, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

9 Embeds 3,512

Statistics

Mio - a distributed Skip Graph based orderd KVS — Presentation Transcript