DISTRIBUTING WORK ACROSS
CLUSTERS
Adventures With Riak Pipe
Last Updated: October 31, 2013

Susan Potter @SusanPotter
Background

Concepts & Intuitions

Applications

WHOAMI

2
2013-10-31

.

.
Distributing Work Across Clusters

WHOAMI

whoami
.
.

An app + middleware dev who has felt Ops pain too
BACKGROUND
Background

Concepts & Intuitions

Applications

MODEL/ASSUMPTIONS

4
Background

Concepts & Intuitions

Applications

MOTIVATING USES

5
Background

Concepts & Intuitions

Applications

COMMON APPROACHES

6
Background

Concepts & Intuitions

Applications

WHY DISTRIBUTE? WHY DECENTRALIZE?

Probability of system failure
→ Large ...
Background

Concepts & Intuitions

Applications

8
CONCEPTS & INTUITIONS
Background

Concepts & Intuitions

Applications

LAYERS

10
Background

Concepts & Intuitions

Applications

TERMINOLOGY (CORE)

→ Partition
→ Vnode
→ Hinted Handoff
→ Read Repair

1...
Background

Concepts & Intuitions

Applications

TERMINOLOGY (PIPE)

→ Pipe
→ Fitting ("phase")
→ Queue
→ Worker

12
Background

Concepts & Intuitions

Applications

EXAMPLE UNIX PIPE

1
2
3

find . -name "*. rb" 
| xargs egrep "#.*? TODO:...
Background

Concepts & Intuitions

Applications

EXAMPLE FUNCTION COMPOSITION

1

( length . mapToUpper . sanitize ) input...
Background

Concepts & Intuitions

Applications

EXAMPLE RIAK PIPE

1
2
3
4
5
6
7

[ # fitting_spec {
,
,
, # fitting_spec...
Background

Concepts & Intuitions

Applications

VNODE AND WORKERS

→

vnode
manages lifecycle and queues of workers

→

v...
Background

Concepts & Intuitions

Applications

VNODE WORKER BEHAVIOR

1
2
3
4
5
6
7

behaviour_info ( callbacks ) ->
[{i...
Background

Concepts & Intuitions

Applications

DISTRIBUTION: CHASHFUN

→ Use random uniform hash fun to saturate workers...
Background

Concepts & Intuitions

Applications

VALIDATION

1

2
3
4
5

% # fitting_spec { module = ... , arg = 43, ...
}...
Background

Concepts & Intuitions

Applications

LEAKY PIPES

20
Background

Concepts & Intuitions

Applications

RIAK (CORE) MECHANICS

→

Handoff
Migrate vnodes from node to node

1
2
3...
Background

Concepts & Intuitions

Applications

FAILURES

→

validate_arg
Seen above

→

Pipe client process dies
YOLO

→...
Background

Concepts & Intuitions

Applications

EXISTING FITTINGS

→

riak_pipe_w_tee
Useful for intermediate results

→
...
APPLICATIONS
Background

Concepts & Intuitions

Applications

KNOWN USES

Riak's Map/Reduce
→ Risk metrics
→ Tenant Usage
→

25
Background

Concepts & Intuitions

Applications

TROUBLESHOOTING

→

riak_pipe:status/1
Provides fittings, processed, fail...
Background

Concepts & Intuitions

Applications

FURTHER WORK

→ More applications (e.g. genetics, 3rd party APIs with rat...
Background

Concepts & Intuitions

Applications

RELATED WORK

→ "Pipe" libraries EVERYWHERE (pipe, scalaz-stream)
→ riak_...
Background

Concepts & Intuitions

Applications

ROYAL FAIL

http://www.flickr.com/photos/dadavidov/

29
Background

Concepts & Intuitions

Applications

QUESTIONS

Questions?

30
Upcoming SlideShare
Loading in...5
×

Ricon/West 2013: Adventures with Riak Pipe

1,075

Published on

Also make sure you check out Riak Pipe talks by Bryan Fink.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,075
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ricon/West 2013: Adventures with Riak Pipe

  1. 1. DISTRIBUTING WORK ACROSS CLUSTERS Adventures With Riak Pipe Last Updated: October 31, 2013 Susan Potter @SusanPotter
  2. 2. Background Concepts & Intuitions Applications WHOAMI 2
  3. 3. 2013-10-31 . . Distributing Work Across Clusters WHOAMI whoami . . An app + middleware dev who has felt Ops pain too
  4. 4. BACKGROUND
  5. 5. Background Concepts & Intuitions Applications MODEL/ASSUMPTIONS 4
  6. 6. Background Concepts & Intuitions Applications MOTIVATING USES 5
  7. 7. Background Concepts & Intuitions Applications COMMON APPROACHES 6
  8. 8. Background Concepts & Intuitions Applications WHY DISTRIBUTE? WHY DECENTRALIZE? Probability of system failure → Large datasets (data locality) → Eliminate SPOFs → Commodity hardware → 7
  9. 9. Background Concepts & Intuitions Applications 8
  10. 10. CONCEPTS & INTUITIONS
  11. 11. Background Concepts & Intuitions Applications LAYERS 10
  12. 12. Background Concepts & Intuitions Applications TERMINOLOGY (CORE) → Partition → Vnode → Hinted Handoff → Read Repair 11
  13. 13. Background Concepts & Intuitions Applications TERMINOLOGY (PIPE) → Pipe → Fitting ("phase") → Queue → Worker 12
  14. 14. Background Concepts & Intuitions Applications EXAMPLE UNIX PIPE 1 2 3 find . -name "*. rb" | xargs egrep "#.*? TODO:" | wc -l Character-based, through file descriptors 13
  15. 15. Background Concepts & Intuitions Applications EXAMPLE FUNCTION COMPOSITION 1 ( length . mapToUpper . sanitize ) input Value based, through functions 14
  16. 16. Background Concepts & Intuitions Applications EXAMPLE RIAK PIPE 1 2 3 4 5 6 7 [ # fitting_spec { , , , # fitting_spec { , , ] name= fetch_trades module = riskmgr_fetch_trades ...} name= calc_var module = riskmgr_calc_var ...} Message-based, across nodes 15
  17. 17. Background Concepts & Intuitions Applications VNODE AND WORKERS → vnode manages lifecycle and queues of workers → vnode worker processes inputs 16
  18. 18. Background Concepts & Intuitions Applications VNODE WORKER BEHAVIOR 1 2 3 4 5 6 7 behaviour_info ( callbacks ) -> [{init ,2}, {process ,3}, {done ,1}]; behaviour_info ( _Other ) -> undefined . %% Optionally two more too , not required . 17
  19. 19. Background Concepts & Intuitions Applications DISTRIBUTION: CHASHFUN → Use random uniform hash fun to saturate workers → See Bryan Fink's previous Pipe presentations 18
  20. 20. Background Concepts & Intuitions Applications VALIDATION 1 2 3 4 5 % # fitting_spec { module = ... , arg = 43, ... } % validate_arg /1 validate_arg (Arg) when is_integer (Arg) -> ok; validate_arg (Arg) when is_atom (Arg) -> ok; validate_arg (_) -> {error , " Argument must be a valid integer or atom"}. 19
  21. 21. Background Concepts & Intuitions Applications LEAKY PIPES 20
  22. 22. Background Concepts & Intuitions Applications RIAK (CORE) MECHANICS → Handoff Migrate vnodes from node to node 1 2 3 % Old Node % Called with last known State of worker archive /1 % returns {ok , Archive } 4 5 6 7 % New Node % Called with Archive and State from old node handoff /2 % returns {ok , NewState } 21
  23. 23. Background Concepts & Intuitions Applications FAILURES → validate_arg Seen above → Pipe client process dies YOLO → nval How many vnodes to ask before failing, default 1 → vnode worker logic Think using dataflow semantics 22
  24. 24. Background Concepts & Intuitions Applications EXISTING FITTINGS → riak_pipe_w_tee Useful for intermediate results → riak_pipe_w_xform Simple delegater: 3-arity function → riak_pipe_w_reduce What you would expect: simple accumulating reduce → riak_kv_pipe_get Take advantage of data locality with cohosted KV store 23
  25. 25. APPLICATIONS
  26. 26. Background Concepts & Intuitions Applications KNOWN USES Riak's Map/Reduce → Risk metrics → Tenant Usage → 25
  27. 27. Background Concepts & Intuitions Applications TROUBLESHOOTING → riak_pipe:status/1 Provides fittings, processed, failures, queue_length, etc stats → riak_pipe_w_crash fitting Used to test Riak Pipe → riak_pipe:active_pipelines/1 See all active pipelines → riak_pipe_cinfo module cluster info interrogation module 26
  28. 28. Background Concepts & Intuitions Applications FURTHER WORK → More applications (e.g. genetics, 3rd party APIs with rate limits) → Decentralized pipe control → Measure "completeness" of resultset 27
  29. 29. Background Concepts & Intuitions Applications RELATED WORK → "Pipe" libraries EVERYWHERE (pipe, scalaz-stream) → riak_pg → Map-Reduce frameworks → Staged Event Driven Architecture → Event Stream Processing → Dataflow multi-stage processing 28
  30. 30. Background Concepts & Intuitions Applications ROYAL FAIL http://www.flickr.com/photos/dadavidov/ 29
  31. 31. Background Concepts & Intuitions Applications QUESTIONS Questions? 30
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×