Polyglot parallelism

Polyglot Parallelism A Case Study in Using Erlang and Ruby at Rackspace

The Problem Part 1

20,000network devices

9 Datacenters, 3 Continents

devices not designed for high-throughput management

we need a highthroughput solution

the time spent in I/O is the primary bottleneck

if you want to speedthings up you have totalk to more devices in parallel

The Problem Part II

huge blobs of data

lots of backupsequals big database

ad-hoc searching is difﬁcult but important

customer SLA meansneed to restore from backup quickly

an event must begenerated for eachdevice interaction

migrations areproblematic with that much data

rigid schema made adapting to new devices difﬁcult

each device type hasdifferent properties

“backup” meansdifferent things for each device type

need to grow with the business

PreviousSolution

multiple Ruby apps

difﬁcult to scale

vendor device managers

New Solution

the simplest thingthat could possibly work

most db writes come fromscheduled jobs

Other ClientsRails ReST API Erlang MongoDB Network Devices

Joe Armstrong

AXD301ATM Switch

99.9999999%

Functional

Dynamically Typed

Single Assignment

A = 1. %=> 1A = 2. %=> badmatch

[B, 2, C] = [1, 2, 3].B = 1. %=> 1C = 3. %=> 3

Immutable DataStructures

D = dict:new().D1 = dict:store(foo, 1, D).D2 = dict:store(bar, 2, D1).

Concurrency Oriented

-module(fact).-export([fac/1]).fac(0) -> 1;fac(N) -> N * fac(N-1).

-module(quicksort).-export([quicksort/1]).quicksort([]) -> [];quicksort([Pivot|Rest]) -> quicksort([Front || Front <- Rest, Front < Pivot]) ++ [Pivot] ++ quicksort([Back || Back <- Rest, Back >= Pivot]).

Details

jobs framework

Runner Callback Workers Module

Callback Runner Worker Module start readyitem process process ready . . . stop

“behaviour” is interface

behaviour_info(callbacks) -> [ {init, 1}, {process_item, 3}, {worker_died, 5}, {job_stopping, 1}, {job_complete, 2}].

running({worker_ready, WorkerPid, ok}, S) -> case queue:out(S#state.items) of {empty, I2} -> stop_worker(WorkerPid, S), {next_state, complete, S#state{items = I2}}; {{value, Item}, I2} -> job_worker:process(WorkerPid, Item, now(), S#state.job_state), {next_state, running, S#state{items = I2}} end;

handle_info({DOWN, _, process, WorkerPid, Info}, StateName, S) -> {Item, StartTime} = clear_worker(WorkerPid, S), Callback = S#state.callback, spawn(Callback, worker_died, [Item, WorkerPid, StartTime, Info, S#state.job_state]), %% Start a replacement worker start_workers(1, Callback), {next_state, StateName, S};

handle_cast({process, Item, StartTime, JS}, S) -> Callback = S#state.callback, Continue = try Callback:process_item(Item, StartTime, JS) catch throw: Error -> error_logger:error_report(Error), ok end, job_runner:worker_ready(S#state.runner, self(), Continue), {noreply, S}.

story time

ReSTful API with WebmachineThe Convention Over Conﬁguration Webserver

http://webmachine.basho.comHTTP Request Lifecycle Diagram

If you know HTTPWebmachine Is SimpleAs Proven by the “Number of Types of Things” Measurement of Complexity

The 3 Most Important Types of Things In Webmachine1. Dispatch Rules (pure data--barely a thing!)2. Resources (composed of simple functions!)3. Requests (simple get/set interface!)

Dispatch Rules { ["devices", server], device_resource, [] } GET /devices/12345 Webmachine inspects the device_resource module fordeﬁned callbacks, and sets the Request record’s “server” value to 12345.

Resources• POEM (Plain Old Erlang Module)• Composed of referentially transparent functions*• Functions are callbacks into the request lifecycle• Approximately 30 possible callback functions, e.g.: • resource_exists → 404 Not Found • is_authorized → 401 Not Authorized * mostly

Resource Functions Perma-404resource_exists(Request, Context) -> {false, Request, Context}. Lucky Authis_authorized(Request, Context) -> S = calendar:time_to_seconds(now()), case S rem 2 of 0 -> {true, Request, Context}; 1 -> {“Basic realm=lucky”, Request, Context} end.

Requests• The ﬁrst argument to each resource function• Set and read request & response data RemoteIP = wrq:peer(Request).wrq:set_resp_header(“X-Answer”, “42”, Request).

Retrieving a JSON Firewall Representationcontent_types_provided(Request, Context) -> Types = [{"application/json", to_json}], {Types, Request, Context}.to_json(Request, Context) -> Device = proplists:get_value(device, Context), UserId = get_user_id(Request), case fe_api_firewall:get_config(Device, UserId) of {ok, Config} -> success_response(Config, Request, Context); {error, Reason} -> error_response(502, Reason, Request, Context) end.

Gotchas

primitive obsession

string-ish “hi how are you” <<“hello there”>>[<<"easy as ">>, [$a, $b, $c], " ☺n"].

hashes vs records

to loop is human, to recur divine

Erlang conditionalsalways return a value

design for testability

don’t spawn,use OTP

Downsides

Erlang changes very slowly

3rd party libraries

standard librarycan be inconsistent

packagemanagement

Questions

http://spkr8.com/t/7806Phil: @philtolandhttp://github.com/tolandhttp://philtoland.comMike: @lifeinzemblahttp://github.com/msassak

Polyglot parallelism

by Phillip Toland

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 56

Statistics

Polyglot parallelism Presentation Transcript

http://lanyrd.com	50
http://www.hanrss.com	6