The document discusses a case study at Rackspace, addressing the challenges of managing 20,000 network devices across multiple data centers, with a need for high throughput and efficient back-end solutions. It details the evolution from a multi-Ruby app structure to a more scalable Erlang and MongoDB setup, emphasizing the importance of concurrency and the flexibility of the new API design. Additionally, it highlights the role of Webmachine in handling HTTP requests within this architecture, along with various technical considerations related to Erlang's features.

1. Polyglot Parallelism
A Case Study in Using Erlang
and Ruby at Rackspace

2. The Problem
Part 1

3. 20,000
network devices

4. 9 Datacenters,
3 Continents

5. devices not designed for
high-throughput
management

6. we need a high
throughput solution

7. the time spent in I/O
is the primary
bottleneck

8. if you want to speed
things up you have to
talk to more devices
in parallel

9. The Problem
Part II

10. huge blobs of data

11. lots of backups
equals big database

12. ad-hoc searching
is difficult but
important

13. customer SLA means
need to restore from
backup quickly

14. an event must be
generated for each
device interaction

15. migrations are
problematic with
that much data

16. rigid schema made
adapting to new
devices difficult

17. each device type has
different properties

18. “backup” means
different things for
each device type

19. need to grow with
the business

20. Previous
Solution

21. multiple Ruby apps

22. difficult to scale

23. vendor device
managers

24. New Solution

26. the simplest thing
that could possibly
work

27. most db writes
come from
scheduled jobs

28. Other Clients
Rails ReST API
Erlang
MongoDB Network
Devices

30. Joe Armstrong

31. AXD301
ATM Switch

32. 99.9999999%

33. Functional

34. Dynamically Typed

35. Single Assignment

36. A = 1. %=> 1
A = 2. %=> badmatch

37. [B, 2, C] = [1, 2, 3].
B = 1. %=> 1
C = 3. %=> 3

38. Immutable
Data
Structures

39. D = dict:new().
D1 = dict:store(foo, 1, D).
D2 = dict:store(bar, 2, D1).

40. Concurrency
Oriented

41. -module(fact).
-export([fac/1]).
fac(0) -> 1;
fac(N) -> N * fac(N-1).

42. -module(quicksort).
-export([quicksort/1]).
quicksort([]) -> [];
quicksort([Pivot|Rest]) ->
quicksort([Front || Front <- Rest, Front < Pivot])
++ [Pivot] ++
quicksort([Back || Back <- Rest, Back >= Pivot]).

43. Details

44. jobs framework

45. Runner
Callback
Workers Module

46. Callback
Runner Worker Module
start
ready
item
process
process
ready
.
.
.
stop

47. “behaviour” is interface

48. behaviour_info(callbacks) -> [
{init, 1},
{process_item, 3},
{worker_died, 5},
{job_stopping, 1},
{job_complete, 2}
].

49. running({worker_ready, WorkerPid, ok}, S) ->
case queue:out(S#state.items) of
{empty, I2} ->
stop_worker(WorkerPid, S),
{next_state, complete, S#state{items = I2}};
{{value, Item}, I2} ->
job_worker:process(WorkerPid, Item, now(),
S#state.job_state),
{next_state, running, S#state{items = I2}}
end;

50. handle_info({'DOWN', _, process, WorkerPid, Info},
StateName, S) ->
{Item, StartTime} = clear_worker(WorkerPid, S),
Callback = S#state.callback,
spawn(Callback, worker_died,
[Item, WorkerPid, StartTime,
Info, S#state.job_state]),
%% Start a replacement worker
start_workers(1, Callback),
{next_state, StateName, S};

51. handle_cast({process, Item, StartTime, JS}, S) ->
Callback = S#state.callback,
Continue =
try
Callback:process_item(Item, StartTime, JS)
catch
throw: Error ->
error_logger:error_report(Error),
ok
end,
job_runner:worker_ready(S#state.runner, self(),
Continue),
{noreply, S}.

52. story time

53. ReSTful API
with Webmachine
The Convention Over Configuration Webserver

54. http://webmachine.basho.com
HTTP Request Lifecycle Diagram

55. If you know HTTP
Webmachine Is Simple
As Proven by the “Number of Types of Things”
Measurement of Complexity

56. The 3 Most Important Types of
Things In Webmachine
1. Dispatch Rules (pure data--barely a thing!)
2. Resources (composed of simple functions!)
3. Requests (simple get/set interface!)

57. Dispatch Rules
{ ["devices", server], device_resource, [] }
GET /devices/12345
Webmachine inspects the device_resource module for
defined callbacks, and sets the Request record’s “server”
value to 12345.

58. Resources
• POEM (Plain Old Erlang Module)
• Composed of referentially transparent functions*
• Functions are callbacks into the request lifecycle
• Approximately 30 possible callback functions, e.g.:
• resource_exists → 404 Not Found
• is_authorized → 401 Not Authorized
* mostly

59. Resource Functions
Perma-404
resource_exists(Request, Context) ->
{false, Request, Context}.
Lucky Auth
is_authorized(Request, Context) ->
S = calendar:time_to_seconds(now()),
case S rem 2 of
0 -> {true, Request, Context};
1 -> {“Basic realm=lucky”, Request, Context}
end.

60. Requests
• The first argument to each resource function
• Set and read request & response data
RemoteIP = wrq:peer(Request).
wrq:set_resp_header(“X-Answer”, “42”, Request).

61. Retrieving a JSON Firewall Representation
content_types_provided(Request, Context) ->
Types = [{"application/json", to_json}],
{Types, Request, Context}.
to_json(Request, Context) ->
Device = proplists:get_value(device, Context),
UserId = get_user_id(Request),
case fe_api_firewall:get_config(Device, UserId) of
{ok, Config} ->
success_response(Config, Request, Context);
{error, Reason} ->
error_response(502, Reason, Request, Context)
end.

62. Gotchas

63. primitive obsession

64. string-ish
“hi how are you”
<<“hello there”>>
[<<"easy as ">>, [$a, $b, $c], "
☺n"].

65. hashes vs records

66. to loop is human,
to recur divine

67. Erlang conditionals
always return a value

68. design for testability

69. don’t
spawn,
use OTP

70. Downsides

71. Erlang changes very
slowly

72. 3rd party libraries

73. standard library
can be inconsistent

74. package
management

75. Questions

76. http://spkr8.com/t/7806
Phil: @philtoland
http://github.com/toland
http://philtoland.com
Mike: @lifeinzembla
http://github.com/msassak