ErLounge SF/Bay: 2010.01.12 Christian Westbrook / CoTweet

1,041 views

Published on

Christian Westbrook / CoTweet (cw@cotweet.com)

at

ErLounge SF/Bay
Jan. 12th, 2010 @ Pier 38

http://meetup.com/erlounge

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,041
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Christian Westbrook2010.1.12 ErLounge SF/Bay
    cw@props.org meetup.com/erlounge
  • Thanks for coming out
  • Nothing technically wrong with this: did, in fact, get working w/ plists
    - parallel processing, but not concurrent services
    - stateless! robot has no idea what’s going on, ever.
    ... one of the misconceptions some beginners have about erlang ->
  • all state: robot config, list of channels to service, perceived state of: { twitter api, db, cache }
    Benefits: smooth workload rather than bursts tapering off each iteration of service loop
    can mix in business logic to govern usage
    But: worst case, worker procs can still block waiting for database -- precluding cache insert
  • Benefits:
  • :
  • don’t hit anything :3
  • ErLounge SF/Bay: 2010.01.12 Christian Westbrook / CoTweet

    1. 1. ErLounge/SF Bay 2010.1.12
    2. 2. Welcome
    3. 3. hello_world.erl
    4. 4. wello_horld.erl
    5. 5. wello_horld.erl
    6. 6. wello_horld.erl
    7. 7. CoTweet Robot: Task
    8. 8. CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ )
    9. 9. CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ ) channel update req. 4 http requests on average = n million http requests / hr
    10. 10. CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ ) • Minimize latency: • Ideally < 300s between updates
    11. 11. CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ ) • Minimize latency (< 300s between updates) • <blink>Survivable</blink> !
    12. 12. Primordial Ooze: Ruby Note: not a real robot
    13. 13. Primordial Ooze: Ruby
    14. 14. Primordial Ooze: Ruby Like any tool, rails is great for some things...
    15. 15. Robot Evolution: I
    16. 16. Robot Evolution: I • Main process: receive/after loop: service_loop() -> spawn(?MODULE, do_work, []), receive after SleepyTimeMS * 1000 -> ok end, service_loop().
    17. 17. Robot Evolution: I • Main process: receive/after loop • Reads user info from database • For every N records ...
    18. 18. Robot Evolution: I • Main process: receive/after loop • Reads user info from database • For every N records: • spawn() a worker to process bucket: • Retrieve updates from Twitter API • Insert into database (returning ID) • Insert (using ID) into memcached
    19. 19. spawn() responsibly. • I <3 supervisors • trap_exit + PID = spawn_link(M, F, [A]). • {PID,Ref} = spawn_monitor(M,F,[A]).
    20. 20. Single Assignment != Stateless
    21. 21. Single Assignment != Stateless • gen_server, gen_event, gen_fsm: State -record(State, {foo,bar}). handle_call({set_foo, V}, _From, State) -> {reply, State#state{ foo = V };
    22. 22. Single Assignment != Stateless • ets • Many concurrent reads? No problem. start() -> ets:new(state, [named_table, public]). set_foo(V) -> ets:insert(state, {foo,V}).
    23. 23. Single Assignment != Stateless • ets • Many concurrent reads? No problem. • Many concurrent writes? • May need to Serialize through a gen_server. handle_call({set_foo, V}, _From, State) -> ets:insert(state, {foo, V}), {reply, State};
    24. 24. Single Assignment != Stateless • gen_server, gen_event, gen_fsm: State • ets: app-wide • serialize writes through a gen_server • mnesia: persistent • process dictionary: put(K,V) / get(K) • Tiny, transient, process-level • (think: private):
    25. 25. Robot Evolution: II
    26. 26. Robot Evolution: II • All state held in ets • Robot comprises many ‘services:’ • Feeder: poll db as needed, update ets • Scheduler: add tasks to work queue • Business logic: • even work load, limit resource(s) • Dispatcher: service queue for workers
    27. 27. Hm, our “workers” may block waiting for result(i.e., ID) of an insert from DB . . .
    28. 28. Hm, our “workers” may block waiting for result(i.e., ID) of an insert from DB . . . Needs more OTP.
    29. 29. Robot Evolution: III • gen_event as simplest MUX ever • no time for downtime: hot deployment
    30. 30. gen_event • Extraordinarily easy to multiplex: • Create an event manager: gen_event • Add handlers • Fire an event • Not just for logging . . .
    31. 31. Deployment -module(my_app). -export([reload_modules/0]). -define(APP_MODS, [my_app, my_service]). reload_modules() -> lists:map(fun(M) -> code:load_file(M) end, ?APP_MODS). See: http://www.erlang.org/cgi-bin/ezmlm-cgi/4/36236
    32. 32. Open Source “Standing on the shoulders of giants...” -- Bob Ippolito S. Mestrom with K. Newby
    33. 33. Open Source • plists (parallel list operations) • lhttpc (much lighter than inets) • rfc_4627 (utf8 JSON decoding) • epgsql (PostGreSQL driver) • erlang_twitter (Twitter API via xmerl) • merle (erlang memcached adapter) • mochi (congrats to MochiMedia!)
    34. 34. Thanks • SocialMedia -- Ken Thom • CoTweet • You
    35. 35. ErLounge • Grab another drink • Share your stories

    ×