• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ErLounge SF/Bay: 2010.01.12 Christian Westbrook / CoTweet
 

ErLounge SF/Bay: 2010.01.12 Christian Westbrook / CoTweet

on

  • 1,133 views

Christian Westbrook / CoTweet (cw@cotweet.com)

Christian Westbrook / CoTweet (cw@cotweet.com)

at

ErLounge SF/Bay
Jan. 12th, 2010 @ Pier 38

http://meetup.com/erlounge

Statistics

Views

Total Views
1,133
Views on SlideShare
1,133
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Christian Westbrook2010.1.12 ErLounge SF/Bay <br /> cw@props.org meetup.com/erlounge
  • Thanks for coming out
  • Nothing technically wrong with this: did, in fact, get working w/ plists <br /> - parallel processing, but not concurrent services <br /> - stateless! robot has no idea what&#x2019;s going on, ever. <br /> ... one of the misconceptions some beginners have about erlang ->
  • all state: robot config, list of channels to service, perceived state of: { twitter api, db, cache } <br /> Benefits: smooth workload rather than bursts tapering off each iteration of service loop <br /> can mix in business logic to govern usage <br /> But: worst case, worker procs can still block waiting for database -- precluding cache insert
  • Benefits:
  • :
  • don&#x2019;t hit anything :3

ErLounge SF/Bay: 2010.01.12 Christian Westbrook / CoTweet ErLounge SF/Bay: 2010.01.12 Christian Westbrook / CoTweet Presentation Transcript

  • ErLounge/SF Bay 2010.1.12
  • Welcome
  • hello_world.erl
  • wello_horld.erl
  • wello_horld.erl
  • wello_horld.erl
  • CoTweet Robot: Task
  • CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ )
  • CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ ) channel update req. 4 http requests on average = n million http requests / hr
  • CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ ) • Minimize latency: • Ideally < 300s between updates
  • CoTweet Robot: Task • Pull updates from Twitter API: • n * 10,000 channels ( n is going up -- o/ ) • Minimize latency (< 300s between updates) • <blink>Survivable</blink> !
  • Primordial Ooze: Ruby Note: not a real robot
  • Primordial Ooze: Ruby
  • Primordial Ooze: Ruby Like any tool, rails is great for some things...
  • Robot Evolution: I
  • Robot Evolution: I • Main process: receive/after loop: service_loop() -> spawn(?MODULE, do_work, []), receive after SleepyTimeMS * 1000 -> ok end, service_loop().
  • Robot Evolution: I • Main process: receive/after loop • Reads user info from database • For every N records ...
  • Robot Evolution: I • Main process: receive/after loop • Reads user info from database • For every N records: • spawn() a worker to process bucket: • Retrieve updates from Twitter API • Insert into database (returning ID) • Insert (using ID) into memcached
  • spawn() responsibly. • I <3 supervisors • trap_exit + PID = spawn_link(M, F, [A]). • {PID,Ref} = spawn_monitor(M,F,[A]).
  • Single Assignment != Stateless
  • Single Assignment != Stateless • gen_server, gen_event, gen_fsm: State -record(State, {foo,bar}). handle_call({set_foo, V}, _From, State) -> {reply, State#state{ foo = V };
  • Single Assignment != Stateless • ets • Many concurrent reads? No problem. start() -> ets:new(state, [named_table, public]). set_foo(V) -> ets:insert(state, {foo,V}).
  • Single Assignment != Stateless • ets • Many concurrent reads? No problem. • Many concurrent writes? • May need to Serialize through a gen_server. handle_call({set_foo, V}, _From, State) -> ets:insert(state, {foo, V}), {reply, State};
  • Single Assignment != Stateless • gen_server, gen_event, gen_fsm: State • ets: app-wide • serialize writes through a gen_server • mnesia: persistent • process dictionary: put(K,V) / get(K) • Tiny, transient, process-level • (think: private):
  • Robot Evolution: II
  • Robot Evolution: II • All state held in ets • Robot comprises many ‘services:’ • Feeder: poll db as needed, update ets • Scheduler: add tasks to work queue • Business logic: • even work load, limit resource(s) • Dispatcher: service queue for workers
  • Hm, our “workers” may block waiting for result(i.e., ID) of an insert from DB . . .
  • Hm, our “workers” may block waiting for result(i.e., ID) of an insert from DB . . . Needs more OTP.
  • Robot Evolution: III • gen_event as simplest MUX ever • no time for downtime: hot deployment
  • gen_event • Extraordinarily easy to multiplex: • Create an event manager: gen_event • Add handlers • Fire an event • Not just for logging . . .
  • Deployment -module(my_app). -export([reload_modules/0]). -define(APP_MODS, [my_app, my_service]). reload_modules() -> lists:map(fun(M) -> code:load_file(M) end, ?APP_MODS). See: http://www.erlang.org/cgi-bin/ezmlm-cgi/4/36236
  • Open Source “Standing on the shoulders of giants...” -- Bob Ippolito S. Mestrom with K. Newby
  • Open Source • plists (parallel list operations) • lhttpc (much lighter than inets) • rfc_4627 (utf8 JSON decoding) • epgsql (PostGreSQL driver) • erlang_twitter (Twitter API via xmerl) • merle (erlang memcached adapter) • mochi (congrats to MochiMedia!)
  • Thanks • SocialMedia -- Ken Thom • CoTweet • You
  • ErLounge • Grab another drink • Share your stories