• Like


Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Pools & Streams: A Naive Approach To Scalable & Flexible Social Feeds.

Uploaded on

My talk for the Rejectconf, prior to the Conferencia Rails (Madrid, November '10) …

My talk for the Rejectconf, prior to the Conferencia Rails (Madrid, November '10)

In this presentation I talk about an architectural pattern for social sites built with Ruby on Rails, extracted from our work in http://peercouture.com

Feel free to address any doubt, suggestion, or comment to manuel@simplelogica.net

Thanks for your attention.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. STREAMS & POOLS A Naive Approach To Scalable & Flexible Social Feeds. @mort - #rejectconf - madrid, nov 10
  • 2. Para Nando García Samblas
  • 4. Manuel González Noriega: I co-founded Simplelógica (http://simplelogica.net) on 2002. On 2009 we soft launched Peer Couture (http://www.peercouture.com), a easygoing place to show off your clothes and take a peek into other people’s wardrobes without being charged with trespassing. A bit different from your average ‘fashion site’ in that wrinkles, stains, and worn- out pajamas are allowed and cheerfully celebrated.
  • 6. Being a simple, flexible, and efficient approach to the issue of presenting social objects scattered across a number of ordered collections (streams) and the composition of dynamic aggregates from several of those collections (pools) through the judicious use of the ‘follow’ pattern.
  • 7. Peer Couture item page
  • 8. Peer Couture user page
  • 9. Peer Couture brand page
  • 10. Peer Couture tag page
  • 11. STREAMS
  • 12. A STREAM IS • A list of heterogeneous or homogeneous objects, ordered according some specific criteria • Reification of the "SELECT * FROM wadus ORDER BY foo DESC" / i.e. we turn that result set into a business object/model/ resource.
  • 13. A STREAM IS class Stream < ActiveRecord::Base belongs_to :subscribable, :polymorphic => true has_many :stream_entries, :dependent => :destroy validates_uniqueness_of :subscribable_id, :scope => [:subscribable_type, :stream_type] validates_exclusion_of :stream_type, :in => %w(public) .... end
  • 14. so a stream • Belongs to a polymorphic subscribable • Has a handy stream_type (public, tag, shop, etc.) • Has many stream entries
  • 15. LET ‘EM DROWN class StreamObserver < ActiveRecord::Observer observe :item def after_create(item) item.send_later(:push_to_streams) end ... end
  • 16. class Item < ActiveRecord::Base def push_to_streams push_to_public_stream push_to_owner_stream push_to_shop_stream if shop? push_to_tag_streams unless tags.blank? end def push_to_owner_stream push_to_stream(user) end def push_to_tag_streams tags.each {|tag| push_to_stream(tag) } end def push_to_shop_stream push_to_stream(self.shop) end ... end
  • 17. class Item < ActiveRecord::Base ... def push_to_stream(subscribable, stype = nil) stype ||= "#{subscribable.class.to_s.downcase}_items" s = Stream.find_by_subscribable_and_type(subscribable, stype) s << self unless s.nil? end ... end
  • 18. class Stream < ActiveRecord::Base ... def <<(streamable) self.stream_entries.create(:streamable => streamable, :content => streamable.content_for_stream) end ... end
  • 19. STREAM ENTRIES • Basic units of the stream • acts_as_list • Most crucial aspect: it stores a denormalized copy of the source's metadata attributes.
  • 20. class StreamEntry < ActiveRecord::Base belongs_to :stream belongs_to :streamable, :polymorphic => true acts_as_list :scope => :stream_id ... end
  • 21. class Item < ActiveRecord::Base .. def content_for_stream ActiveRecord::Base.include_root_in_json = false inc = { :user => { :methods => [:display_name] } } #inc[:shop] = {} if shop? json = self.to_json(:include => inc) ActiveRecord::Base.include_root_in_json = true json end .. end
  • 22. THE CIRCLE OF LIFE • Be aware of the the source's lifecycle. Changes to the source must flow downwards to the streams.
  • 23. class StreamObserver < ActiveRecord::Observer observe :item .. def after_update(item) item.send_later(:update_streams) end .. end
  • 24. WHAT WE GOT SO FAR • /users/foo/items => stream/xx • /tags/foo => stream/xx • /shops/foo => stream/xx • No DB joins. • An unified architectural layer.
  • 25. POOLS
  • 28. ‘Follow’, and equivalent verbs, all result in the merging of several streams into one single, customized aggregate (pool)
  • 31. STREAM SUBSCRIPTION class StreamSubscription < ActiveRecord::Base belongs_to :user belongs_to :stream # “Wadus is now following you on Twitter” after_create :notify_stream_owner .. end
  • 32. class StreamEntry < ActiveRecord::Base .. after_create :push_to_pools .. private def push_to_pools stream.stream_subscriptions.each { |sub| sub << {:stream_entry_id => self.id, :content => self.content, :streamable_created_at => streamable.created_at, :streamable_updated_at => streamable.updated_at} } end .. end
  • 33. class StreamSubscription < ActiveRecord::Base .. def <<(options) self.user.pool_entries.create(options.merge(:reason => self.reason)) end .. end class PoolEntry < ActiveRecord::Base .. belongs_to :user belongs_to :stream_entry .. end
  • 34. POOL ENTRY • Again, acts_as_list. • Should have the capability of random inserts and reordering according a specific criteria. • Second copy of the source's metadata floating around. • Keep a 'reason' helper available to remind the user of why that item is featured on her timeline/catwalk/wadus. • Again, it must remain synchronized to the source's lifecycle => Push changes downstream. • Keep improving on grouping of entries.
  • 35. class Array def pack_entries eids = {} # Por cada entrada entries.each do |entry| # Creamos un identificador str = entry.identifier # Si el identificador existe en el hash de almacenaje if eids.has_key?(str) # Modificamos el reason existente, añadiendole el nuevo eids[str].reason = "#{eids[str].reason}|#{entry.reason}" else # Si no existe, añadimos al array de almacenaje eids[str] = entry end end # Devolvemos el hash de almacenaje convertido en array y ordenado por posicion eids.values.sort {|x,y| y.streamable_created_at <=> x.streamable_created_at } end
  • 36. OTHER POOLS UNDER THE SUN • User input is not the only conceivable way of creating a pool. • Editorially-defined pools: "All the items tagged with a color name", "Items uploaded from the EU" • Rule-based pools: "All the items from the most popular brands this month"
  • 38. NoSQL.
  • 39. A robust, carefully thought, subsystem of chained objects. With Events (Comment, Fav), EventListeners (OnComment, OnFav), bubbling up/down of events. So you could favorite a pool entry, and the event would "swim upstream" towards the source (item, tweet). Or you could add a comment on the item's page, and that would "swim downstream" to the stream, and eventually to any number of pools. Does this fuck REST up? MVC?
  • 40. Easy consistency check, propagating some kind of checksum of the source's contents to the streams and pools, enabling easy background verification of whether source-stream-pool remain in synch.
  • 41. Embedded images/resources: Base64 + data: URIS for storing the source's assets directly in the serialized content of stream entries and pool entries. Uses disk space/saves on HTTP requests. Limited support for data: URIS, JS-fallback.
  • 42. CAVEAT CODER • Lots of experimenting to be done. • Fanout-on-write vs. Fanout-on-read (http://bit.ly/acuDPT) • Maybe only appropriate for medium- sized sites?
  • 43. FIN.