• Like

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Pools & Streams: A Naive Approach To Scalable & Flexible Social Feeds.

  • 1,727 views
Uploaded on

My talk for the Rejectconf, prior to the Conferencia Rails (Madrid, November '10) …

My talk for the Rejectconf, prior to the Conferencia Rails (Madrid, November '10)

In this presentation I talk about an architectural pattern for social sites built with Ruby on Rails, extracted from our work in http://peercouture.com

Feel free to address any doubt, suggestion, or comment to manuel@simplelogica.net

Thanks for your attention.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,727
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
15
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide











































Transcript

  • 1. STREAMS & POOLS A Naive Approach To Scalable & Flexible Social Feeds. @mort - #rejectconf - madrid, nov 10
  • 2. Para Nando García Samblas
  • 3. PREAMBLE: PEER COUTURE & ME
  • 4. Manuel González Noriega: I co-founded Simplelógica (http://simplelogica.net) on 2002. On 2009 we soft launched Peer Couture (http://www.peercouture.com), a easygoing place to show off your clothes and take a peek into other people’s wardrobes without being charged with trespassing. A bit different from your average ‘fashion site’ in that wrinkles, stains, and worn- out pajamas are allowed and cheerfully celebrated.
  • 5. STATING THE PROBLEM
  • 6. Being a simple, flexible, and efficient approach to the issue of presenting social objects scattered across a number of ordered collections (streams) and the composition of dynamic aggregates from several of those collections (pools) through the judicious use of the ‘follow’ pattern.
  • 7. Peer Couture item page
  • 8. Peer Couture user page
  • 9. Peer Couture brand page
  • 10. Peer Couture tag page
  • 11. STREAMS
  • 12. A STREAM IS • A list of heterogeneous or homogeneous objects, ordered according some specific criteria • Reification of the "SELECT * FROM wadus ORDER BY foo DESC" / i.e. we turn that result set into a business object/model/ resource.
  • 13. A STREAM IS class Stream < ActiveRecord::Base belongs_to :subscribable, :polymorphic => true has_many :stream_entries, :dependent => :destroy validates_uniqueness_of :subscribable_id, :scope => [:subscribable_type, :stream_type] validates_exclusion_of :stream_type, :in => %w(public) .... end
  • 14. so a stream • Belongs to a polymorphic subscribable • Has a handy stream_type (public, tag, shop, etc.) • Has many stream entries
  • 15. LET ‘EM DROWN class StreamObserver < ActiveRecord::Observer observe :item def after_create(item) item.send_later(:push_to_streams) end ... end
  • 16. class Item < ActiveRecord::Base def push_to_streams push_to_public_stream push_to_owner_stream push_to_shop_stream if shop? push_to_tag_streams unless tags.blank? end def push_to_owner_stream push_to_stream(user) end def push_to_tag_streams tags.each {|tag| push_to_stream(tag) } end def push_to_shop_stream push_to_stream(self.shop) end ... end
  • 17. class Item < ActiveRecord::Base ... def push_to_stream(subscribable, stype = nil) stype ||= "#{subscribable.class.to_s.downcase}_items" s = Stream.find_by_subscribable_and_type(subscribable, stype) s << self unless s.nil? end ... end
  • 18. class Stream < ActiveRecord::Base ... def <<(streamable) self.stream_entries.create(:streamable => streamable, :content => streamable.content_for_stream) end ... end
  • 19. STREAM ENTRIES • Basic units of the stream • acts_as_list • Most crucial aspect: it stores a denormalized copy of the source's metadata attributes.
  • 20. class StreamEntry < ActiveRecord::Base belongs_to :stream belongs_to :streamable, :polymorphic => true acts_as_list :scope => :stream_id ... end
  • 21. class Item < ActiveRecord::Base .. def content_for_stream ActiveRecord::Base.include_root_in_json = false inc = { :user => { :methods => [:display_name] } } #inc[:shop] = {} if shop? json = self.to_json(:include => inc) ActiveRecord::Base.include_root_in_json = true json end .. end
  • 22. THE CIRCLE OF LIFE • Be aware of the the source's lifecycle. Changes to the source must flow downwards to the streams.
  • 23. class StreamObserver < ActiveRecord::Observer observe :item .. def after_update(item) item.send_later(:update_streams) end .. end
  • 24. WHAT WE GOT SO FAR • /users/foo/items => stream/xx • /tags/foo => stream/xx • /shops/foo => stream/xx • No DB joins. • An unified architectural layer.
  • 25. POOLS
  • 26. THE FOLLOW PATTERN
  • 27. THE FOLLOW PATTERN
  • 28. ‘Follow’, and equivalent verbs, all result in the merging of several streams into one single, customized aggregate (pool)
  • 29. TWITTER TIMELINE
  • 30. PEER COUTURE CATWALK
  • 31. STREAM SUBSCRIPTION class StreamSubscription < ActiveRecord::Base belongs_to :user belongs_to :stream # “Wadus is now following you on Twitter” after_create :notify_stream_owner .. end
  • 32. class StreamEntry < ActiveRecord::Base .. after_create :push_to_pools .. private def push_to_pools stream.stream_subscriptions.each { |sub| sub << {:stream_entry_id => self.id, :content => self.content, :streamable_created_at => streamable.created_at, :streamable_updated_at => streamable.updated_at} } end .. end
  • 33. class StreamSubscription < ActiveRecord::Base .. def <<(options) self.user.pool_entries.create(options.merge(:reason => self.reason)) end .. end class PoolEntry < ActiveRecord::Base .. belongs_to :user belongs_to :stream_entry .. end
  • 34. POOL ENTRY • Again, acts_as_list. • Should have the capability of random inserts and reordering according a specific criteria. • Second copy of the source's metadata floating around. • Keep a 'reason' helper available to remind the user of why that item is featured on her timeline/catwalk/wadus. • Again, it must remain synchronized to the source's lifecycle => Push changes downstream. • Keep improving on grouping of entries.
  • 35. class Array def pack_entries eids = {} # Por cada entrada entries.each do |entry| # Creamos un identificador str = entry.identifier # Si el identificador existe en el hash de almacenaje if eids.has_key?(str) # Modificamos el reason existente, añadiendole el nuevo eids[str].reason = "#{eids[str].reason}|#{entry.reason}" else # Si no existe, añadimos al array de almacenaje eids[str] = entry end end # Devolvemos el hash de almacenaje convertido en array y ordenado por posicion eids.values.sort {|x,y| y.streamable_created_at <=> x.streamable_created_at } end
  • 36. OTHER POOLS UNDER THE SUN • User input is not the only conceivable way of creating a pool. • Editorially-defined pools: "All the items tagged with a color name", "Items uploaded from the EU" • Rule-based pools: "All the items from the most popular brands this month"
  • 37. CITIUS, ALTIUS, FORTIUS
  • 38. NoSQL.
  • 39. A robust, carefully thought, subsystem of chained objects. With Events (Comment, Fav), EventListeners (OnComment, OnFav), bubbling up/down of events. So you could favorite a pool entry, and the event would "swim upstream" towards the source (item, tweet). Or you could add a comment on the item's page, and that would "swim downstream" to the stream, and eventually to any number of pools. Does this fuck REST up? MVC?
  • 40. Easy consistency check, propagating some kind of checksum of the source's contents to the streams and pools, enabling easy background verification of whether source-stream-pool remain in synch.
  • 41. Embedded images/resources: Base64 + data: URIS for storing the source's assets directly in the serialized content of stream entries and pool entries. Uses disk space/saves on HTTP requests. Limited support for data: URIS, JS-fallback.
  • 42. CAVEAT CODER • Lots of experimenting to be done. • Fanout-on-write vs. Fanout-on-read (http://bit.ly/acuDPT) • Maybe only appropriate for medium- sized sites?
  • 43. FIN.