Pools & Streams: A Naive Approach To Scalable & Flexible Social Feeds.

STREAMS & POOLS A Naive Approach To Scalable & Flexible Social Feeds. @mort - #rejectconf - madrid, nov 10

Para Nando García Samblas

PREAMBLE: PEER COUTURE & ME

Manuel González Noriega: I co-founded Simplelógica (http://simplelogica.net) on 2002. On 2009 we soft launched Peer Couture (http://www.peercouture.com), a easygoing place to show off your clothes and take a peek into other people’s wardrobes without being charged with trespassing. A bit different from your average ‘fashion site’ in that wrinkles, stains, and worn- out pajamas are allowed and cheerfully celebrated.

STATING THE PROBLEM

Being a simple, ﬂexible, and efﬁcient approach to the issue of presenting social objects scattered across a number of ordered collections (streams) and the composition of dynamic aggregates from several of those collections (pools) through the judicious use of the ‘follow’ pattern.

Peer Couture item page

Peer Couture user page

Peer Couture brand page

Peer Couture tag page

STREAMS

A STREAM IS • A list of heterogeneous or homogeneous objects, ordered according some speciﬁc criteria • Reiﬁcation of the "SELECT * FROM wadus ORDER BY foo DESC" / i.e. we turn that result set into a business object/model/ resource.

A STREAM IS class Stream < ActiveRecord::Base belongs_to :subscribable, :polymorphic => true has_many :stream_entries, :dependent => :destroy validates_uniqueness_of :subscribable_id, :scope => [:subscribable_type, :stream_type] validates_exclusion_of :stream_type, :in => %w(public) .... end

so a stream • Belongs to a polymorphic subscribable • Has a handy stream_type (public, tag, shop, etc.) • Has many stream entries

LET ‘EM DROWN class StreamObserver < ActiveRecord::Observer observe :item def after_create(item) item.send_later(:push_to_streams) end ... end

class Item < ActiveRecord::Base def push_to_streams push_to_public_stream push_to_owner_stream push_to_shop_stream if shop? push_to_tag_streams unless tags.blank? end def push_to_owner_stream push_to_stream(user) end def push_to_tag_streams tags.each {|tag| push_to_stream(tag) } end def push_to_shop_stream push_to_stream(self.shop) end ... end

class Item < ActiveRecord::Base ... def push_to_stream(subscribable, stype = nil) stype ||= "#{subscribable.class.to_s.downcase}_items" s = Stream.find_by_subscribable_and_type(subscribable, stype) s << self unless s.nil? end ... end

class Stream < ActiveRecord::Base ... def <<(streamable) self.stream_entries.create(:streamable => streamable, :content => streamable.content_for_stream) end ... end

STREAM ENTRIES • Basic units of the stream • acts_as_list • Most crucial aspect: it stores a denormalized copy of the source's metadata attributes.

class StreamEntry < ActiveRecord::Base belongs_to :stream belongs_to :streamable, :polymorphic => true acts_as_list :scope => :stream_id ... end

class Item < ActiveRecord::Base .. def content_for_stream ActiveRecord::Base.include_root_in_json = false inc = { :user => { :methods => [:display_name] } } #inc[:shop] = {} if shop? json = self.to_json(:include => inc) ActiveRecord::Base.include_root_in_json = true json end .. end

THE CIRCLE OF LIFE • Be aware of the the source's lifecycle. Changes to the source must ﬂow downwards to the streams.

class StreamObserver < ActiveRecord::Observer observe :item .. def after_update(item) item.send_later(:update_streams) end .. end

WHAT WE GOT SO FAR • /users/foo/items => stream/xx • /tags/foo => stream/xx • /shops/foo => stream/xx • No DB joins. • An uniﬁed architectural layer.

THE FOLLOW PATTERN

‘Follow’, and equivalent verbs, all result in the merging of several streams into one single, customized aggregate (pool)

TWITTER TIMELINE

PEER COUTURE CATWALK

STREAM SUBSCRIPTION class StreamSubscription < ActiveRecord::Base belongs_to :user belongs_to :stream # “Wadus is now following you on Twitter” after_create :notify_stream_owner .. end

class StreamEntry < ActiveRecord::Base .. after_create :push_to_pools .. private def push_to_pools stream.stream_subscriptions.each { |sub| sub << {:stream_entry_id => self.id, :content => self.content, :streamable_created_at => streamable.created_at, :streamable_updated_at => streamable.updated_at} } end .. end

class StreamSubscription < ActiveRecord::Base .. def <<(options) self.user.pool_entries.create(options.merge(:reason => self.reason)) end .. end class PoolEntry < ActiveRecord::Base .. belongs_to :user belongs_to :stream_entry .. end

POOL ENTRY • Again, acts_as_list. • Should have the capability of random inserts and reordering according a speciﬁc criteria. • Second copy of the source's metadata ﬂoating around. • Keep a 'reason' helper available to remind the user of why that item is featured on her timeline/catwalk/wadus. • Again, it must remain synchronized to the source's lifecycle => Push changes downstream. • Keep improving on grouping of entries.

class Array def pack_entries eids = {} # Por cada entrada entries.each do |entry| # Creamos un identificador str = entry.identifier # Si el identificador existe en el hash de almacenaje if eids.has_key?(str) # Modificamos el reason existente, añadiendole el nuevo eids[str].reason = "#{eids[str].reason}|#{entry.reason}" else # Si no existe, añadimos al array de almacenaje eids[str] = entry end end # Devolvemos el hash de almacenaje convertido en array y ordenado por posicion eids.values.sort {|x,y| y.streamable_created_at <=> x.streamable_created_at } end

OTHER POOLS UNDER THE SUN • User input is not the only conceivable way of creating a pool. • Editorially-deﬁned pools: "All the items tagged with a color name", "Items uploaded from the EU" • Rule-based pools: "All the items from the most popular brands this month"

CITIUS, ALTIUS, FORTIUS

NoSQL.

A robust, carefully thought, subsystem of chained objects. With Events (Comment, Fav), EventListeners (OnComment, OnFav), bubbling up/down of events. So you could favorite a pool entry, and the event would "swim upstream" towards the source (item, tweet). Or you could add a comment on the item's page, and that would "swim downstream" to the stream, and eventually to any number of pools. Does this fuck REST up? MVC?

Easy consistency check, propagating some kind of checksum of the source's contents to the streams and pools, enabling easy background veriﬁcation of whether source-stream-pool remain in synch.

Embedded images/resources: Base64 + data: URIS for storing the source's assets directly in the serialized content of stream entries and pool entries. Uses disk space/saves on HTTP requests. Limited support for data: URIS, JS-fallback.

CAVEAT CODER • Lots of experimenting to be done. • Fanout-on-write vs. Fanout-on-read (http://bit.ly/acuDPT) • Maybe only appropriate for medium- sized sites?

Pools & Streams: A Naive Approach To Scalable & Flexible Social Feeds.

by Manuel Noriega

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

Pools & Streams: A Naive Approach To Scalable & Flexible Social Feeds. Presentation Transcript