Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood


Published on

In this session I'd share the design, architecture and implementation of some of the most common elements of any social platform - Open API, profiles, searches, lists and activity streams. These "pillers" of a social platform bear most of the weight behind a jazzy UI, and scaling them has its own challenges. I will also talk about how we built the Social Platform at IGN from ground up, including not-so-unique challenges like integration with legacy systems.

Published in: Technology
1 Comment
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood

  1. 1.
  2. 2. Components of Social Platform<br />Challenges<br />Technology<br />API<br />Engineering Process<br />
  3. 3. User Profile<br />Relationships<br />Activity Streams<br />
  4. 4. Basic user information<br />Extended information based on the context<br />Player Cards<br />Bragging rights <br />Points, levels<br />Achievements, badges<br />Activities<br />
  5. 5. Friending<br />Unidirectional (the Twitter model)<br />Bidirectional (the Facebok model) <br />Association<br />Comments<br />Ratings, Thumbs<br />Bookmarking/favoriting<br />Recommendations<br />
  6. 6. Who is doing what and when<br />All about Actors, Actions, Objects and Targets<br /> standard vs. OpenSocial<br />Commentable<br />
  7. 7. Authentication<br />Performance<br />ActivityStrems<br />Integration<br />Flexibility<br />Testing<br />
  8. 8. People are tired of creating accounts on every site<br />Need to support existing login method if the platform caters to an existing audience<br />Existing auth may not work well with Open API initiatives<br />Open API and Oauth<br />2 legged: Service to Service<br />3 legged: User to App to Service<br />
  9. 9. Identify the bottlenecks <br />Measure everything<br />Use CDNs for all static content<br />Front end optimization via async loading<br />Database optimization via indexes, sharding<br />Caching <br />Scaling the sorts<br />Scaling up vs. Scaling out<br />CAP theorem<br />Relational vs. NOSQL storage<br />Read vs. Write heaviness<br />
  10. 10. Query vs. Propagation<br />Queries are read heavy<br />Propagation is write heavy<br />Deletion is a pain with propagation<br />Activity Aggregation<br />Aggregation on actor vs. object<br />Normalized vs. Denormalized storage<br />Comments<br />Decorating the activities on each request<br />
  11. 11. Integration with legacy touchpoints<br />Opening up the API<br />More channels like Mobile<br />More independent applications<br />Rate limiting and access control<br />Don’t forget existing data<br />Data outlives code<br />
  12. 12. Flexibility in the code to adapt changing requirements quickly and seamlessly<br />Good design<br />DRY SOCs<br />Flexibility in the infrastructure to adapt changing traffic and behavior<br />Virtualization<br />Heavy replication<br />Flexibility in the team to respond to changes<br />Process <br />
  13. 13. Automated Testing wherever possible<br />Developer Focus on test coverage (80+%)<br />Continuous Integration and Deployment<br />Cucumber + Hudson<br />Cross browser testing (yes, including IE)<br />
  14. 14.
  15. 15. Java services<br />Tomcat with Shindig 1.1, 4 nodes<br />REST/JSON<br />Ruby <br />Rails Admin App for moderation and points/levels<br />Migration Scripts<br />Twitter bot for routing #myign tweets to the platform<br />Misc. scripts to invalidate memache keys and test service endpoints<br />
  16. 16. Memcached<br />Extremely trivial to set up and maintain<br />Almost never dies<br />Massive scale out<br />Careful with<br />Cache hotspots<br />Concurrent writes<br />On the fly scale-out<br />Key/Value size limits<br />
  17. 17. MySQL<br />Proven, cheap to develop and operate<br />Maslow’s hammer<br />Easy scale out<br />Hard to store (and retrieve) network graphs<br />Write scaling with single master<br />Not the best choice for activitystreams<br />Schema changes lock the table(s)<br />
  18. 18. Awesome write scaling<br />Great for activity propagation model<br />In place updates<br />Using $push and $set<br />Excellent for storing social relationships as documents<br />Very easy to cluster<br />We are running replica pairs, plan to move to replica sets<br />Schema-less<br />No need to run alter scripts on 18M-row table<br />
  19. 19. Queryable<br />Rich Query language ($in, $size, $exists, $slice)<br />MapReduce for heavy data crunching<br />Supports Indexing<br />You can even index collections inside a document<br />Storage <br />~4x storage compared to relational data<br />Emerging technology<br />Index defragmentation <br />$or and indexing (to be supported in 1.7)<br />Load balancing support in the driver (coming soon)<br />
  20. 20. RabbitMQ for messaging<br />Ease of clustering<br />Written in Erlang for high performance and availability<br />Used for<br />Propagation of activities<br />Sending out email alerts<br />Indexing data in Solr<br />
  21. 21. Person<br />GET @self, @friends, @followers, @all, PUT/POST @self, @friends<br />Activities<br />GET @global, @self, @friends, POST @self <br />MediaItems<br />GET @self, @all and POST @self<br />AppData<br />For applications to store/retrieve data as key-value pairs GET/POST @self<br />Status <br />GET @friends, @self, @followers , POST @self<br />
  22. 22. Must have for any Java/Ruby webapp<br />Monitoring and troubleshooting<br />Save a ton of $ and time by efficient root cause analysis tools<br />Agents for Ruby and Java<br />IGN Engineers helped write PHP and Memcached agents<br />
  23. 23. Social Applications and community<br />Check the pulse of the community<br />UserVoice (<br />Less is more<br />Distinguish yourself and focus on your niche<br />Be Agile - Release early, release often<br />Do not shock your audience<br />Announce the changes/features on a blog<br />Eat your own dog food<br /><br />
  24. 24. Released July 2010 as beta<br />Daily API requests ~25M<br />Daily page views ~30K<br />Daily Uniques ~12K<br />6ms response times<br />Expected traffic 8-10x with more integration and mobile platform<br />
  25. 25. Manish Pandit<br />Engineering Manager, Social Platform at IGN<br />Email:<br />Twitter: @lobster1234<br />LinkedIn:<br />Blog:<br />MyIGN:<br />
  26. 26.<br /><br /><br /><br />