Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood

2,479 views
2,396 views

Published on

In this session I'd share the design, architecture and implementation of some of the most common elements of any social platform - Open API, profiles, searches, lists and activity streams. These "pillers" of a social platform bear most of the weight behind a jazzy UI, and scaling them has its own challenges. I will also talk about how we built the Social Platform at IGN from ground up, including not-so-unique challenges like integration with legacy systems.

Published in: Technology
1 Comment
0 Likes
Statistics
Notes
  • http://www.sendspace.com/file/8kn03w
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
2,479
On SlideShare
0
From Embeds
0
Number of Embeds
674
Actions
Shares
0
Downloads
39
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood

  1. 1.
  2. 2. Components of Social Platform<br />Challenges<br />Technology<br />API<br />Engineering Process<br />
  3. 3. User Profile<br />Relationships<br />Activity Streams<br />
  4. 4. Basic user information<br />Extended information based on the context<br />Player Cards<br />Bragging rights <br />Points, levels<br />Achievements, badges<br />Activities<br />
  5. 5. Friending<br />Unidirectional (the Twitter model)<br />Bidirectional (the Facebok model) <br />Association<br />Comments<br />Ratings, Thumbs<br />Bookmarking/favoriting<br />Recommendations<br />
  6. 6. Who is doing what and when<br />All about Actors, Actions, Objects and Targets<br />Activitystrea.ms standard vs. OpenSocial<br />Commentable<br />
  7. 7. Authentication<br />Performance<br />ActivityStrems<br />Integration<br />Flexibility<br />Testing<br />
  8. 8. People are tired of creating accounts on every site<br />Need to support existing login method if the platform caters to an existing audience<br />Existing auth may not work well with Open API initiatives<br />Open API and Oauth<br />2 legged: Service to Service<br />3 legged: User to App to Service<br />
  9. 9. Identify the bottlenecks <br />Measure everything<br />Use CDNs for all static content<br />Front end optimization via async loading<br />Database optimization via indexes, sharding<br />Caching <br />Scaling the sorts<br />Scaling up vs. Scaling out<br />CAP theorem<br />Relational vs. NOSQL storage<br />Read vs. Write heaviness<br />
  10. 10. Query vs. Propagation<br />Queries are read heavy<br />Propagation is write heavy<br />Deletion is a pain with propagation<br />Activity Aggregation<br />Aggregation on actor vs. object<br />Normalized vs. Denormalized storage<br />Comments<br />Decorating the activities on each request<br />
  11. 11. Integration with legacy touchpoints<br />Opening up the API<br />More channels like Mobile<br />More independent applications<br />Rate limiting and access control<br />Don’t forget existing data<br />Data outlives code<br />
  12. 12. Flexibility in the code to adapt changing requirements quickly and seamlessly<br />Good design<br />DRY SOCs<br />Flexibility in the infrastructure to adapt changing traffic and behavior<br />Virtualization<br />Heavy replication<br />Flexibility in the team to respond to changes<br />Process <br />
  13. 13. Automated Testing wherever possible<br />Developer Focus on test coverage (80+%)<br />Continuous Integration and Deployment<br />Cucumber + Hudson<br />Cross browser testing (yes, including IE)<br />
  14. 14.
  15. 15. Java services<br />Tomcat with Shindig 1.1, 4 nodes<br />REST/JSON<br />Ruby <br />Rails Admin App for moderation and points/levels<br />Migration Scripts<br />Twitter bot for routing #myign tweets to the platform<br />Misc. scripts to invalidate memache keys and test service endpoints<br />
  16. 16. Memcached<br />Extremely trivial to set up and maintain<br />Almost never dies<br />Massive scale out<br />Careful with<br />Cache hotspots<br />Concurrent writes<br />On the fly scale-out<br />Key/Value size limits<br />
  17. 17. MySQL<br />Proven, cheap to develop and operate<br />Maslow’s hammer<br />Easy scale out<br />Hard to store (and retrieve) network graphs<br />Write scaling with single master<br />Not the best choice for activitystreams<br />Schema changes lock the table(s)<br />
  18. 18. Awesome write scaling<br />Great for activity propagation model<br />In place updates<br />Using $push and $set<br />Excellent for storing social relationships as documents<br />Very easy to cluster<br />We are running replica pairs, plan to move to replica sets<br />Schema-less<br />No need to run alter scripts on 18M-row table<br />
  19. 19. Queryable<br />Rich Query language ($in, $size, $exists, $slice)<br />MapReduce for heavy data crunching<br />Supports Indexing<br />You can even index collections inside a document<br />Storage <br />~4x storage compared to relational data<br />Emerging technology<br />Index defragmentation <br />$or and indexing (to be supported in 1.7)<br />Load balancing support in the driver (coming soon)<br />
  20. 20. RabbitMQ for messaging<br />Ease of clustering<br />Written in Erlang for high performance and availability<br />Used for<br />Propagation of activities<br />Sending out email alerts<br />Indexing data in Solr<br />
  21. 21. Person<br />GET @self, @friends, @followers, @all, PUT/POST @self, @friends<br />Activities<br />GET @global, @self, @friends, POST @self <br />MediaItems<br />GET @self, @all and POST @self<br />AppData<br />For applications to store/retrieve data as key-value pairs GET/POST @self<br />Status <br />GET @friends, @self, @followers , POST @self<br />
  22. 22. Must have for any Java/Ruby webapp<br />Monitoring and troubleshooting<br />Save a ton of $ and time by efficient root cause analysis tools<br />Agents for Ruby and Java<br />IGN Engineers helped write PHP and Memcached agents<br />
  23. 23. Social Applications and community<br />Check the pulse of the community<br />UserVoice (http://ign.uservoice.com)<br />Less is more<br />Distinguish yourself and focus on your niche<br />Be Agile - Release early, release often<br />Do not shock your audience<br />Announce the changes/features on a blog<br />Eat your own dog food<br />http://people.ign.com/ign-labs<br />
  24. 24. Released July 2010 as beta<br />Daily API requests ~25M<br />Daily page views ~30K<br />Daily Uniques ~12K<br />6ms response times<br />Expected traffic 8-10x with more integration and mobile platform<br />
  25. 25. Manish Pandit<br />Engineering Manager, Social Platform at IGN<br />Email: pandit.manish-at-gmail.com<br />Twitter: @lobster1234<br />LinkedIn: http://www.linkedin.com/in/mpandit<br />Blog: http://contrarianwisdom.blogspot.com<br />MyIGN: http://people.ign.com/mpanditign<br />
  26. 26. http://corp.ign.com<br />http://labs.ign.com<br />http://my.ign.com<br />http://people.ign.com/ign-labs<br />

×