Your SlideShare is downloading. ×
Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood

2,269
views

Published on

In this session I'd share the design, architecture and implementation of some of the most common elements of any social platform - Open API, profiles, searches, lists and activity streams. These …

In this session I'd share the design, architecture and implementation of some of the most common elements of any social platform - Open API, profiles, searches, lists and activity streams. These "pillers" of a social platform bear most of the weight behind a jazzy UI, and scaling them has its own challenges. I will also talk about how we built the Social Platform at IGN from ground up, including not-so-unique challenges like integration with legacy systems.

Published in: Technology

1 Comment
0 Likes
Statistics
Notes
  • http://www.sendspace.com/file/8kn03w
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total Views
2,269
On Slideshare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
38
Comments
1
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1.
  • 2. Components of Social Platform
    Challenges
    Technology
    API
    Engineering Process
  • 3. User Profile
    Relationships
    Activity Streams
  • 4. Basic user information
    Extended information based on the context
    Player Cards
    Bragging rights
    Points, levels
    Achievements, badges
    Activities
  • 5. Friending
    Unidirectional (the Twitter model)
    Bidirectional (the Facebok model)
    Association
    Comments
    Ratings, Thumbs
    Bookmarking/favoriting
    Recommendations
  • 6. Who is doing what and when
    All about Actors, Actions, Objects and Targets
    Activitystrea.ms standard vs. OpenSocial
    Commentable
  • 7. Authentication
    Performance
    ActivityStrems
    Integration
    Flexibility
    Testing
  • 8. People are tired of creating accounts on every site
    Need to support existing login method if the platform caters to an existing audience
    Existing auth may not work well with Open API initiatives
    Open API and Oauth
    2 legged: Service to Service
    3 legged: User to App to Service
  • 9. Identify the bottlenecks
    Measure everything
    Use CDNs for all static content
    Front end optimization via async loading
    Database optimization via indexes, sharding
    Caching
    Scaling the sorts
    Scaling up vs. Scaling out
    CAP theorem
    Relational vs. NOSQL storage
    Read vs. Write heaviness
  • 10. Query vs. Propagation
    Queries are read heavy
    Propagation is write heavy
    Deletion is a pain with propagation
    Activity Aggregation
    Aggregation on actor vs. object
    Normalized vs. Denormalized storage
    Comments
    Decorating the activities on each request
  • 11. Integration with legacy touchpoints
    Opening up the API
    More channels like Mobile
    More independent applications
    Rate limiting and access control
    Don’t forget existing data
    Data outlives code
  • 12. Flexibility in the code to adapt changing requirements quickly and seamlessly
    Good design
    DRY SOCs
    Flexibility in the infrastructure to adapt changing traffic and behavior
    Virtualization
    Heavy replication
    Flexibility in the team to respond to changes
    Process
  • 13. Automated Testing wherever possible
    Developer Focus on test coverage (80+%)
    Continuous Integration and Deployment
    Cucumber + Hudson
    Cross browser testing (yes, including IE)
  • 14.
  • 15. Java services
    Tomcat with Shindig 1.1, 4 nodes
    REST/JSON
    Ruby
    Rails Admin App for moderation and points/levels
    Migration Scripts
    Twitter bot for routing #myign tweets to the platform
    Misc. scripts to invalidate memache keys and test service endpoints
  • 16. Memcached
    Extremely trivial to set up and maintain
    Almost never dies
    Massive scale out
    Careful with
    Cache hotspots
    Concurrent writes
    On the fly scale-out
    Key/Value size limits
  • 17. MySQL
    Proven, cheap to develop and operate
    Maslow’s hammer
    Easy scale out
    Hard to store (and retrieve) network graphs
    Write scaling with single master
    Not the best choice for activitystreams
    Schema changes lock the table(s)
  • 18. Awesome write scaling
    Great for activity propagation model
    In place updates
    Using $push and $set
    Excellent for storing social relationships as documents
    Very easy to cluster
    We are running replica pairs, plan to move to replica sets
    Schema-less
    No need to run alter scripts on 18M-row table
  • 19. Queryable
    Rich Query language ($in, $size, $exists, $slice)
    MapReduce for heavy data crunching
    Supports Indexing
    You can even index collections inside a document
    Storage
    ~4x storage compared to relational data
    Emerging technology
    Index defragmentation
    $or and indexing (to be supported in 1.7)
    Load balancing support in the driver (coming soon)
  • 20. RabbitMQ for messaging
    Ease of clustering
    Written in Erlang for high performance and availability
    Used for
    Propagation of activities
    Sending out email alerts
    Indexing data in Solr
  • 21. Person
    GET @self, @friends, @followers, @all, PUT/POST @self, @friends
    Activities
    GET @global, @self, @friends, POST @self
    MediaItems
    GET @self, @all and POST @self
    AppData
    For applications to store/retrieve data as key-value pairs GET/POST @self
    Status
    GET @friends, @self, @followers , POST @self
  • 22. Must have for any Java/Ruby webapp
    Monitoring and troubleshooting
    Save a ton of $ and time by efficient root cause analysis tools
    Agents for Ruby and Java
    IGN Engineers helped write PHP and Memcached agents
  • 23. Social Applications and community
    Check the pulse of the community
    UserVoice (http://ign.uservoice.com)
    Less is more
    Distinguish yourself and focus on your niche
    Be Agile - Release early, release often
    Do not shock your audience
    Announce the changes/features on a blog
    Eat your own dog food
    http://people.ign.com/ign-labs
  • 24. Released July 2010 as beta
    Daily API requests ~25M
    Daily page views ~30K
    Daily Uniques ~12K
    6ms response times
    Expected traffic 8-10x with more integration and mobile platform
  • 25. Manish Pandit
    Engineering Manager, Social Platform at IGN
    Email: pandit.manish-at-gmail.com
    Twitter: @lobster1234
    LinkedIn: http://www.linkedin.com/in/mpandit
    Blog: http://contrarianwisdom.blogspot.com
    MyIGN: http://people.ign.com/mpanditign
  • 26. http://corp.ign.com
    http://labs.ign.com
    http://my.ign.com
    http://people.ign.com/ign-labs