Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood

Components of Social Platform
Challenges
Technology
API
Engineering Process

User Profile
Relationships
Activity Streams

Basic user information
Extended information based on the context
Player Cards
Bragging rights
Points, levels
Achievements, badges
Activities

Friending
Unidirectional (the Twitter model)
Bidirectional (the Facebok model)
Association
Comments
Ratings, Thumbs
Bookmarking/favoriting
Recommendations

Who is doing what and when
All about Actors, Actions, Objects and Targets
Activitystrea.ms standard vs. OpenSocial
Commentable

Authentication
Performance
ActivityStrems
Integration
Flexibility
Testing

People are tired of creating accounts on every site
Need to support existing login method if the platform caters to an existing audience
Existing auth may not work well with Open API initiatives
Open API and Oauth
2 legged: Service to Service
3 legged: User to App to Service

Identify the bottlenecks
Measure everything
Use CDNs for all static content
Front end optimization via async loading
Database optimization via indexes, sharding
Caching
Scaling the sorts
Scaling up vs. Scaling out
CAP theorem
Relational vs. NOSQL storage
Read vs. Write heaviness

Query vs. Propagation
Queries are read heavy
Propagation is write heavy
Deletion is a pain with propagation
Activity Aggregation
Aggregation on actor vs. object
Normalized vs. Denormalized storage
Comments
Decorating the activities on each request

Integration with legacy touchpoints
Opening up the API
More channels like Mobile
More independent applications
Rate limiting and access control
Don’t forget existing data
Data outlives code

Flexibility in the code to adapt changing requirements quickly and seamlessly
Good design
DRY SOCs
Flexibility in the infrastructure to adapt changing traffic and behavior
Virtualization
Heavy replication
Flexibility in the team to respond to changes
Process

Automated Testing wherever possible
Developer Focus on test coverage (80+%)
Continuous Integration and Deployment
Cucumber + Hudson
Cross browser testing (yes, including IE)

Java services
Tomcat with Shindig 1.1, 4 nodes
REST/JSON
Ruby
Rails Admin App for moderation and points/levels
Migration Scripts
Twitter bot for routing #myign tweets to the platform
Misc. scripts to invalidate memache keys and test service endpoints

Memcached
Extremely trivial to set up and maintain
Almost never dies
Massive scale out
Careful with
Cache hotspots
Concurrent writes
On the fly scale-out
Key/Value size limits

MySQL
Proven, cheap to develop and operate
Maslow’s hammer
Easy scale out
Hard to store (and retrieve) network graphs
Write scaling with single master
Not the best choice for activitystreams
Schema changes lock the table(s)

Awesome write scaling
Great for activity propagation model
In place updates
Using $push and $set
Excellent for storing social relationships as documents
Very easy to cluster
We are running replica pairs, plan to move to replica sets
Schema-less
No need to run alter scripts on 18M-row table

Queryable
Rich Query language ($in, $size, $exists, $slice)
MapReduce for heavy data crunching
Supports Indexing
You can even index collections inside a document
Storage
~4x storage compared to relational data
Emerging technology
Index defragmentation
$or and indexing (to be supported in 1.7)
Load balancing support in the driver (coming soon)

RabbitMQ for messaging
Ease of clustering
Written in Erlang for high performance and availability
Used for
Propagation of activities
Sending out email alerts
Indexing data in Solr

Person
GET @self, @friends, @followers, @all, PUT/POST @self, @friends
Activities
GET @global, @self, @friends, POST @self
MediaItems
GET @self, @all and POST @self
AppData
For applications to store/retrieve data as key-value pairs GET/POST @self
Status
GET @friends, @self, @followers , POST @self

Must have for any Java/Ruby webapp
Monitoring and troubleshooting
Save a ton of $ and time by efficient root cause analysis tools
Agents for Ruby and Java
IGN Engineers helped write PHP and Memcached agents

Social Applications and community
Check the pulse of the community
UserVoice (http://ign.uservoice.com)
Less is more
Distinguish yourself and focus on your niche
Be Agile - Release early, release often
Do not shock your audience
Announce the changes/features on a blog
Eat your own dog food
http://people.ign.com/ign-labs

Released July 2010 as beta
Daily API requests ~25M
Daily page views ~30K
Daily Uniques ~12K
6ms response times
Expected traffic 8-10x with more integration and mobile platform

Manish Pandit
Engineering Manager, Social Platform at IGN
Email: pandit.manish-at-gmail.com
Twitter: @lobster1234
LinkedIn: http://www.linkedin.com/in/mpandit
Blog: http://contrarianwisdom.blogspot.com
MyIGN: http://people.ign.com/mpanditign

http://corp.ign.com
http://labs.ign.com
http://my.ign.com
http://people.ign.com/ign-labs

http://contrarianwisdom.blogspot.com	381
http://contrarianwisdom.blogspot.in	18
http://coderwall.com	12
http://contrarianwisdom.blogspot.co.uk	5
http://contrarianwisdom.blogspot.ca	3
http://contrarianwisdom.blogspot.fr	3
http://contrarianwisdom.blogspot.pt	2
http://contrarianwisdom.blogspot.jp	2
http://contrarianwisdom.blogspot.com.au	2
http://webcache.googleusercontent.com	1
http://contrarianwisdom.blogspot.de	1
http://contrarianwisdom.blogspot.com.ar	1
http://contrarianwisdom.blogspot.com.br	1

Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood

by Manish Pandit on Oct 09, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 432

Statistics

Silicon Valley Code Camp 2010: Social Platforms : What goes on under the hood — Presentation Transcript