Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)

Rapidly Developing FarmVille
How we built and scaled a #1 Facebook game in 5 weeks
AmittMahajan
Lead Developer – FarmVille
March 9, 2010

FarmVille

5 week development cycle
FarmVille
6 developers (PHP/Flash)
2 artists
3 producers/designers

At Launch
FarmVille
18K users/day after 24 hrs
1M users/day after 4 days

Today
FarmVille
110M+ Installs
31M Players/Day

My Goals
Reduce the amount of time it takes you to develop your games
Give you a blueprint for how to reliably scale your games to millions without your servers falling over

What slows down developers?
Other developers
Design / Art
Production (Copy)
Lack of knowledge

Increasing Developer Efficiency
Flash (Client)
PHP (Server)
All developers know both
Design DOESN’T rule all
Developers co-own features with designers

Data Driven Design
Shoot for a content pipeline that doesn’t need a developer
Data driven can be as easy as a designer-editable XML file
No devneeded!

String Tables
A string table is an external file that holds strings for the app
Best practice to prepare for localization
Developers are not blocked by production
Allows quick response to Facebook TOS changes

Abstracted Network Layer
Feature developers get client/server communication and serialization for free
Goal

Abstracted Network Layer
PHP
Flash
Feature A
Feature B
Feature A Service
Feature B Service
Networked Actions
Action Dispatcher
Network Layer
Network Layer
Validation Checks
Validation Checks
AMF Protocol (RPC)
Action Batching
Result Batching

Network Layer Batching
No need to do the work twice
Reduced server load in FarmVille by 50%!
Server
Client
Only One Request
onPlow()
Plow
Plow
onPlow()
Plant
Plant
onPlant()
Plow
…

Network Layer Validation
Solves problem of unfiltered input
Ensures data is received by server in-order it was sent

Social Network Wrapper
Single place to perform Facebook API calls
Working with a highly dynamic API can be difficult
Abstracting FB calls makes them easier to integrate
Allows for cross-platform games

Continuous Deployment
Build latest version of source-repo
Deploy to test “auto app” on Facebook to surface production issues early

Continuous Deployment
FarmVille QA Process
Auto app
Staging
Production
Quick
Smoke
Full Test
Pass

FarmVille Traffic Growth
1 Million DAUsevery week for 20 weeks
source: developeranalytics.com

Scalable Server Architecture
We had no choice but to scale on the cloud
Every part of the server architecture scales horizontally
No single points of failure
Take the DB out of the equation

Round Robin DNS
…
Load Balancer #1
Load Balancer #N
Auto-scaling web array (PHP)
…
Web Server #M
Web Server #1
User Data
& Updates
Memcache Pool
Lazy Writes
DB Layer
Data
Reads
…
DB-#K
Master
DB-#1
Master
…
DB-#K
Slave
DB-#1
Slave

All you need to implement is…
…
Web Server #M
Web Server #1
…everything else is off-the-shelf components!
…

Reducing Load-times
Users want responsive pages
Show something immediately
Player’s won’t wait for your app to load
Stream non-critical content

Avoid Remote Calls
Remote server calls are slow and unreliable
Aim for no remote calls during load
Embed data into Javascript
Facebook
Iframe
Inline-JS here
Flash ClientEmbed

Caching Slow Calls
Build in FacebookAPI caching within the social network wrapper
Write a DB wrapper to generate and cache SQL

FarmVille Page Profiler
Catch and eliminate all un-cached Facebook and DB calls
Goal

Fault Tolerance
Servers do go down randomly
Build redundancy on all levels of server architecture
Facebook is a dependency also
Aim for DB-less & Facebook-less modes
“Defcon”-style error management

Runtime Config
How do we progressively keep the game running?
Create all features with kill-switches
Create a web dashboard to allow non-technical folks to help out

Is it still running?
Notify your team when things break using server monitoring (Nagios/Munin)
What to watch
Server load/traffic graphs
Memcacheevictions

Stats Tracking
How do we know what users are seeing?
Have the client send back statistics
Number of errors
Load-times
Take a metrics driven approach to error handling

Before Launching
Get confidence that your stuff actually works
Perform load testing
Social games are a marathon not a sprint
Sleep before launching!

Questions?

Thank You!

http://www.amitt.com	3685
http://sarveshspn.blogspot.in	53
http://sarveshspn.blogspot.com	21
http://feeds.feedburner.com	18
http://michaelbatistich.com	13
http://www.linkedin.com	7
http://sarveshspn.blogspot.ca	4
http://translate.googleusercontent.com	2
http://webcache.googleusercontent.com	2
http://michaelbatistich.tumblr.com	2
http://sarveshspn.blogspot.com.br	2
http://sarveshspn.blogspot.kr	2
http://posterous.com	2
http://www.google.fr	1
http://sarveshspn.blogspot.it	1

Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)

by Amitt Mahajan on Mar 27, 2011

Accessibility

Categories

Upload Details

Usage Rights

15 Embeds 3,815

Statistics

Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010) Presentation Transcript