Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)

  • 11,226 views
Uploaded on

Amitt Mahajan discusses the techniques used in the development of the hit social game FarmVille. This presentation covers the processes used by the FarmVille development team to quickly release the …

Amitt Mahajan discusses the techniques used in the development of the hit social game FarmVille. This presentation covers the processes used by the FarmVille development team to quickly release the game and then scale it to support millions of daily players.

(Originally presented at GDC 2010)

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
11,226
On Slideshare
0
From Embeds
0
Number of Embeds
7

Actions

Shares
Downloads
181
Comments
0
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Rapidly Developing FarmVille
    How we built and scaled a #1 Facebook game in 5 weeks
    AmittMahajan
    Lead Developer – FarmVille
    March 9, 2010
  • 2. FarmVille
  • 3. 5 week development cycle
    FarmVille
    6 developers (PHP/Flash)
    2 artists
    3 producers/designers
  • 4. At Launch
    FarmVille
    18K users/day after 24 hrs
    1M users/day after 4 days
  • 5. Today
    FarmVille
    110M+ Installs
    31M Players/Day
  • 6. My Goals
    Reduce the amount of time it takes you to develop your games
    Give you a blueprint for how to reliably scale your games to millions without your servers falling over
  • 7. What slows down developers?
    Other developers
    Design / Art
    Production (Copy)
    Lack of knowledge
  • 8. Increasing Developer Efficiency
    Flash (Client)
    PHP (Server)
    All developers know both
    Design DOESN’T rule all
    Developers co-own features with designers
  • 9. Data Driven Design
    Shoot for a content pipeline that doesn’t need a developer
    Data driven can be as easy as a designer-editable XML file
    No devneeded!
  • 10. String Tables
    A string table is an external file that holds strings for the app
    Best practice to prepare for localization
    Developers are not blocked by production
    Allows quick response to Facebook TOS changes
  • 11. Abstracted Network Layer
    Feature developers get client/server communication and serialization for free
    Goal
  • 12. Abstracted Network Layer
    PHP
    Flash
    Feature A
    Feature B
    Feature A Service
    Feature B Service
    Networked Actions
    Action Dispatcher
    Network Layer
    Network Layer
    Validation Checks
    Validation Checks
    AMF Protocol (RPC)
    Action Batching
    Result Batching
  • 13. Network Layer Batching
    No need to do the work twice
    Reduced server load in FarmVille by 50%!
    Server
    Client
    Only One Request
    onPlow()
    Plow
    Plow
    onPlow()
    Plant
    Plant
    onPlant()
    Plow

  • 14. Network Layer Validation
    Solves problem of unfiltered input
    Ensures data is received by server in-order it was sent
  • 15. Social Network Wrapper
    Single place to perform Facebook API calls
    Working with a highly dynamic API can be difficult
    Abstracting FB calls makes them easier to integrate
    Allows for cross-platform games
  • 16. Continuous Deployment
    Build latest version of source-repo
    Deploy to test “auto app” on Facebook to surface production issues early
  • 17. Continuous Deployment
    FarmVille QA Process
    Auto app
    Staging
    Production
    Quick
    Smoke
    Full Test
    Pass
  • 18. FarmVille Traffic Growth
    1 Million DAUsevery week for 20 weeks
    source: developeranalytics.com
  • 19. Scalable Server Architecture
    We had no choice but to scale on the cloud
    Every part of the server architecture scales horizontally
    No single points of failure
    Take the DB out of the equation
  • 20. Round Robin DNS

    Load Balancer #1
    Load Balancer #N
    Auto-scaling web array (PHP)

    Web Server #M
    Web Server #1
    User Data
    & Updates
    Memcache Pool
    Lazy Writes
    DB Layer
    Data
    Reads

    DB-#K
    Master
    DB-#1
    Master

    DB-#K
    Slave
    DB-#1
    Slave
  • 21. All you need to implement is…

    Web Server #M
    Web Server #1
    …everything else is off-the-shelf components!

  • 22. Reducing Load-times
    Users want responsive pages
    Show something immediately
    Player’s won’t wait for your app to load
    Stream non-critical content
  • 23. Avoid Remote Calls
    Remote server calls are slow and unreliable
    Aim for no remote calls during load
    Embed data into Javascript
    Facebook
    Iframe
    Inline-JS here
    Flash ClientEmbed
  • 24. Caching Slow Calls
    Build in FacebookAPI caching within the social network wrapper
    Write a DB wrapper to generate and cache SQL
  • 25. FarmVille Page Profiler
    Catch and eliminate all un-cached Facebook and DB calls
    Goal
  • 26. Fault Tolerance
    Servers do go down randomly
    Build redundancy on all levels of server architecture
    Facebook is a dependency also
    Aim for DB-less & Facebook-less modes
    “Defcon”-style error management
  • 27. Runtime Config
    How do we progressively keep the game running?
    Create all features with kill-switches
    Create a web dashboard to allow non-technical folks to help out
  • 28. Is it still running?
    Notify your team when things break using server monitoring (Nagios/Munin)
    What to watch
    Server load/traffic graphs
    Memcacheevictions
  • 29. Stats Tracking
    How do we know what users are seeing?
    Have the client send back statistics
    Number of errors
    Load-times
    Take a metrics driven approach to error handling
  • 30. Before Launching
    Get confidence that your stuff actually works
    Perform load testing
    Social games are a marathon not a sprint
    Sleep before launching!
  • 31. Questions?
  • 32. Thank You!