This is the presentation I gave at the International Informix Users (IIUG) group in 2011. The presentation gives an introduction and overview of the architectures used in interactive entertainment (Online Games, social, MMO). I regret having to remove discussions covering video production, and video delivery, as well as CDN. Time did not allow, and those topics were not of as much interest to the audience.

1. Architectures in Entertainment
George Dolbier Session: Z99
IBM
Day/Date/Time

2. Today’s Agenda
IBM, Games? Really? And who is this guy?
Interactive Entertainment Industry
Architectural patterns for Scalable Entertainment
Internals Massively Multiplayer Online Games

3. Who is IBM again?
And who is this guy?

4. IBM is embedded the gaming industry
starting with the CPUs in every game console
Souce www.nexgenwars.com 2/27/2011

5. To the largest Online Infrastructures:
EvE Online
Active Servers
Weight
CPUS
RAM
Estimated
FLOPS
~ 195
About 2.5 Metric Tonnes
> 420 CPU cores > 1 THZ
> 7.5 Terabytes
> 7.5 Terra
Total Bandwidth Estimate ~ 400 Gbs

6. George Dolbier, IBM CTO Interactive Media
20+ year Technological innovator and Business Developer successfully pushing the limits of possibilities daily. A classically
trained computer scientist with product production experience ranging from embedded software development to database
kernel engineer at Informix and Oracle and operating system kernel engineer at Sequent. Management and business
experience ranges from IT operation and tactical management of small task oriented teams to executive level positions
managing large geographically dispersed groups.
Relevant Experience
Architecture/design/development/deployment/operations of 30+ Interactive Media projects including social netowrks, streaming
video, Online Games, Cinema in 15+ countries
20+ years in SW Technology, 15 years in Interactive Media
Architect and software designer with deep experience in producing software products and web scale services in the, social media,
MMO, Film and connected console game
Successful projects in 15+ countries in the 3 major Geographies of Asia North America and Europe
Developer/Publisher relations include:
CCP, Lucas, EA, Unity, Epic, Sony, Norsk Tipping, Obsidian, Zynga, Monumental, Cartoon Network, Hasbro
Technology Reltationships include
BigWorld, Unity, Instant Action (FKA Garage Games) Vivox, Sleepy Giant, Parature, Epic
Executive experience
Manage, drive and mentor teams up to up to 1500
Manage geographically dispersed teams
Produce consistent sustainable double digit Y2Y profit and revenue growth
Led creation of 10 million dollar Joint venture between IBM, Intel and Cisco
Led creation of relationships between IBM and Games industry providers (BigWorld,Vivox,GNi)
Drove business unit to build global revenue from 0 to 180+ million in under 4 years
Technical Delivery experience
Deployed 6000 servers infrastructure in china to support 3 million players in under 6 weeks
Doubled Epic’s production environment allowing project to be developed on time on budget
Implemented production environment at ILM Used on StarTrek,TransFormers2, Harry Potter
Architected & Implemented HPC Cluster that discovered genetic cause of bone marrow cancer
Education
Key Skills
• IBM Executive
Architect Certified
• Open Group Certified
Master Architect
• Portland State
University – BS
Computer Science
• Online Technology
Architecture
• Online Operations
• Global Rollout
• Social Media Architecture
• IM Production
• DFX Production
• Follow the Sun dev.
mgmt.
• Strategy & Operational
• Production Optimization
• Technical Capacity
Analysis
• OT&B Delivery

7. Industry Introduction
Convergent Divergencies
A little History
It’s How big?

8. Interactive Entertainment is Convergence
Online
Technical
• Persistence
• Centralized Simulation
• Scale
• CC Pipeline
Business
• Software As a Service
• Re-accruing revenue
• Subscription
• Economic Systems/Item Sales
• Ongoing Operational Expense Interactivity
• Scale
• Autonomy
• Co-Ordination
• Co-Operation
• Arbitration
• Economics
• Communication
Community
• Communications
• In Game Support
• Access to the game from outside
Interactive Entertainment Is…
A scalable, persistent, form of entertainment
delivered as a service

9. It’s How Big?
• EvE Online: 340,000 Subscribers
• Bejeweled Blitz: 10 Million Monthly active Users
• World of Warcraft: 12 Million Monthly Subscribers
• Lineage II: 19 Million Players world wide
• FrontierVille: 19 Million Monthly Active Users
• Farmville: 85 Million Monthly active users
• Cityville: 100 Million monthly active users
http://www.eveonline.com/pressreleases/
http://us.blizzard.com/en-us/company/press/pressreleases.html?101007
http://pc.ign.com/articles/925/925099p1.html
http://www.appdata.com/leaderboard/apps (2/27/2011)

10. The Facebook Top 40
Top 40 FB Apps: Monthly Active
0
20,000,000
40,000,000
60,000,000
80,000,000
100,000,000
CityVille
Texas
Phrases
Conduit
QuizTaco!
Yahoo!
PetSociety
Yelp
MindJolt
FriendBuzz
Zoosk
Birthday
Games
MallWorld
MAU
Top 40 FB pps: MonthlyActive
(minus farmville & Cityville)
0
10,000,000
20,000,000
30,000,000
40,000,000
50,000,000
MAU
All Over:
2 Million MAU

11. Architectural patterns
for
Scalable Entertainment
Environment
Archetypes
Patterns

12. © 2008 IBM Corporation
Every Architecture exists in an Environment
SettingIDCStudio
InteractionOperationProductionCreation
Segment
Creation:
 Content is created
Production:
 Content is transformed
Operation:
 Content is distributed
and operated
Interaction:
 Content is interacted
with
Time
Concept Launch Flight
Technology deployment
timeline
Environments
Studio:
 Highly Collaborative Environment
where concepts are created and
transformed into products. This is
where the virtual world is created
IDC:
 Consolidated, efficient, secure
Environment where data and
communications come together.
This is where the virtual world runs
Setting:
 Home, Office, Mobile, Capability,
device, location, and connectivity
that allows a game or virtual world
to be interacted with
Environments
are differentiated by
economic model as well
as physical, technical and
Human factors
Disciplines
are differentiated by the goals
of individuals, and how those
individuals relate to content

13. © 2008 IBM Corporation
Entertainment Creation: The Studio
Segment
Creation:
 Content is created
Production:
 Content is transformed
Operation:
 Content is distributed
and operated
Interaction:
 Content is interacted
with
Time
Concept Launch Flight
Technology deployment
timeline
Environments
Studio:
Highly Collaborative Environment
where concepts are created and
transformed into products. This is
where the virtual world is created
IDC:
Consolidated, efficient, secure
Environment where data and
communications come together.
This is where the virtual world runs
Setting:
Home, Office, Mobile, Capability, devic
e, location, and connectivity that
allows a game or virtual world to
be interacted with
Environments
are differentiated by
economic model as well
as physical, technical and
Human factors
Disciplines
are differentiated by the goals
of individuals, and how those
individuals relate to content
Setting
Interaction
IDC
Operation
Studio
ProductionCreation

14. © 2008 IBM Corporation
Content Creation Architecture

15. © 2008 IBM Corporation
Entertainment as a Service: The Internet Datacenter
Segment
Creation:
 Content is created
Production:
 Content is transformed
Operation:
 Content is distributed
and operated
Interaction:
 Content is interacted
with
Time
Concept Launch Flight
Technology deployment
timeline
Environments
Studio:
Highly Collaborative Environment
where concepts are created and
transformed into products. This is
where the virtual world is created
IDC:
Consolidated, efficient, secure
Environment where data and
communications come together.
This is where the virtual world runs
Setting:
Home, Office, Mobile, Capability,
device, location, and connectivity
that allows a game or virtual world
to be interacted with
Environments
are differentiated by
economic model as well
as physical, technical and
Human factors
Disciplines
are differentiated by the goals
of individuals, and how those
individuals relate to content
Studio
ProductionCreation
Setting
Interaction
IDC
Operation
Some might call this the service deliver environment… (we shun them)

16. Single
Online Entertainment Archetypes
Hybrid
Sharded

17. Why one v.s. the other
Sharded
Content
Multiple Operations
Authored Content
Step Scale
Moderate latency
Universe
Content
Single Operations
User created Content
Scale Out
Latency Tolerant
Hybrid
Content
Mixed Operations
Instanced Content
Bi-Directional Scale
Low Latency

18. © 2008 IBM Corporation
Interactive Content Delivery
This is NOT streaming video….
This is NOT A transactional workload

19. Internals:
Massively Multiplayer Online Games
Architecture for millions of players

20. More than just the game engine

21. Massively Triumvirate
Economic SupportGame

22. Economic Services
Financial administration
Interface
Game Interface
User Financial Account
Database
Account Management
Website
Tax/VAT Tax
processing system
Support Administration
Interface
FireWall
DOS/DDOS
Account Transaction
Database
Decision Support
Database
Fraud Detection/
Reporting System
Financial Transaction
processing
Payment Processor
Gateways

23. Support Services
FireWall
CSR Interface
Support Interface to
Financial System
In Game Support
Interface
Problem
Management database
Decision Support
Database
External Support
Website
Support
Ticketing
System
CRM
system
Support Knowledge
Base (Wiki)
Interface to bug
tracking system
Interface to game/
Systems & Network
Management
DOS/DDOS

24. Game Services
Support
Administration
Interface
Financial
administration
Interface
Player behavior
reporting system
Map/Object Data
Mnaagement and
comms log
Player Stats
Game Website
Simulation
Engine
FireWall
DOS/DDOS
Routing/load balencing
“Instance”
Manager
Authentication
System
Patch Server
CDN
Authentication
System
Txt Chat Server
Voice Chat Server
Simulation
Engine
Simulation
Engine
Simulation
Engine
“Instance”
Manager
XBLA/PSN
GW
Transaction management/
caching
Transaction management
caching
In Game Economic system
Console
Network
Game
Manager
Game
Manager
Game
Manager
Game
Manager
Simulation
Engine
Simulation
Engine
Simulation
Engine
Simulation
Engine
Txt Chat Log

25. CCP’s EVE online
Specific examples of the importance of DB
performance in online entertainment
These next slides where originally presented by
Derek Wyse, CCP VP Ops, at the EvE Fanfest 2011

26. Our Goals
• Keep up with load
• Redundancy at every level
• Simplification of design
• Options!

27. Redundancy

28. Simplification
• Devices
• Old TQ = 8 devices
• Proposed Design = 10 devices
• Final Design = 6 devices
• Entire system can go live as a unit...
No partial upgrade need
• System maintainable on all units live
(using the standby path)

29. OMGWTF DATA
• Step one: (Upgrade the Storage Network)
– WIN: 8Gb/s to 32Gb/s and no sharing ;)
• Step two: (Upgrade the RAM)
– WIN: 128GB of DDR2 to 512GB of DDR3, 111GB of DB Cache to 460
DB Cache and it all takes our Page Life Expectancy (the time data
stays in RAM) from 7.9 mins to 2.8 hours! Also, the old TQ database
was at 97% Buffer Cache Hit Ratio while the New TQ database is at
99.7%.
• Step three: (Upgrade the CPU)
– WIN: 12 to 32 logical processors plus a bonus for the new architecture
gain over the old processor in our test and in lab tests. Plus,
seriously... that screen shot by its self makes the server 400% faster.
• Step four: (Upgrade the Storage)
– WIN: Transfer speed from 2.9Gb/s to 10Gb/s tested. Total storage
capacity on SSD's doubled. Aggregate storage on all production
capable tiers increased from 2TB non redundant to 11.5TB and 1 for 1
redundant hardware.

32. OMGWTF DATA
• Each server has:
– 2 x 8 Core Intel x7560 processors at 2.26GHz
– 32 x 16GB strips of RAM for a total of 512GB DDR3 RAM
– 4 x Gigabit Ethernet cards
– 2 x Dual-port 8Gb/s Fiberchannel cards
– 4 x 15K 300GB SAS drives internal
– 9 x 300 GB SSD's drives RAID 5 attached via IBM V7000
– 36 x 600 GB SAS drives RAID 10 attached via IBM V7000
• The whole system has:
– Over 1TB of RAM
– 64 logical processors (w/ hyper-threading enabled)
– 32Gb/s of storage throughput capacity
– 200,000 IOPS capacity
– 51TB of RAW storage capacity (23TB after RAID and spares)

33. OMGWTF DATA
• Backup
– Server Disk System Time Speed
– Old TQ DB Current (30) FC Disks 73 mins 2.8 Gb/s
– New TQ DB Current (30) FC Disks 60 mins 3.2 Gb/s
– New TQ DB New (34) SAS Disks 23 mins 8.5 Gb/s
• Restore
– Server Disk System Time
– Old TQ DB Current (30) FC Disks 180 mins
– New TQ DB Current (30) FC Disks 66 mins
– New TQ DB New (34) SAS Disks 59 mins

34. Internals:
Social Games
A couple orders of magnitude larger
How to support HUNDREDS of millions

35. Special Notes: Social Games
• Farmville had 1 Million daily players after 4 days and
10Million after 60 days.
• 75 million monthly users in 9 months (from 0 to >France)
• Most “games” are 80-90% Read, primarily non-
transactional workload
• Social Games heavy write load: Farmville 30%
• Social game load spikes can be > 50%
http://highscalability.com/blog/2010/2/8/how-farmville-scales-to-harvest-75-million-players-a-month.html

36. Architecture stack for Social Game
Data Persistance Layer
DataCenter
Sesson Mgmt
WebServices
DB Cache
Physical server
DB
DB
DB
Physical server
DB
DB
DB
Physical server
DB
DB
DB
DB Cache DB Cache DB Cache DB Cache
GameSerivces
PHP
Game Services
GameSerivces
PHP
GameSerivces
PHP
Web Layer
Sesson Mgmt
WebServices
Sesson Mgmt
WebServices
Cache
Cache Cache Cache Cache
Sesson Mgmt
WebServices
Physical server
DB
DB
DB
Comm Services
router
Load balencer
firewall
Load balencer Load balencer
router
firewall

37. Questions ?!?
4/27/2013Template Presentation - Session Z99 37

38. Thank You

39. Architectures in Entertainment
George Dolbier
georged@us.ibm.com