Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Architectures in EntertainmentGeorge Dolbier Session: Z99IBMDay/Date/Time
Today’s AgendaIBM, Games? Really? And who is this guy?Interactive Entertainment IndustryArchitectural patterns for Scalabl...
Who is IBM again?And who is this guy?
IBM is embedded the gaming industrystarting with the CPUs in every game consoleSouce www.nexgenwars.com 2/27/2011
To the largest Online Infrastructures:EvE OnlineActive ServersWeightCPUSRAMEstimatedFLOPS~ 195About 2.5 Metric Tonnes> 420...
George Dolbier, IBM CTO Interactive Media20+ year Technological innovator and Business Developer successfully pushing the ...
Industry IntroductionConvergent DivergenciesA little HistoryIt’s How big?
Interactive Entertainment is ConvergenceOnlineTechnical• Persistence• Centralized Simulation• Scale• CC PipelineBusiness• ...
It’s How Big?• EvE Online: 340,000 Subscribers• Bejeweled Blitz: 10 Million Monthly active Users• World of Warcraft: 12 Mi...
The Facebook Top 40Top 40 FB Apps: Monthly Active020,000,00040,000,00060,000,00080,000,000100,000,000CityVilleTexasPhrases...
Architectural patternsforScalable EntertainmentEnvironmentArchetypesPatterns
© 2008 IBM CorporationEvery Architecture exists in an EnvironmentSettingIDCStudioInteractionOperationProductionCreationSeg...
© 2008 IBM CorporationEntertainment Creation: The StudioSegmentCreation: Content is createdProduction: Content is transf...
© 2008 IBM CorporationContent Creation Architecture
© 2008 IBM CorporationEntertainment as a Service: The Internet DatacenterSegmentCreation: Content is createdProduction: ...
SingleOnline Entertainment ArchetypesHybridSharded
Why one v.s. the otherShardedContentMultiple OperationsAuthored ContentStep ScaleModerate latencyUniverseContentSingle Ope...
© 2008 IBM CorporationInteractive Content DeliveryThis is NOT streaming video….This is NOT A transactional workload
Internals:Massively Multiplayer Online GamesArchitecture for millions of players
More than just the game engine
Massively TriumvirateEconomic SupportGame
Economic ServicesFinancial administrationInterfaceGame InterfaceUser Financial AccountDatabaseAccount ManagementWebsiteTax...
Support ServicesFireWallCSR InterfaceSupport Interface toFinancial SystemIn Game SupportInterfaceProblemManagement databas...
Game ServicesSupportAdministrationInterfaceFinancialadministrationInterfacePlayer behaviorreporting systemMap/Object DataM...
CCP’s EVE onlineSpecific examples of the importance of DBperformance in online entertainmentThese next slides where origin...
Our Goals• Keep up with load• Redundancy at every level• Simplification of design• Options!
Redundancy
Simplification• Devices• Old TQ = 8 devices• Proposed Design = 10 devices• Final Design = 6 devices• Entire system can go ...
OMGWTF DATA• Step one: (Upgrade the Storage Network)– WIN: 8Gb/s to 32Gb/s and no sharing ;)• Step two: (Upgrade the RAM)–...
OMGWTF DATA• Each server has:– 2 x 8 Core Intel x7560 processors at 2.26GHz– 32 x 16GB strips of RAM for a total of 512GB ...
OMGWTF DATA• Backup– Server Disk System Time Speed– Old TQ DB Current (30) FC Disks 73 mins 2.8 Gb/s– New TQ DB Current (3...
Internals:Social GamesA couple orders of magnitude largerHow to support HUNDREDS of millions
Special Notes: Social Games• Farmville had 1 Million daily players after 4 days and10Million after 60 days.• 75 million mo...
Architecture stack for Social GameData Persistance LayerDataCenterSesson MgmtWebServicesDB CachePhysical serverDBDBDBPhysi...
Questions ?!?4/27/2013Template Presentation - Session Z99 37
Thank You
Architectures in EntertainmentGeorge Dolbiergeorged@us.ibm.com
Entertainment Architectures 2011
Entertainment Architectures 2011
Upcoming SlideShare
Loading in …5
×

Entertainment Architectures 2011

297 views

Published on

This is the presentation I gave at the International Informix Users (IIUG) group in 2011. The presentation gives an introduction and overview of the architectures used in interactive entertainment (Online Games, social, MMO). I regret having to remove discussions covering video production, and video delivery, as well as CDN. Time did not allow, and those topics were not of as much interest to the audience.

Published in: Technology, Business
  • Be the first to comment

Entertainment Architectures 2011

  1. 1. Architectures in EntertainmentGeorge Dolbier Session: Z99IBMDay/Date/Time
  2. 2. Today’s AgendaIBM, Games? Really? And who is this guy?Interactive Entertainment IndustryArchitectural patterns for Scalable EntertainmentInternals Massively Multiplayer Online Games
  3. 3. Who is IBM again?And who is this guy?
  4. 4. IBM is embedded the gaming industrystarting with the CPUs in every game consoleSouce www.nexgenwars.com 2/27/2011
  5. 5. To the largest Online Infrastructures:EvE OnlineActive ServersWeightCPUSRAMEstimatedFLOPS~ 195About 2.5 Metric Tonnes> 420 CPU cores > 1 THZ> 7.5 Terabytes> 7.5 TerraTotal Bandwidth Estimate ~ 400 Gbs
  6. 6. George Dolbier, IBM CTO Interactive Media20+ year Technological innovator and Business Developer successfully pushing the limits of possibilities daily. A classicallytrained computer scientist with product production experience ranging from embedded software development to databasekernel engineer at Informix and Oracle and operating system kernel engineer at Sequent. Management and businessexperience ranges from IT operation and tactical management of small task oriented teams to executive level positionsmanaging large geographically dispersed groups.Relevant ExperienceArchitecture/design/development/deployment/operations of 30+ Interactive Media projects including social netowrks, streamingvideo, Online Games, Cinema in 15+ countries20+ years in SW Technology, 15 years in Interactive MediaArchitect and software designer with deep experience in producing software products and web scale services in the, social media,MMO, Film and connected console gameSuccessful projects in 15+ countries in the 3 major Geographies of Asia North America and EuropeDeveloper/Publisher relations include:CCP, Lucas, EA, Unity, Epic, Sony, Norsk Tipping, Obsidian, Zynga, Monumental, Cartoon Network, HasbroTechnology Reltationships includeBigWorld, Unity, Instant Action (FKA Garage Games) Vivox, Sleepy Giant, Parature, EpicExecutive experienceManage, drive and mentor teams up to up to 1500Manage geographically dispersed teamsProduce consistent sustainable double digit Y2Y profit and revenue growthLed creation of 10 million dollar Joint venture between IBM, Intel and CiscoLed creation of relationships between IBM and Games industry providers (BigWorld,Vivox,GNi)Drove business unit to build global revenue from 0 to 180+ million in under 4 yearsTechnical Delivery experienceDeployed 6000 servers infrastructure in china to support 3 million players in under 6 weeksDoubled Epic’s production environment allowing project to be developed on time on budgetImplemented production environment at ILM Used on StarTrek,TransFormers2, Harry PotterArchitected & Implemented HPC Cluster that discovered genetic cause of bone marrow cancerEducationKey Skills• IBM ExecutiveArchitect Certified• Open Group CertifiedMaster Architect• Portland StateUniversity – BSComputer Science• Online TechnologyArchitecture• Online Operations• Global Rollout• Social Media Architecture• IM Production• DFX Production• Follow the Sun dev.mgmt.• Strategy & Operational• Production Optimization• Technical CapacityAnalysis• OT&B Delivery
  7. 7. Industry IntroductionConvergent DivergenciesA little HistoryIt’s How big?
  8. 8. Interactive Entertainment is ConvergenceOnlineTechnical• Persistence• Centralized Simulation• Scale• CC PipelineBusiness• Software As a Service• Re-accruing revenue• Subscription• Economic Systems/Item Sales• Ongoing Operational Expense Interactivity• Scale• Autonomy• Co-Ordination• Co-Operation• Arbitration• Economics• CommunicationCommunity• Communications• In Game Support• Access to the game from outsideInteractive Entertainment Is…A scalable, persistent, form of entertainmentdelivered as a service
  9. 9. It’s How Big?• EvE Online: 340,000 Subscribers• Bejeweled Blitz: 10 Million Monthly active Users• World of Warcraft: 12 Million Monthly Subscribers• Lineage II: 19 Million Players world wide• FrontierVille: 19 Million Monthly Active Users• Farmville: 85 Million Monthly active users• Cityville: 100 Million monthly active usershttp://www.eveonline.com/pressreleases/http://us.blizzard.com/en-us/company/press/pressreleases.html?101007http://pc.ign.com/articles/925/925099p1.htmlhttp://www.appdata.com/leaderboard/apps (2/27/2011)
  10. 10. The Facebook Top 40Top 40 FB Apps: Monthly Active020,000,00040,000,00060,000,00080,000,000100,000,000CityVilleTexasPhrasesConduitQuizTaco!Yahoo!PetSocietyYelpMindJoltFriendBuzzZooskBirthdayGamesMallWorldMAUTop 40 FB pps: MonthlyActive(minus farmville & Cityville)010,000,00020,000,00030,000,00040,000,00050,000,000MAUAll Over:2 Million MAU
  11. 11. Architectural patternsforScalable EntertainmentEnvironmentArchetypesPatterns
  12. 12. © 2008 IBM CorporationEvery Architecture exists in an EnvironmentSettingIDCStudioInteractionOperationProductionCreationSegmentCreation: Content is createdProduction: Content is transformedOperation: Content is distributedand operatedInteraction: Content is interactedwithTimeConcept Launch FlightTechnology deploymenttimelineEnvironmentsStudio: Highly Collaborative Environmentwhere concepts are created andtransformed into products. This iswhere the virtual world is createdIDC: Consolidated, efficient, secureEnvironment where data andcommunications come together.This is where the virtual world runsSetting: Home, Office, Mobile, Capability,device, location, and connectivitythat allows a game or virtual worldto be interacted withEnvironmentsare differentiated byeconomic model as wellas physical, technical andHuman factorsDisciplinesare differentiated by the goalsof individuals, and how thoseindividuals relate to content
  13. 13. © 2008 IBM CorporationEntertainment Creation: The StudioSegmentCreation: Content is createdProduction: Content is transformedOperation: Content is distributedand operatedInteraction: Content is interactedwithTimeConcept Launch FlightTechnology deploymenttimelineEnvironmentsStudio:Highly Collaborative Environmentwhere concepts are created andtransformed into products. This iswhere the virtual world is createdIDC:Consolidated, efficient, secureEnvironment where data andcommunications come together.This is where the virtual world runsSetting:Home, Office, Mobile, Capability, device, location, and connectivity thatallows a game or virtual world tobe interacted withEnvironmentsare differentiated byeconomic model as wellas physical, technical andHuman factorsDisciplinesare differentiated by the goalsof individuals, and how thoseindividuals relate to contentSettingInteractionIDCOperationStudioProductionCreation
  14. 14. © 2008 IBM CorporationContent Creation Architecture
  15. 15. © 2008 IBM CorporationEntertainment as a Service: The Internet DatacenterSegmentCreation: Content is createdProduction: Content is transformedOperation: Content is distributedand operatedInteraction: Content is interactedwithTimeConcept Launch FlightTechnology deploymenttimelineEnvironmentsStudio:Highly Collaborative Environmentwhere concepts are created andtransformed into products. This iswhere the virtual world is createdIDC:Consolidated, efficient, secureEnvironment where data andcommunications come together.This is where the virtual world runsSetting:Home, Office, Mobile, Capability,device, location, and connectivitythat allows a game or virtual worldto be interacted withEnvironmentsare differentiated byeconomic model as wellas physical, technical andHuman factorsDisciplinesare differentiated by the goalsof individuals, and how thoseindividuals relate to contentStudioProductionCreationSettingInteractionIDCOperationSome might call this the service deliver environment… (we shun them)
  16. 16. SingleOnline Entertainment ArchetypesHybridSharded
  17. 17. Why one v.s. the otherShardedContentMultiple OperationsAuthored ContentStep ScaleModerate latencyUniverseContentSingle OperationsUser created ContentScale OutLatency TolerantHybridContentMixed OperationsInstanced ContentBi-Directional ScaleLow Latency
  18. 18. © 2008 IBM CorporationInteractive Content DeliveryThis is NOT streaming video….This is NOT A transactional workload
  19. 19. Internals:Massively Multiplayer Online GamesArchitecture for millions of players
  20. 20. More than just the game engine
  21. 21. Massively TriumvirateEconomic SupportGame
  22. 22. Economic ServicesFinancial administrationInterfaceGame InterfaceUser Financial AccountDatabaseAccount ManagementWebsiteTax/VAT Taxprocessing systemSupport AdministrationInterfaceFireWallDOS/DDOSAccount TransactionDatabaseDecision SupportDatabaseFraud Detection/Reporting SystemFinancial TransactionprocessingPayment ProcessorGateways
  23. 23. Support ServicesFireWallCSR InterfaceSupport Interface toFinancial SystemIn Game SupportInterfaceProblemManagement databaseDecision SupportDatabaseExternal SupportWebsiteSupportTicketingSystemCRMsystemSupport KnowledgeBase (Wiki)Interface to bugtracking systemInterface to game/Systems & NetworkManagementDOS/DDOS
  24. 24. Game ServicesSupportAdministrationInterfaceFinancialadministrationInterfacePlayer behaviorreporting systemMap/Object DataMnaagement andcomms logPlayer StatsGame WebsiteSimulationEngineFireWallDOS/DDOSRouting/load balencing“Instance”ManagerAuthenticationSystemPatch ServerCDNAuthenticationSystemTxt Chat ServerVoice Chat ServerSimulationEngineSimulationEngineSimulationEngine“Instance”ManagerXBLA/PSNGWTransaction management/cachingTransaction managementcachingIn Game Economic systemConsoleNetworkGameManagerGameManagerGameManagerGameManagerSimulationEngineSimulationEngineSimulationEngineSimulationEngineTxt Chat Log
  25. 25. CCP’s EVE onlineSpecific examples of the importance of DBperformance in online entertainmentThese next slides where originally presented byDerek Wyse, CCP VP Ops, at the EvE Fanfest 2011
  26. 26. Our Goals• Keep up with load• Redundancy at every level• Simplification of design• Options!
  27. 27. Redundancy
  28. 28. Simplification• Devices• Old TQ = 8 devices• Proposed Design = 10 devices• Final Design = 6 devices• Entire system can go live as a unit...No partial upgrade need• System maintainable on all units live(using the standby path)
  29. 29. OMGWTF DATA• Step one: (Upgrade the Storage Network)– WIN: 8Gb/s to 32Gb/s and no sharing ;)• Step two: (Upgrade the RAM)– WIN: 128GB of DDR2 to 512GB of DDR3, 111GB of DB Cache to 460DB Cache and it all takes our Page Life Expectancy (the time datastays in RAM) from 7.9 mins to 2.8 hours! Also, the old TQ databasewas at 97% Buffer Cache Hit Ratio while the New TQ database is at99.7%.• Step three: (Upgrade the CPU)– WIN: 12 to 32 logical processors plus a bonus for the new architecturegain over the old processor in our test and in lab tests. Plus,seriously... that screen shot by its self makes the server 400% faster.• Step four: (Upgrade the Storage)– WIN: Transfer speed from 2.9Gb/s to 10Gb/s tested. Total storagecapacity on SSDs doubled. Aggregate storage on all productioncapable tiers increased from 2TB non redundant to 11.5TB and 1 for 1redundant hardware.
  30. 30. OMGWTF DATA• Each server has:– 2 x 8 Core Intel x7560 processors at 2.26GHz– 32 x 16GB strips of RAM for a total of 512GB DDR3 RAM– 4 x Gigabit Ethernet cards– 2 x Dual-port 8Gb/s Fiberchannel cards– 4 x 15K 300GB SAS drives internal– 9 x 300 GB SSDs drives RAID 5 attached via IBM V7000– 36 x 600 GB SAS drives RAID 10 attached via IBM V7000• The whole system has:– Over 1TB of RAM– 64 logical processors (w/ hyper-threading enabled)– 32Gb/s of storage throughput capacity– 200,000 IOPS capacity– 51TB of RAW storage capacity (23TB after RAID and spares)
  31. 31. OMGWTF DATA• Backup– Server Disk System Time Speed– Old TQ DB Current (30) FC Disks 73 mins 2.8 Gb/s– New TQ DB Current (30) FC Disks 60 mins 3.2 Gb/s– New TQ DB New (34) SAS Disks 23 mins 8.5 Gb/s• Restore– Server Disk System Time– Old TQ DB Current (30) FC Disks 180 mins– New TQ DB Current (30) FC Disks 66 mins– New TQ DB New (34) SAS Disks 59 mins
  32. 32. Internals:Social GamesA couple orders of magnitude largerHow to support HUNDREDS of millions
  33. 33. Special Notes: Social Games• Farmville had 1 Million daily players after 4 days and10Million after 60 days.• 75 million monthly users in 9 months (from 0 to >France)• Most “games” are 80-90% Read, primarily non-transactional workload• Social Games heavy write load: Farmville 30%• Social game load spikes can be > 50%http://highscalability.com/blog/2010/2/8/how-farmville-scales-to-harvest-75-million-players-a-month.html
  34. 34. Architecture stack for Social GameData Persistance LayerDataCenterSesson MgmtWebServicesDB CachePhysical serverDBDBDBPhysical serverDBDBDBPhysical serverDBDBDBDB Cache DB Cache DB Cache DB CacheGameSerivcesPHPGame ServicesGameSerivcesPHPGameSerivcesPHPWeb LayerSesson MgmtWebServicesSesson MgmtWebServicesCacheCache Cache Cache CacheSesson MgmtWebServicesPhysical serverDBDBDBComm ServicesrouterLoad balencerfirewallLoad balencer Load balencerrouterfirewall
  35. 35. Questions ?!?4/27/2013Template Presentation - Session Z99 37
  36. 36. Thank You
  37. 37. Architectures in EntertainmentGeorge Dolbiergeorged@us.ibm.com

×