SlideShare a Scribd company logo
Steve Mushero, CEO & CTO
China Net Cloud
云络网络科技公司
Gaming Lessons Learned
Dealing with Massive Scale
& Terrible Network Infrastructure in China
Playtime's Over
High Performance Transaction Systems
October 27, 2009
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Things to think about
 China is big
 China is complex
 China’s Internet is big
 But different
 China loves playing games
 China’s games are online, online, & online
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
China is big
 China is big – size of the U.S.
 China is big – places far apart, 250-500ms
 China is big – 350,000,000 Internet users
 China is big – 300-500 million videos/day
 China is big – 1, 2, 5, 10 Gbps links common
 China is big – 75 cities of 1+ million people
 China is big – Another billion country folk, too
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Size matters - Tiny Internet Cafe
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
China is Complex
 Three ‘regions’
 North
 South
 Students
 Single monopoly ISP per region
 Single ISP per province within region
 Poor links between them
 Licensing & regulatory
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Where are you ?
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Pull the String Tighter
 Bad for phones
 Terrible for multi-player games
 Horrific for playing six-person basketball
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
BGP, what BGP ?
 China does not really use public BGP
 No, really !
 Remember those 2-3 regions ?
 Can’t usually dual/multi-home
 Get 2-3 IPs and Policy Route, GeoDNS
 Buy tunnels and proxy out-of-region – Ugh !
 Shard servers physically in regions/provinces
 Pay all your money to the CDNs for backhauls
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Bandwidth is Everywhere
 1-2 Mbps per house x 200 million
 Data Centers with 5, 10, 25 Gbps common
 LAN often bottleneck !
but . . .
 Inter-city bandwidth heavily loaded – video/BT
 Geo-DNS is per province
 Sucks when 100 million people share 1 DNS zone
 Can’t locate by city, so inter-city links loaded
 Students can’t route – where’s the potato ?
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Latency
 Here, there, and everywhere
 Tied to bandwidth and packet loss
 Tied to regional interconnects
 High latency is painful to games
 Variable latency is death to games
 Most games built assuming no/stable latency
 Re-write when coming to China
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Fun with Data Centers
 Everywhere, but very diverse
 10x Price differences BJ / SH to countryside
 Very small – often old office buildings
 Limited power – 2.5KW, often 10-15 servers per rack
 Poor service, connections & overloading a problem
 e.g. Shanghai inbound traffic congested at night
 Must understand the city, province, backbone
 Choose your cities, data centers, and links carefully
 Your first server must be in your home city
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Mobile is the Future ?
 World’s largest mobile market – 500+ million phones
 Already have MMORPGs, ads, other uses
 iPhone 3G launches this week
 Other 3G rolling out now – iStore, Android, BBerry
BUT . . .
 Mobile Data Centers limit bandwidth, so locate servers
 Congestion (think AT&T 3G in U.S.)
 High and variable latency (maybe)
 Compete with video, bit-torrent traffic
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
China Loves Games
 All On-Line – few consoles or PC games
 World’s largest market, by far – billions of $$
 Simple Casual
 Multi-Player Casual
 Simple MMORPG
 FPS MMORPG
 Mobile MMOPRPG
 Hyrbids
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Simple Casual Games
 Example: Spil Games (www.spilgames.com)
 Tic-Tac-Toe
 Card Games
 Jungle Shooter
 No skill required
 Usually free, no virtual goods
 Usually Flash or HTML
 Usually single player
 Little/No server communication
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Multi-Player Casual
 Example: Radiance
 (www.Radiance.cn)
 Volleyball
 Basketball
 Car Racing
 Usually free
 Usually virtual goods
 $1-2/game, $20-30 for goods – shirt to go 80-> 90% accuracy
 Complexity & Skill often undesirable – K to kill, S to shoot
 Flash or Heavy Client-based
 Server or P2P-based
 Lots of network traffic
 Very latency sensitive
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Casual Game Player
 No skill desired
 Use one hand
 Why ? Smoking !
 Multi-task
 Game
 Watch Movie
 Chat (MSN/QQ)
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Multi-tasking
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
MMORPG
 Example: World-of-WarCraft
 Often pay-to-play
 Huge clients – MB/GB
 Heavy network traffic
 Heavy server loads
 Latency sensitive
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
MMO Traffic
 Related to many things
 Number of on-line users
 Log in/out
 Chat
 Buying items
 Number of interacting users, per place
 Number of objects in use by each user
 Reaches Gbps quickly to server
 Have a model for 50Gbps at 1 million users
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Results ?
 Very challenging game play
 Worst for multi-player sports
 BBall, VBall, racing, tennis
 Latency 300-3,000ms
 No ball or 2 balls
 Locality is almost everything
 P2P not much better
 Many strategies required
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
General Strategies
 Cluster users by region / server
 Self vs. Auto-Select
 Eject slowpokes
 But, players want to go where the action is
 Measure & sort by real-time speed
 Reduce chatter, cut bandwidth needs
 Predict motion to reduce messages
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Some Strategies
 Choose server location / links carefully
 Buy links to North / South / Students
 Cache everywhere (Client, CDN, Region, Cafe)
 Traffic & Assets are huge
 Build for variable latency
 Retry, a lot, again
 Find experienced operations people
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
More Strategies
 Gather players
 Measure them
 Let players evict slow ones
 If P2P controller disconnects, game dies
 Simple, but painful
 Pre-install; 1GB download, lost on reboot
 1GB download/100MB updates monthly
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Lessons Learned
 The network hates you
 The only certainty is uncertainty
 Especially latency & basic availability
 If P2P, test users & use servers wisely
 Know where users are, if you can
 Manage them to zones & locality
 Measure & reduce bandwidth
 CDN everything possible
 Know the network, ideally in real-time
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
Summary
 China an exciting place, on many levels
 Internet challenging & fairly complex
 Truly massive game audience
 Innovative compensation for infrastructure
HPTS '09 - China Gaming - Steve MusheroChinaNetCloud
谢谢 - Thank You
Steve.Mushero@ChinaNetCloud.com
28
ChinaNetCloud
Computing Infrastructure for High-Growth Companies
China Net Cloud Technology (Shanghai) Company Limited
X2 Space 1-601
1238 Xietu Lu, Shanghai, 200032 China
www.ChinaNetCloud.com
Sales@ChinaNetCloud.com
+86-(21)-6422-1946 Confidential© 2009 ChinaNetCloud

More Related Content

Similar to Playtime's Over - Gaming Lessons Learned Dealing with Massive Scale & Terrible Network Infrastructure in China

Tech solutions and tricks in real time mobile multiplayer
Tech solutions and tricks in real time mobile multiplayerTech solutions and tricks in real time mobile multiplayer
Tech solutions and tricks in real time mobile multiplayer
DevGAMM Conference
 
Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?
Academia Sinica
 
ChinaNetCloud - China Internet Infrastructure Summary
ChinaNetCloud - China Internet Infrastructure SummaryChinaNetCloud - China Internet Infrastructure Summary
ChinaNetCloud - China Internet Infrastructure Summary
ChinaNetCloud
 
The 2008 Pc Builders Bible
The 2008 Pc Builders BibleThe 2008 Pc Builders Bible
The 2008 Pc Builders Bible
Sais Abdelkrim
 
SJNC13.pptx
SJNC13.pptxSJNC13.pptx
SJNC13.pptx
ssuserf6349e
 
ZNet Framework
ZNet FrameworkZNet Framework
ZNet FrameworkRay Yun
 
How to Plan for Performance and Scale for Multiplayer Games
How to Plan for Performance and Scale for Multiplayer GamesHow to Plan for Performance and Scale for Multiplayer Games
How to Plan for Performance and Scale for Multiplayer Games
Cloudflare
 
QoS in Network Gaming
QoS in Network GamingQoS in Network Gaming
QoS in Network Gaming
ruiquelhas
 
Harlan Beverly Lag The Barrier to innovation gdc austin 2009
Harlan Beverly Lag The Barrier to innovation gdc austin 2009Harlan Beverly Lag The Barrier to innovation gdc austin 2009
Harlan Beverly Lag The Barrier to innovation gdc austin 2009
Harlan Beverly
 
Cloudflare lower network latency = faster website loads
Cloudflare lower network latency = faster website loadsCloudflare lower network latency = faster website loads
Cloudflare lower network latency = faster website loads
Vu Long Tran
 
Video Game Industry Trends
Video Game Industry TrendsVideo Game Industry Trends
Video Game Industry Trends
Alex Ferrara
 
3 Person Teamwork
3 Person Teamwork3 Person Teamwork
3 Person TeamworkLee Thomas
 
Building fast,scalable game server in node.js
Building fast,scalable game server in node.jsBuilding fast,scalable game server in node.js
Building fast,scalable game server in node.js
Xie ChengChao
 
Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )
Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )
Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )
Anand Bhojan
 
Mobile Cloud Computing for Games - Gamelet
Mobile Cloud Computing for Games - Gamelet Mobile Cloud Computing for Games - Gamelet
Mobile Cloud Computing for Games - Gamelet
Anand Bhojan
 
Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]
Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]
Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]
Anand Bhojan
 
Dedicated Game Servers
Dedicated Game ServersDedicated Game Servers
Dedicated Game Serverswebhostingguy
 
Lets Play Together
Lets Play TogetherLets Play Together
Lets Play Together
Amir H. Fassihi
 
Quantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud GamesQuantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud Games
Academia Sinica
 

Similar to Playtime's Over - Gaming Lessons Learned Dealing with Massive Scale & Terrible Network Infrastructure in China (20)

Tech solutions and tricks in real time mobile multiplayer
Tech solutions and tricks in real time mobile multiplayerTech solutions and tricks in real time mobile multiplayer
Tech solutions and tricks in real time mobile multiplayer
 
Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?Games on Demand: Are We There Yet?
Games on Demand: Are We There Yet?
 
ChinaNetCloud - China Internet Infrastructure Summary
ChinaNetCloud - China Internet Infrastructure SummaryChinaNetCloud - China Internet Infrastructure Summary
ChinaNetCloud - China Internet Infrastructure Summary
 
The 2008 Pc Builders Bible
The 2008 Pc Builders BibleThe 2008 Pc Builders Bible
The 2008 Pc Builders Bible
 
SJNC13.pptx
SJNC13.pptxSJNC13.pptx
SJNC13.pptx
 
ZNet Framework
ZNet FrameworkZNet Framework
ZNet Framework
 
R bernardino hand_in_assignment_week_1
R bernardino hand_in_assignment_week_1R bernardino hand_in_assignment_week_1
R bernardino hand_in_assignment_week_1
 
How to Plan for Performance and Scale for Multiplayer Games
How to Plan for Performance and Scale for Multiplayer GamesHow to Plan for Performance and Scale for Multiplayer Games
How to Plan for Performance and Scale for Multiplayer Games
 
QoS in Network Gaming
QoS in Network GamingQoS in Network Gaming
QoS in Network Gaming
 
Harlan Beverly Lag The Barrier to innovation gdc austin 2009
Harlan Beverly Lag The Barrier to innovation gdc austin 2009Harlan Beverly Lag The Barrier to innovation gdc austin 2009
Harlan Beverly Lag The Barrier to innovation gdc austin 2009
 
Cloudflare lower network latency = faster website loads
Cloudflare lower network latency = faster website loadsCloudflare lower network latency = faster website loads
Cloudflare lower network latency = faster website loads
 
Video Game Industry Trends
Video Game Industry TrendsVideo Game Industry Trends
Video Game Industry Trends
 
3 Person Teamwork
3 Person Teamwork3 Person Teamwork
3 Person Teamwork
 
Building fast,scalable game server in node.js
Building fast,scalable game server in node.jsBuilding fast,scalable game server in node.js
Building fast,scalable game server in node.js
 
Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )
Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )
Energy Efficient Mobile Applications with Mobile Cloud Computing ( MCC )
 
Mobile Cloud Computing for Games - Gamelet
Mobile Cloud Computing for Games - Gamelet Mobile Cloud Computing for Games - Gamelet
Mobile Cloud Computing for Games - Gamelet
 
Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]
Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]
Gamelets - Multiplayer Mobile Games with Distributed Micro-Clouds [Full Text]
 
Dedicated Game Servers
Dedicated Game ServersDedicated Game Servers
Dedicated Game Servers
 
Lets Play Together
Lets Play TogetherLets Play Together
Lets Play Together
 
Quantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud GamesQuantifying User Satisfaction in Mobile Cloud Games
Quantifying User Satisfaction in Mobile Cloud Games
 

Recently uploaded

1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...
JeyaPerumal1
 
Bài tập unit 1 English in the world.docx
Bài tập unit 1 English in the world.docxBài tập unit 1 English in the world.docx
Bài tập unit 1 English in the world.docx
nhiyenphan2005
 
一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理
一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理
一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理
eutxy
 
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shopHistory+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
laozhuseo02
 
The+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptxThe+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptx
laozhuseo02
 
一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理
一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理
一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理
ufdana
 
This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!
nirahealhty
 
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
3ipehhoa
 
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024
APNIC
 
Italy Agriculture Equipment Market Outlook to 2027
Italy Agriculture Equipment Market Outlook to 2027Italy Agriculture Equipment Market Outlook to 2027
Italy Agriculture Equipment Market Outlook to 2027
harveenkaur52
 
How to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptxHow to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptx
Gal Baras
 
Latest trends in computer networking.pptx
Latest trends in computer networking.pptxLatest trends in computer networking.pptx
Latest trends in computer networking.pptx
JungkooksNonexistent
 
Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...
Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...
Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...
CIOWomenMagazine
 
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
3ipehhoa
 
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx
Brad Spiegel Macon GA
 
test test test test testtest test testtest test testtest test testtest test ...
test test  test test testtest test testtest test testtest test testtest test ...test test  test test testtest test testtest test testtest test testtest test ...
test test test test testtest test testtest test testtest test testtest test ...
Arif0071
 
guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...
Rogerio Filho
 
Comptia N+ Standard Networking lesson guide
Comptia N+ Standard Networking lesson guideComptia N+ Standard Networking lesson guide
Comptia N+ Standard Networking lesson guide
GTProductions1
 
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfMeet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Florence Consulting
 
一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理
一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理
一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理
keoku
 

Recently uploaded (20)

1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...
 
Bài tập unit 1 English in the world.docx
Bài tập unit 1 English in the world.docxBài tập unit 1 English in the world.docx
Bài tập unit 1 English in the world.docx
 
一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理
一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理
一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理
 
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shopHistory+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
 
The+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptxThe+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptx
 
一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理
一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理
一比一原版(CSU毕业证)加利福尼亚州立大学毕业证成绩单专业办理
 
This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!
 
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
 
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024
 
Italy Agriculture Equipment Market Outlook to 2027
Italy Agriculture Equipment Market Outlook to 2027Italy Agriculture Equipment Market Outlook to 2027
Italy Agriculture Equipment Market Outlook to 2027
 
How to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptxHow to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptx
 
Latest trends in computer networking.pptx
Latest trends in computer networking.pptxLatest trends in computer networking.pptx
Latest trends in computer networking.pptx
 
Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...
Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...
Internet of Things in Manufacturing: Revolutionizing Efficiency & Quality | C...
 
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
 
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx
 
test test test test testtest test testtest test testtest test testtest test ...
test test  test test testtest test testtest test testtest test testtest test ...test test  test test testtest test testtest test testtest test testtest test ...
test test test test testtest test testtest test testtest test testtest test ...
 
guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...
 
Comptia N+ Standard Networking lesson guide
Comptia N+ Standard Networking lesson guideComptia N+ Standard Networking lesson guide
Comptia N+ Standard Networking lesson guide
 
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfMeet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
 
一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理
一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理
一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理
 

Playtime's Over - Gaming Lessons Learned Dealing with Massive Scale & Terrible Network Infrastructure in China

  • 1. Steve Mushero, CEO & CTO China Net Cloud 云络网络科技公司 Gaming Lessons Learned Dealing with Massive Scale & Terrible Network Infrastructure in China Playtime's Over High Performance Transaction Systems October 27, 2009
  • 2. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Things to think about  China is big  China is complex  China’s Internet is big  But different  China loves playing games  China’s games are online, online, & online
  • 3. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud China is big  China is big – size of the U.S.  China is big – places far apart, 250-500ms  China is big – 350,000,000 Internet users  China is big – 300-500 million videos/day  China is big – 1, 2, 5, 10 Gbps links common  China is big – 75 cities of 1+ million people  China is big – Another billion country folk, too
  • 4. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Size matters - Tiny Internet Cafe
  • 5. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud China is Complex  Three ‘regions’  North  South  Students  Single monopoly ISP per region  Single ISP per province within region  Poor links between them  Licensing & regulatory
  • 6. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Where are you ?
  • 7. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Pull the String Tighter  Bad for phones  Terrible for multi-player games  Horrific for playing six-person basketball
  • 8. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud BGP, what BGP ?  China does not really use public BGP  No, really !  Remember those 2-3 regions ?  Can’t usually dual/multi-home  Get 2-3 IPs and Policy Route, GeoDNS  Buy tunnels and proxy out-of-region – Ugh !  Shard servers physically in regions/provinces  Pay all your money to the CDNs for backhauls
  • 9. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Bandwidth is Everywhere  1-2 Mbps per house x 200 million  Data Centers with 5, 10, 25 Gbps common  LAN often bottleneck ! but . . .  Inter-city bandwidth heavily loaded – video/BT  Geo-DNS is per province  Sucks when 100 million people share 1 DNS zone  Can’t locate by city, so inter-city links loaded  Students can’t route – where’s the potato ?
  • 10. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Latency  Here, there, and everywhere  Tied to bandwidth and packet loss  Tied to regional interconnects  High latency is painful to games  Variable latency is death to games  Most games built assuming no/stable latency  Re-write when coming to China
  • 11. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Fun with Data Centers  Everywhere, but very diverse  10x Price differences BJ / SH to countryside  Very small – often old office buildings  Limited power – 2.5KW, often 10-15 servers per rack  Poor service, connections & overloading a problem  e.g. Shanghai inbound traffic congested at night  Must understand the city, province, backbone  Choose your cities, data centers, and links carefully  Your first server must be in your home city
  • 12. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Mobile is the Future ?  World’s largest mobile market – 500+ million phones  Already have MMORPGs, ads, other uses  iPhone 3G launches this week  Other 3G rolling out now – iStore, Android, BBerry BUT . . .  Mobile Data Centers limit bandwidth, so locate servers  Congestion (think AT&T 3G in U.S.)  High and variable latency (maybe)  Compete with video, bit-torrent traffic
  • 13. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud China Loves Games  All On-Line – few consoles or PC games  World’s largest market, by far – billions of $$  Simple Casual  Multi-Player Casual  Simple MMORPG  FPS MMORPG  Mobile MMOPRPG  Hyrbids
  • 14. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Simple Casual Games  Example: Spil Games (www.spilgames.com)  Tic-Tac-Toe  Card Games  Jungle Shooter  No skill required  Usually free, no virtual goods  Usually Flash or HTML  Usually single player  Little/No server communication
  • 15. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Multi-Player Casual  Example: Radiance  (www.Radiance.cn)  Volleyball  Basketball  Car Racing  Usually free  Usually virtual goods  $1-2/game, $20-30 for goods – shirt to go 80-> 90% accuracy  Complexity & Skill often undesirable – K to kill, S to shoot  Flash or Heavy Client-based  Server or P2P-based  Lots of network traffic  Very latency sensitive
  • 16. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Casual Game Player  No skill desired  Use one hand  Why ? Smoking !  Multi-task  Game  Watch Movie  Chat (MSN/QQ)
  • 17. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Multi-tasking
  • 18. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud MMORPG  Example: World-of-WarCraft  Often pay-to-play  Huge clients – MB/GB  Heavy network traffic  Heavy server loads  Latency sensitive
  • 19. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud MMO Traffic  Related to many things  Number of on-line users  Log in/out  Chat  Buying items  Number of interacting users, per place  Number of objects in use by each user  Reaches Gbps quickly to server  Have a model for 50Gbps at 1 million users
  • 20. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Results ?  Very challenging game play  Worst for multi-player sports  BBall, VBall, racing, tennis  Latency 300-3,000ms  No ball or 2 balls  Locality is almost everything  P2P not much better  Many strategies required
  • 21. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud General Strategies  Cluster users by region / server  Self vs. Auto-Select  Eject slowpokes  But, players want to go where the action is  Measure & sort by real-time speed  Reduce chatter, cut bandwidth needs  Predict motion to reduce messages
  • 22. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Some Strategies  Choose server location / links carefully  Buy links to North / South / Students  Cache everywhere (Client, CDN, Region, Cafe)  Traffic & Assets are huge  Build for variable latency  Retry, a lot, again  Find experienced operations people
  • 23. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud More Strategies  Gather players  Measure them  Let players evict slow ones  If P2P controller disconnects, game dies  Simple, but painful  Pre-install; 1GB download, lost on reboot  1GB download/100MB updates monthly
  • 24. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Lessons Learned  The network hates you  The only certainty is uncertainty  Especially latency & basic availability  If P2P, test users & use servers wisely  Know where users are, if you can  Manage them to zones & locality  Measure & reduce bandwidth  CDN everything possible  Know the network, ideally in real-time
  • 25. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud Summary  China an exciting place, on many levels  Internet challenging & fairly complex  Truly massive game audience  Innovative compensation for infrastructure
  • 26. HPTS '09 - China Gaming - Steve MusheroChinaNetCloud 谢谢 - Thank You Steve.Mushero@ChinaNetCloud.com 28 ChinaNetCloud Computing Infrastructure for High-Growth Companies China Net Cloud Technology (Shanghai) Company Limited X2 Space 1-601 1238 Xietu Lu, Shanghai, 200032 China www.ChinaNetCloud.com Sales@ChinaNetCloud.com +86-(21)-6422-1946 Confidential© 2009 ChinaNetCloud

Editor's Notes

  1. <number>
  2. <number>
  3. <number>